Report Generation Explained

This section explores how the report generator is invoked from the top-level process model entry points. The TestStand report generator is implemented in the TestStand process models, and the way the report generator is called differs based on the TestStand version you are using.

Report Generation Changes in TestStand 2012

The TestStand 2012 and later process models include significant changes to implement a plug-in architecture for model sequence files. This architecture is more complex, but has a number of benefits over the previous design; for example:

• Ability to generate multiple reports for a single test, each with independent report settings.
• Ability to generate reports asynchronously.

Due to the plug-in architecture changes, the way that the process models call the report generator changed significantly in TestStand 2012.  The version specific sections below describe how the process models calls the format-specific report generation sequence files, which implement report generation. Despite the process model changes, the implementation of these format-specific report generation sequence files is very similar in TestStand 2010 SP1 and TestStand 2012. The image below provides an overview of the architecture changes and whether they contain significant changes in TestStand 2012 and later.

The TestStand 2012 model architecture changes the way the report generator is called, but not the report generator itself

Report Generation Sequence File Locations

While reading this document, it is helpful to inspect the report generation sequence files to view the implementation of each of the components.  The table below contains the location of report generation files

On-The-Fly Reporting

When you enable On-The-Fly Reporting option on the Contents tab of the Report Options dialog box, the process models progressively generate the report concurrently with the execution instead of waiting until UUT testing completes. In this case, the report is not generated in the same way. The ProcessOTFStepResult sequence, located in the format-specific sequence files, is instead invoked from an engine callback periodically through the test.  the on-the-fly specific sections below describe this process for TestStand 2010 and previous and for TestStand 2012 and later.

TestStand 2012 and Later (post UUT Report)

In these TestStand versions, the report generator is implemented using a process model plug-in (NI_ReportGenerator.seq).  The diagram below shows how the report generator is invoked:

The plug-in architecture allows all report generation code to be encapsulated in a single plug-in sequence file.  Report generation functionality is implemented in sequences called Model Plug-in entry points, which are called by the process model entry points at various points in execution.  Most of the functionality of the report generation plug-in is contained within the following plug-in entry points:

1. Model Plugin - Initialize - load the currently configured report options
2. Model Plugin - UUT Start - Obtain the report file path and create or load the report file
3. Model Plugin - UUT Done - generate the report text and save it to the report file

The image below shows the point in execution where each of these processes are invoked by the Single Pass Entry point of the Sequential process model.

The report text itself is generated within the TestReport sequence, which is called from within the Model Plugin - UUT Done sequence. TestReport in turn calls sequences in the format-specific sequence file to generate the report text.  The diagram below shows the call hierarchy from the entry point to the format-specific sequence file for generating the report body.

Call hierarchy for TestStand 2012 and later Post UUT report generation

TestStand 2012 and later (On-The-Fly Report)

When On-The-Fly reporting is enabled, the ProcessModelPostResults engine callback, which is invoked periodically throughout the execution. Unlike in TestStand 2010 and previous, this callback is not invoked after every step in the client sequence. When invoked, the Parameters.Results array contains a subset of the Resultlist containing all result data since the last invocation of the callback.

The ProcessModelPostResults calls the Model Plugin - OnTheFly Step Results sequence in the report generation plug-in, which in turn calls the ProcessOTFStepResults sequence in the format-specific sequence file to generate the report for the Results subset and append it to the report.

Call hierarchy for TestStand 2012 and later On-The-Fly report generation

TestStand 2010 SP1 and Previous (Post UUT Report)

In TestStand 2010 SP1, report generation functionality is implemented in many steps directly in the process model entry point. The image below shows the report generation steps in the Single Pass Entry point of the Sequential process model which implement these tasks:

1. Load the currently configured report options
2. Obtain the report file path, create or load the report file, and configure extra results to be included in the report, such as step limits
3. Generate the report text and save it to the report file
4. Post-processing the report, and regenerate the report path if the UUT status is part of the path

Note: The report is generated in section 3 below only when the on-the fly setting is disabled.  When on-the-fly reporting is enabled, the report is generated in the ProcessModelPostResultListEntry engine callback, which is called after each step as the test executes.  For more information on on-the fly reporting, refer to the On-The-Fly section of this document.

The report text itself is generated within the TestReport sequence, which in turn calls sequences in the format-specific sequence file to generate the report text.  The diagram below shows the call hierarchy from the entry point to the format-specific sequence file.

TestStand 2010 SP1 and Previous (On-The-Fly Report)

When On-The-Fly reporting is enabled, the ProcessModelPostResultListEntry engine callback, which is invoked each time a new result is added to the ResultList. When invoked, the Parameters.Result container contains the newly added result.

The ProcessModelPostResultListEntry sequence calls the ProcessOTFStepResults sequence in the format-specific sequence file to generate the report for the Results subset and append it to the report.

The report generation process begins within the TestReport callback. The report consists of three parts: Header, Body, and Footer. The TestReport callback invokes sequences in the format-specific report generation sequence files in the Get Report Header, Get Report Body and Get Report Footer steps.

In TestStand 2010 and previous, the text of each of these parts is stored in separate variables (Locals.ReportHeader, Locals.ReportBody and Locals.ReportFooter) until the Concatenate Report Header, Body, and Footer step of TestReport sequence is executed.  In TestStand 2012, the TestReport sequence uses a ReportSection object to contain the three report parts, then uses the ReportSection.GetAllText method to generate the full report string. 

Steps in the TestReport Sequence which generate the report

Generating the Report Header and Footer

The report header and footer are generated directly in the AddReportHeader and AddReportFooter sequences in the format-specific sequence files.   The TestReport callback calls the ModifyReportHeader callback after generating the header, and the ModifyReportFooter callback after generating the footer. 

You can modify these callbacks directly, or override these callbacks in a client sequence file to make changes to the header and footer after they are generated.  The callbacks contain a string parameter (Parameters.ReportHeader or Parameters.ReportFooter), which you can modify to change the header and footer string in the report.

Generating the Report Body

The report body generation process differs based on the report format.  The sections below describe the format-specific report generation process for post UUT report generation.

ATML and XML

For ATML reports, the AddReportBody sequence calls the ATMLsupport.dll to generate the report body. You can customize an ATML report TestStand creates by implementing a DLL that exports one or more exported extension functions. TestStand calls the functions while creating the report. For more information on implementing these extension functions, refer to the Customizing the ATML Test Results Report Generator topic in the TestStand help.

For the XML format, the AddReportBody sequence calls the PropertyObject.GetXML method of the TestStand API to generate the report XML.  This process cannot be customized, but all data in the ResultList will be present in the generated XML.

You can customize the XML or ATML report appearance, formatting, and included data by modifying the ATML style sheets.  For more information and examples of style sheet customization refer to the XML Reports section of the TestStand Help.

HTML and ASCII

For HTML and ASCII reports, the report body is generated differently depending on whether Sequence or DLL is selected for the report generator for producing the report body.

• Sequence option - the Get Report Body (Sequence) step generates the report by calling the AddReportBody sequence in the format-specific sequence file.  This method of report generation is slower, but customization is easier since the implementation is sequence based.
  
• DLL option - the Get Report Body (DLL) step calls into the ModelSupport2.dll to generate the report text.  This method of report generation is faster, but customization is more difficult.  While the source code is available for this DLL, customization is not as straightforward.
  

This section explores the Sequence implementation of report generation.  The list and diagram below shows the call hierarchy of the report body generation process. 

1. TestReport calls AddReportBody to generate the report body text
2. AddReportBody calls AddSequenceResultToReport to generate the report text for all steps in the MainSequence of the client sequence file
3. AddSequenceResultToReport adds sequence information to the report text, then calls PutResultsInReport to generate the report text for all steps in the sequence.
4. PutResultsInReport calls the PutOneResultInReport sequence for each step result in the ResultList. 
5. PutOneResultInReport generates the report text for a single step.  For sequence call steps, PutOneResultInReport recursively calls AddSequenceResultToReport to generate the report for the subsequence.
6. PutOneResultInReport calls the ModifyReportEntry callback after generating the step report text, which you can override in the client sequence file to make changes to the step report text

The steps in the PutOneResultInReport sequence with a name beginning with "add" are steps which insert text into the step result.  Before adding the data, each of these steps first perform a check to see whether the relevant property exists by using the PropertyExists function in the precondition expression for the step. This is necessary because not all steps will have some of the logged properties.  For example, this sequence contains Add High Limit and Add Low Limit steps, which only apply to the Numeric Limit Test step. In the case that the step does not have limit properties, the precondition expression will cause these steps to be skipped.

On-The-Fly Report Generation

Report generation for on-the-fly reporting is implemented differently from Post UUT report generation.  In this case, the implementation is different in TestStand 2010 and previous versus TestStand 2012 and later.

TestStand 2012 and Later

In TestStand 2012 and later, The on-the-fly report generator uses the ProcessOTFStepResults sequence, which is called periodically throughout the test to generate results for all new result entries.  This sequence generates the report for a multiple steps, and calls the GenerateReportAsReportSections to iterate through each new result. for each result, GenerateReportAsReportSections calls PutOneResultInReport to generate the report for the result data.  As with Post-UUT report generation, the ModifyReportEntry callback is called when the sequence report generation option is selected in the report options.

TestStand 2010 and Previous

In TestStand 2010 and previous, the on-the-fly report generator uses the ProcessOTFStepResult sequence, which is called for each new result.  This sequence generates the report for a single step, and therefore calls the PutOneResultInReport to generate the report for the result data.  As with Post-UUT report generation, the ModifyReportEntry callback is called when the sequence report generation option is selected in the report options.

After reading this document, you should have a good understanding of what components are involved in TestStand report generation, and how these components interact to generate the report.  You can continue to explore the TestStand report generator by investigating the sequence files referenced in this document. You can use TestStand debugging features to see the code in action, for example setting a breakpoint at the start of the PutOneResultInReport, and using a watch expression to view the value of Locals.ReportEntry to see the report string generate as you step through the sequence.

For higher level information on best practices for customizing TestStand reporting, refer to the Best Practices for NI TestStand Report Generation and Customization document before continuing.