Guide to Effective Test Software Development with TestStand

Test management software is a set of tools for managing the many components of a test application. Test management software ultimately handles measurements from an external Unit Under Test (UUT). Measurements are analyzed and compiled into reports that describe the integrity of a particular UUT. In modular test system architectures, test management software uses a set of layered components. The most immediate layer consists of code modules written in application development environments (ADEs), such as LabVIEW, LabWindows/CVI, and Microsoft Visual Studio. In order to perform instrument control or data acquisition, code modules access one or more driver services, such as NI-DAQmx, NI-488, IVI, and OPC. Each service accesses an interface with a specific hardware platform or bus, including GPIB, serial, data acquisition, modular instruments, vision, and motion. The lower layer consists of sensors that read the measurements from the UUT.

Modular test system architectures ultimately yield faster delivery of products to the market, achieve greater product quality, and reduce development and production costs. A major benefit of modular software development is code reusability. Individual components of a test system, such as a set of code modules that return measurements for a particular UUT, may be reused for multiple test applications. Code reuse reduces the development time necessary for future projects. Developing a modular system also reduces the complexity of a project because each module can be produced and tested individually. Additionally, modular architecture permits more concurrent development, thus decreasing the overall production time. Finally, test management software uses the capabilities of multiple ADEs to provide easy integration of multiple development platforms.

TestStand, as shown in Figure 1, directly implements a modular test system architecture. TestStand is a management tool that interacts directly with code modules written in almost any ADE.

Figure 1. An Example of the TestStand Sequence Editor

Initially, TestStand acts as a test system automation controller by automating all test system operations through test sequences composed of steps that execute sequencing operations or invoke code modules. Each code module performs a task consisting of measurement acquisition or non-test-related I/O, which includes tasks that do not directly relate to the integrity of a particular UUT, such as reading a bar code scanner for a UUT serial number. Each step invokes code modules directly through the set of TestStand module adapters. The adapters handle communication with specific module types, such as LabVIEW VIs, C/C++ DLLs, ActiveX automation servers, and .NET assemblies.

Additionally, TestStand separates UUT-specific test operations from system-level operations by distinguishing between UUT test sequences and process model sequences. Process model sequences perform all non-UUT-specific actions, including result tracking, report generation, database logging, and parallel execution. A process model also controls the execution of UUT-specific test operations and provides for looped UUT testing and concurrent UUT test executions.

You use ADEs primarily to develop code modules executed from TestStand. Code modules implement tasks such as measurement I/O, advanced data processing, and test-specific dialogs. The ultimate goal of a modular test system is to develop and maintain as little code as possible. In that respect, significant emphasis should be placed on developing highly modular and reusable code modules. The guidelines enumerated in this article provide a rubric for achieving a modular test system.

You also use ADEs to develop operator interfaces for a deployed TestStand system. Operator interfaces provide a deployable test application that executes on test stations. The application acts as an interface for any test operator executing tests. Operator interfaces are built from the open TestStand API and user interface components, which are exposed through a set of ActiveX automation servers. You can develop a TestStand operator interface in any ADE capable of ActiveX functionality.

TestStand includes several tools that facilitate modular test system development.

TestStand Sequence Editor

The TestStand Sequence Editor is an application program in which you create, edit, execute, and debug sequences. The sequence editor provides access to all TestStand features, such as step types and process models, and features debugging tools familiar to other ADEs. The debugging tools include setting breakpoints; stepping into, out of, or over steps; tracing through program executions; displaying variables; and monitoring variables and expressions during executions. You can use the sequence editor to start multiple concurrent executions. You can execute multiple instances of the same sequence, or you can execute different sequences at the same time. Each execution instance has its own execution window. In trace mode, the execution window displays the steps in the currently executing sequence. If the execution is suspended, the execution window displays the next step to execute and provides debugging options.

TestStand Operator Interfaces

TestStand includes several operator interfaces, as shown in Figure 2. Each operator interface is a separate application program developed in LabVIEW, LabWindows/CVI, Microsoft Visual Basic .NET, C#, or C++ (MFC), and is available in both source and executable formats. The operator interfaces are fully customizable. Like the sequence editor, you can use the operator interfaces to start multiple concurrent executions, set breakpoints, and single-step. However, you cannot modify sequences in operator interfaces, and the operator interfaces do not display sequence variables, sequence parameters, step properties, and so on. The source code for the operator interfaces provides a starting point for customizing the existing operator interfaces or creating your own operator interface.

|

Figure 2. An Example of a TestStand Operator Interface

TestStand User Interface Controls

The TestStand User Interface (UI) Controls are a collection of ActiveX controls for creating custom user interfaces in TestStand. These controls simplify common user interface tasks, such as displaying sequences and executions, and facilitate the quick generation of complete, full-featured operator interfaces. You can use these controls in any programming environment that can host ActiveX controls.

TestStand API

The TestStand API is a set of DLLs that are exported through an ActiveX automation server. The sequence editor and TestStand UI Controls use the TestStand API, which you can call from any programming environment and that provides access to ActiveX automation servers, including code modules you write in LabVIEW, LabWindows/CVI, and Visual Studio. All TestStand operations, including sequence execution and development, user management, and report generation, can be controlled programmatically using the TestStand API.

Variables and Properties

TestStand stores data values in variables and properties. Variables are properties you can freely create in certain contexts. You can have variables that are global to a sequence file or local to a particular sequence. You can also have station global variables, which have values that persist across different executions and even across different invocations of the sequence editor or operator interfaces. The TestStand Engine maintains the value of station global variables in a file on the computer on which it is installed. You can use TestStand variables to share data between tests written in different programming languages, even if the languages do not have compatible data representations. You can pass values you store in variables and properties to code modules. You can also use the TestStand API to access variable and property values directly from code modules. Each step in a sequence can have properties. For example, a step might have a floating-point measurement code property. The type of step determines its set of properties.

Report and Database Configuration Wizards

The report and database configuration wizards provide easy modification of the format and data of test results. Use the report configuration wizard to select common report formats, such as HTML, ASCII, XML, and ATML, or select any database with an ActiveX Data Object interface. The target structure and data are completely customizable from the wizard.  Additionally, multi-UUT tests can be compiled into a single report or separated into individual reports. 

User Management Wizard

Use the user management wizard to control the privilege level of any user. You can add or remove users from a list that is global to an entire station. You can quickly generate users from a set of common profiles, such as operator, developer, and administrator.

TestStand Deployment Utility

The TestStand Deployment Utility is a wizard that simplifies test deployment by determining dependencies for test sequences and code modules. You can also build an installation kit that makes distribution to several systems easy. The installation kit can include NI drivers, such as NI-DAQmx, NI-488, and NI-VISA, as well as run-time engines, including LabVIEW and LabWindows/CVI. With the distribution kit, you can also deploy user profiles, station globals, and station settings.

Process Models

TestStand installs a sequential model, a parallel model, and a batch model. Each model includes functionality for report generation, database logging, flow control, and multi-UUT or single-UUT testing. Additionally, the supporting code module source is provided with the process models. The source code provides a starting point for customizing existing process models or creating your own process models.

Example Programs

TestStand ships with a large set of example sequences and code modules that cover topics such as controlling parallel executions, dynamic user management, report modification, database customization, and customizing sequences. The examples are an excellent resource for learning the TestStand API. Additionally, examples are implemented in multiple ADEs for easier adoption by developers with differing backgrounds.

Due to the many layers available in TestStand, it can sometimes be difficult to determine the practices that minimize development effort while producing a scalable, reusable test system. For instance, should range checking be accomplished in a TestStand sequence or in a code module? At what level should new threads be spawned? Each decision yields different results.

Measurement I/O

You can acquire measurements from code modules or in steps from a TestStand sequence. In either case, TestStand handles the measurements. However, implementing measurement I/O in code modules provides greater modularity. Initially, modifications to one code module affect all steps that invoke the code module. In cases where a code module is reused several times, this feature can save development time. Additionally, you can reuse the same code modules in future tests. It is much more difficult to find and copy specific steps that perform measurements. Finally, multiple related code modules can be used together in different combinations, yielding greater flexibility.

A common template for instrument I/O is a set of code modules. One module initializes the instrument, returning an instrument handle. The second module reads a measurement from the instrument. The third module releases the driver resources using the instrument handle. The instrument handle is stored in TestStand and passed to subsequent modules to access the instrument drivers. Separating the code module operations provides scalability. Multiple measurements can be made using the measurement module while still using only one initialization and one release module. If the initialization and release code were in the same module, the instrument would have to be reinitialized every time a measurement is made. TestStand provides a Cleanup step group where the resource cleanup can occur. In case of a run-time error, this step group still executes. This template of measurement I/O produces robust and safe test systems.

Performing measurements in a step should only be done for one-time measurements. Using steps provides for quick test development. Additionally, performing tests inside TestStand does not require an external ADE for development or a run-time engine for deployment.

Limit Testing

Limit testing checks that a measurement fits within a certain range. You can perform this operation in a code module or a step. Limit checking in TestStand provides greater flexibility for a test system. Limit checking in a code module requires either hard coding the range of a test or passing parameters that determine the range. In TestStand, you can store limits in a particular step. You can easily modify limits in the sequence editor. Moreover, you can use a single code module with several different limits over many different steps. TestStand also provides tools to dynamically import limits for every limit test step from a file or database. You can easily modify limits in a text editor without requiring changes to an existing test system.

Switching

Hardware switches provide a method of routing signals. Switches are often required for large test applications to ensure that the proper signals are read. With switches, you can also read several sensors on a relatively limited set of hardware. TestStand provides top-level control of switch hardware through the embedded use of Switch Executive. It is possible to route signals from any step using Switch Executive. Developers that use the Switch Executive configuration in TestStand have greater flexibility. External code modules often require recompilation resulting in a rebuild of a particular test system. With TestStand and Switch Executive, changes are relatively easy.

Information Display to Operators

Operators execute test applications on test stations. Test applications that display information to operators are often more powerful and flexible. However, it is necessary to consider where to display information. Should the information display occur in the sequence or in the operator interface? This often depends upon the type of test application. Avoid test-specific items in an operator interface, especially particular steps or sequences required for an operator interface to run correctly. This removes flexibility from the test system. Moreover, the operator interface cannot be reused for future test systems. Instead, the operator interface should act as a test application framework capable of executing many different test sequences.

NI recommends using the sequence layer to display test-specific information to the operator. In most cases, the information displayed, such as UUT information and test flow decisions, is test-specific. Loose coupling between the operator interface and sequence provides all the benefits of modular test applications, including concurrent development and reuse. In cases where it is necessary for the operator interface to display test information, TestStand provides user interface messages, which act as a communication protocol between the sequence layer and the operator interface layer. However, operator interfaces should not rely on receiving specific user interface messages because ultimately the operator interface would be tied to a limited set of test sequences.

Reporting and Database Logging

In TestStand, the process model handles reporting and database logging. The process model is meant to separate system level operations from test-specific operations. Reporting and database logging are automatically implemented regardless of the test application. You can focus development effort on specific test application problems rather than recoding the same infrastructure. Moreover, you can easily make modifications to the structure or data included in a test report using simple configuration wizards. TestStand also provides the source code for the three default process models. Therefore, customizing the existing framework requires relatively little effort compared to creating a process model from scratch. Customizations are also reusable for future test applications.

User Management Tools

User management sets restrictions on the functionality of a test application to ensure that test applications are executed in the intended manner. The sequence editor provides a user management tool for creating users, deleting users, and modifying privileges. The user management tool is the simplest method for modifying user information. Moreover, the tool generates a distributable file so user information can be shared across several test stations.

TestStand also includes a set of functions for modifying users through the TestStand API. User information can be modified dynamically through the sequence layer, operator interface layer, or code module layer. Implementation at the sequence or code module layer is advantageous for automated user modification. Adding user management code to the operator interface also allows you to add new users or remove existing users.

Multithreaded Execution

Multithreading allows you to use resources and execute tests concurrently. New threads can be generated at the process model layer, sequence layer, or code module layer. TestStand provides the Parallel and Batch concurrent execution process models. The process model automatically handles the creation and destruction of threads so you can focus on test-specific items. Each thread generated in the process model is considered a test socket, which is a thread that tests a single UUT. With the process model, test sockets run concurrently, thus saving total test time. The process model also manages multiple reports and execution flow. Moreover, you can easily scale the number of test sockets using a model configuration wizard.

Threads can also be generated at the sequence layer. Sequence layer threads provide greater control, but require more overhead. The test sequence completely handles execution flow. Generating threads at the sequence level is excellent for handling concurrent executions that perform different tasks. Sequence-level threads are also important if more control is needed over the lifetime of a specific thread. Generating threads from a code module written in external ADEs places the full burden of multithreaded applications on the developer. The developer is fully responsible for forking and destroying separate threads, which is potentially dangerous if a thread is not properly destroyed before the test application is completed. The top-level control of the sequence layer or process model layer provides more flexibility. Changes are easier to make at the test management level rather than individual code modules. Ultimately, generating threads in code modules requires more development effort and does not use the existing tools in TestStand.

• TestStand Test Management Software
• SystemLink Systems Management
• LabVIEW for Automating Test and Validation Systems
• Download practical guides for building a test system.
• Learn how to Build a High Performance Modular Test System Using the NI PXI Platform