Whether you are building a new automated test system, or upgrading an existing test system, it is important to fully understand the total cost of test. The capital cost of test equipment is just one of the costs involved in developing and maintaining an automated test system. In many cases, the biggest cost is not capital equipment, but the time spent building and automating the test system.
The test executive is a vital part of a modular automated test system architecture – the overarching software that manages, sequences, executes, and reports the results of the tests that you run on your products. When building your test system, one of the key questions you have to answer is whether to build your own test executive from scratch (using application development software such as NI LabVIEW, C++, or .NET) or to buy a commercial-off-the-shelf (COTS) test executive.
A test executive is the high-level software application for organizing, deploying, executing, and reporting on automated tests. Test executives typically have two modes of operation:
In most cases, the actual test programs that control the instrumentation hardware are separate software modules written in either a test programming language, such as LabVIEW or NI LabWindows™/CVI, or a general-purpose programming language, such as C/C++, scripting languages, or .NET. By following a modular software architecture that decouples the test code modules, which are often specific to the device being tested, from the test execution system, which is common to all the different devices being tested, you have a scalable and flexible architecture that is less expensive to develop, support, and maintain in the long term.
For more information on a modular software architecture, refer to:
Developing a Modular Test Software Architecture With NI TestStand and LabVIEW.
The following list outlines some of the common functionality that test executives accomplish:
Many developers undertake the task of building their own test executives before they fully realize the magnitude of the task. The following section outlines some of the specific costs incurred by developing and maintaining a test executive as well as training employees to use the test executive.
For a comparison, the costs are contrasted with the analogous cost incurred by choosing a COTS test executive. For the purpose of this cost breakdown, we have picked NI TestStand as the COTS solution because it is a ready-to-run test executive, but still allows for full customizability for your specific test applications
Note: The costs and man-hours used in this comparison are estimates based on our experience working with hundreds of automated test customers, however, they are just estimates of a common case. The important points here are the identification of the multitude of tasks required, and the significance of the investment required to complete these tasks.
The most obvious cost associated with building a custom test executive is the up-front development cost of actually implementing all of the features of the test executive. The following table outlines specific features required to build a test executive. The table includes an estimate of the time required to implement each of these features from scratch, as well as the time required to implement the same features using NI TestStand.
| Feature Description | Development Time in Man-Days | |
Custom | NI TestStand | |
| Test Sequencing Development Environment Tools for specifying test programs; defining looping, branching, and conditional execution logic; specifying parameters to be passed to tests; specifying test limits and so on | 100 Can vary significantly depending on the complexity of the sequence logic and variety of compatible test languages available | 1 |
| Custom Operator Interface Custom interface written in an application development language such as LabVIEW, C/C++, or .NET. Load, display, and run sequences; controls for executing and debugging sequences; pass/fail banners, operator prompts, and so on | 40 | 8 Example operator interfaces in NI TestStand serve as a starting point for customizations |
| Sequence Execution Engine Execute test sequences and call test code modules, while evaluating results and generating reports | 15 | 0 |
| Database Logging Store test data and results in various formats for Oracle, SQL Server Access, mySQL, and other ODBC and ADO databases | 10 | 2 |
| Report Generation Generate reports in various formats (ASCII, XML, HTML, ATML, Excel) | 15 | 1 Built-in report generation that can be customized |
| User Management Password-protected access to test executive capabilities with different access levels. Environment for setting up/managing user information | 5 | 0 |
| Parallel Testing Architecture for loading and running test programs in parallel, while still providing full functionality of unit tracking, logging, etc. Should include constructs to prevent race conditions such as locks and semaphores | 100 | 1 |
| Unit (Serial Number) Tracking Track DUTs/UUTs by serial number for identification | 5 | 1 |
| Interface to Coding Languages Architecture that allows calling test code written in a programming language (such as LabVIEW, C/C++, or .NET) and getting results and measurements back. Assumes interfacing with two languages | 15 Varies based on the number and variety of compatible test languages/file formats | 0 NI TestStand calls LabVIEW VIs, LabWindows/CVI DLLs, ActiveX/COM modules, .NET Assemblies, scripting languages, and executables |
| Deployment Tool Create image/installer to replicate test system on production systems | 20 | 3 Built-in deployment utility to create installers including NI TestStand Engine, test code, operator interfaces, configurations, and additional installers |
| Localization Ability to change languages for displayed text. Assuming one additional language. | 15 Varies significantly based on the number of localized languages | 2 Modular localization imports all strings from ini text files for easy localization |
| Documentation Both the development tools and the operator interface must be documented with hardcopy manuals and online help | 20 | 1 Both NI TestStand and the shipping operator interfaces are fully documented |
When evaluating the cost of developing custom software, it is important to realize all of the costs associated with an engineer. For instance, even though the cost of an engineer may be calculated by the hour or day, it is rare that this includes intangibles, such as benefits, healthcare, and pension, or the cost of office tools such as a computer, a desk, and office space. When all of these aspects are included, the average cost of a capable software engineer can easily reach $150,000 USD per year. However, in the analysis that follows, we will assume that, in USD, the cost of an engineer is $100,000 per year, or roughly $400 per working day.
| Custom | NI TestStand | |
| Development Time in Days | 365 | 18 |
| Development Cost Per Day | $400 | $400 |
| Number of Test Developers | 10 | 10 |
| Cost of Software Per Developer | $0 | $2,380 |
| Number of Test Systems | 50 | 50 |
| Cost of Software Per Deployment | $0 | $741 |
| Total Development Cost | $146,000 | $68,050 |
Total savings by buying a COTS test executive: $77,950 (53%)
Developing your test management solution is only the beginning of the task. Software is not a one-time cost, but an ongoing effort. Software must be maintained over time in order to:
Many times, this maintenance effort can become a major undertaking and consume months of development time. For instance, when new operating systems are released, simply porting the same application to a new OS can be a monumental task. Custom-developed software must be updated and maintained by the end-user or third-party developer on a contract basis and can be very expensive.
On the other hand, ready-to-run COTS test executive packages such as NI TestStand are updated by the vendor regularly, simply to stay competitive and viable. This policy enables users to take advantage of the latest technologies (and get fixes to bugs and new functionality) at a fraction of the cost.
Again, associating a definitive cost on maintenance and support is impossible because every case is different. However, in this example, we assume that the overhead of maintaining an in-house test executive requires the effort of 15 percent of the test development department (with these engineers often being called Test Executive Architects), and that the cost of maintaining the software increases by five percent every year. On the other hand, the overhead for NI TestStand would be substantially lower, at around five percent of the department, because fewer customizations are needed. Test Architects are still needed and we also included the cost associated with upgrading to a newer version of NI TestStand.
Keeping our earlier assumptions, and a test system lifetime of 10 years, we can develop the following cost analysis:
| Custom | NI TestStand | |
| Percentage of Time (Compared to Development Time) Spent in Maintenance | 15% | 5% |
| Maintenances Per Year | 2 | 2 |
| Annual Cost of Support and Upgrades | $0 | Service is included with active software subscription. You must maintain an active license to access service. |
| First Year Total Maintenance Cost | $20,000 | $14,284 |
| Year-Over-Year Increase in Maintenance Cost | 5% | 5% |
| Total Maintenance Cost (10 Years) | $550,911 | $294,599 |
Total savings by buying a COTS test executive: $256,312 (69%)
When standardizing on a standard test package, a company must provide detailed training to the appropriate users. For test developers to be productive at creating test code using a test executive, they must first be trained on how to use the test executive. In addition to basic training, developers can also greatly benefit from learning best practices and working through example problems.
Unfortunately, training is a major expense that many companies overlook when deciding on a software platform. When developing a custom test executive, keep in mind that in addition to developing the software, you will also have to provide standard training classes as well as local experts who can deliver the training material.
Many COTS software, including NI TestStand, offer on-site (as well as online) employee training that ranges from basic introductory topics, such as how to develop and deploy tests, to more advanced topics, like customizing the test executive and creating operator interfaces. In addition, these courses are taught in most major languages.
Although the actual cost of training varies between organizations, we can see the cost of developing and delivering training by making some assumptions. In this example, we continue with the assumptions from earlier.
| Custom | NI TestStand | |
| Course development time (course material and examples) in days | 40 | 3 |
| Course duration in days | 5 | 5 |
| Cost of course setup, computers, materials, etc | $10,000 | 0 |
| Cost of NI TestStand course per developer (Part 1 – Test Development) | - | $2,175 |
| Cost of developing training material | $16,000 | $0 |
| Cost of delivering training – includes opportunity cost of developers | $28,000 | $33,400 |
| Total Cost of Initial Training | $63,000 | $41,000 |
Initial training savings by buying a COTS test executive: $22,000 (35%)
Note: Although the savings in the training cost is dwarfed by savings in other categories, such as development and maintenance, keep in mind that
There are inherent trade-offs in deciding whether to build a custom test executive or buy a commercial off the shelf solution. A COTS test executive has the potential to reduce cost by taking away a lot of the development, maintenance, and training burden. However, every test application is different, and it is critical to evaluate whether the built-in functionality can adequately meet the wide variety of testing requirements for your specific application.
NI TestStand is a ready-to-run test executive that is fully customizable for any test application that is designed to deliver the cost benefits of a purchased product while maintaining the flexibility of a custom-developed solution. This customizability is made possible through a very modular software architecture that exposes all the functionality of the underlying execution engine via an easy-to-use API.