Best Practices for Embedded Software Testing of Safety Compliant Systems

Functional safety refers to process oriented safety certification standards that are applied to embedded systems for validation. As an example, IEC 61508 is a well-known functional safety standard that has been adapted to different industries such as automotive (ISO 26262) and medical (IEC 60601) and shares similarities to safety standards in the aerospace industry (DO-178B and DO-254). In addition to the lifecycle processes that are mandated for the embedded device to be certified, many of these standards enumerate similar lifecycle requirements for the qualification of the test system to be used.

Manufacturers manage development of complex systems by incorporating functional safety practices into the lifecycle of products. This is accomplished through risk analysis, requirements tracing,  and by documenting the testing and code analysis process throughout the development stage. For example, ISO 26262 specifically calls out the need for determining an Automotive Safety Integrity Level (ASIL) to determine the strictness of the testing process.

Figure 1. Knowing the severity, exposure, and controllability of a failure help to determine the ASIL of a requirement.

Producers need to start testing components and embedded software early in the product’s lifecycle to find defects earlier, thereby reducing the overall cost of product development. This helps to identify and address errors that would normally be discovered later in the development process. Correcting these errors before final verification leads to an overall savings in development time and project cost along with reducing the probability of a product recall. Manufacturers can perform several tests throughout the product’s lifecycle to help identify bugs throughout the process.

Figure 2. Different types of test can be performed throughout the development cycle to help reduce project time and cost later.

The figure above represents a software development cycle that shows recommended testing methods for different parts of the development process. Methods such as Hardware-in-the-Loop (HIL) and Model-in-the-Loop testing are specifically called out as recommended testing methods in various safety standards.  

A reconfigurable platform helps producers customize various tests throughout the development phase and reuse components across multiple applications. Component reuse enables manufacturers to use components from certified testing systems as building blocks for larger applications, which saves valuable resources such as time and money.

Figure 3.  Common test components can be reused in different stages of testing throughout a product's development.

As testing requirements change throughout the development processes, producers need a platform that enables them to manage these changes and reconfigure tests to meet their needs. For example, a car manufacturer testing in accordance with ISO 26262 may be able to reuse portions of an ECU test system across multiple cars. These components are throughout the development phase and include models, stimulus profiles, user interfaces, and data logging.

Figure 4.  Test components vary greatly and represent many different pieces of a project.

National Instruments tools are ideal for developing a standardized testing system to perform functional safety tests. NI offers an integrated hardware and software platform that allows producers to quickly develop a testing application that satisfies functional safety requirements. Because NI offers a modular hardware platform, similar testing applications can be scaled to fit the testing requirements of smaller or more elaborate embedded systems. NI has a software defined, reconfigurable platform that allows producers to test a component throughout the design process.

Software tools such as LabVIEW, LabWindows/CVI, NI VeriStand, NI Requirements Gateway and NI TestStand are used extensively within the automated test industry. These software tools allow users to easily interface with hardware products such as PXI systems and configure a modular test application. A PXI system fitted with a controller and various I/O modules allows a variety of components to be tested on a single platform. As testing requirements change, producers can add modules and modify software programs at a lower development cost than having a 3rd party developer reconfigure the system for a new test.

DO-178B and other safety standards specifically call out recommended testing methods such as HIL. The NI HIL platform provides an open hardware and software platform along with the greatest variety, value, and availability of products. Some of the hardware products NI offers for testing applications include:

• Multifunction data acquisition modules for analog and digital I/O High-performance bus interfaces for networks including CAN, LIN, FlexRay, MIL-STD-1553, and ARINC 429
• Fault Insertion Units for simulating opens and shorts in electronic test systems
• Real-time processors for deterministic execution of hardware I/O, data logging, stimulus generation, and model execution
• Reconfigurable I/O (RIO) with a user-programmable FPGA for more powerful and flexible test systems.

National Instruments offers all of the above hardware products in PXI form factor, which allows an embedded controller to communicate with and synchronize modules across multiple chassis. Real-time PXI controllers easily interface with software tools such as NI Requirements Gateway, NI VeriStand, and NI TestStand for requirements tracing, HIL simulation, and test automation. These software tools help certify testing applications in an efficient and cost effective manner and can be applied in functional safety practices across many industries.

Figure 5. National Instruments software tools can work together to address the many aspects of testing to functional safety compliance.

An effective, cost-efficient quality and testing strategy must be based on clearly defined requirements. Requirements help you specify the exact functionality of a product and guarantee that it is tested correctly and completely. Requirements management tools make the process of storing and analyzing the requirements cost-efficient.  Furthermore, compliance standards emphasize the need to prove the links between higher-level and lower-level requirements and the implementation of requirements in the product.  Most engineering projects start with high-level specifications, followed by the definition of more detailed specifications as the project progresses. Specifications contain technical and procedural requirements that guide the product through each engineering phase. In addition, working documents, such as hardware schematics, simulation models, software source code, and test specifications and procedures must adhere to and cover the requirements defined by specifications.

Figure 6.  NI Requirements Gateway can be used to show direct traceability from original project requirements to executed tests and test results.

Tracking the relationship from requirements to test, measurement, and control software is crucial for validating implementation, analyzing the full impact of changing requirements, and understanding the impact of test failures. Performing this coverage and impact analysis helps engineers meet their requirements and streamline development efforts.

Developing projects and electronic systems in compliance with functional safety standards requires strict adherence to requirements.  Every step taken throughout the project’s life cycle must be traced back and be directly inspired by a system-level requirement.  No matter what stage of development the project is in, testing is done on some level to validate the overall operation of the system.  Since this verification takes place at several points and needs to show traceability to requirements, it is critical that the requirements are consistent and not changing over time.  As with any project, this can be a challenge and when requirements for a system are modified over time, it can not only be very expensive to design to the new additions, but also verify the new requirements.  As a result, companies are looking for ways to have as much confidence in their requirements as possible in the beginning, which will result in less changes later and a significant reduction in the cost of test.

In looking at a standard like ISO 26262, requirements are assigned an Automotive Safety Integrity Level (ASIL), which is based on three key factors:  the severity of a failure, the frequency of the exposure of a failure, and the controllability to prevent, or fix, a failure.  The more information that is known about these three factors, the more confidence there will be in assigning the ASIL level, which will result in a well-defined requirement that is less likely to change later.  For example, if a prototype of an ECU is developed early, then the frequency of exposure of a specific failure can be monitored and recorded to understand how likely it is occurring, providing essential information in one of the factors for ASIL definition.

Typically, prototyping of a device like an ECU can serve two purposes:  to help with requirements definition and for requirements verification at an early stage in the development cycle.  The idea here is to take the requirements definition of prototyping and move it earlier in development.  Later, when the requirements verification is performed through a step like rapid control prototyping, a significant amount of time and effort can be saved by using the proposed same framework where specific test components can be reused from the earlier prototyping stage.  The time and effort put into the setup for the early prototyping will allow for specific components to be reused later, thus, minimizing the impact of early prototyping on the project’s cost and making it much more affordable than changing requirements later in the process.

Figure 7.  Performing early prototyping to better-define requirements in the same test architecture for other types of verification can save time in the overall development.

Let’s take a step back and just think about any project that is being developed.  In general, when someone wants to validate what they are working on and need to verify it to some form of requirements, they need to make sure that their testing is accurate.  Now this is especially true for regulated industries that enforce functional safety standards.  If something is verified incorrectly, there could be extremely negative consequences when the project is in production and in a safety-critical situation.  At the same time, due to the increase in electronic complexity in many industries like automotive and aerospace, there is more and more to test, but not necessarily an increase in the amount of time allotted to do so, which introduces the desire and need to automate some of the verification.  However, if you are to use some kind tool for test automation, how do you know if this tool is working as expected?  And the cost to develop your own test automation tool can be very high, especially if you want to ensure that it is designed with functional safety in mind.  On top of all this, there is the need for very specific and thorough documentation for overall validation.  Creating this documentation can be very time intensive to do it right, so if a tool was used, it would need to produce the appropriate artifacts.  So is manual validation the only way?

Using commercial-off-the-shelf verification tools that are qualified for functional safety projects in specific ways can address this need and give you the appropriate confidence in your test tool.  In fact, a NI partner, CertTech LLC, has done exactly this by creating a qualification kit for the test automation software tool, NI TestStand.  NI TestStand is a ready-to-run test management software that is designed to help you develop, execute, and deploy automated test systems faster.  Due to CertTech’s extensive experience in regulated industries and functional safety standards, they understand the requirements for using qualified tools specified by standards like DO-178C and ISO 26262.  The qualification kit for NI TestStand provides comprehensive requirements and test coverage for the most commonly used features, a full suite of tests verifying the provided requirements, and a readily extensible framework, so coverage may be extended as needed.  In addition, the necessary documentation is produced from the tool to be used as the necessary artifacts for compliance.  The documentation needed is essential because the overall goal is to show complete transparency for the verification process so the test can be recreated and every detail is clear on what was done.

With some of the newer functional safety standards like ISO 26262 and DO-178C, there is specific information requiring the projects to use ‘qualified tools’ for verification and validation activities that will not be manually reviewed, which inherently places additional emphasis on using qualified tools like NI TestStand.  These standards require that the overall impact of the tool not testing properly be assessed and then assign what standards like ISO 26262 call the Tool Confidence Level (TCL).  There are two main components that determine the TCL. The first is the Tool Impact (TI) and the second is the Tool Error Detection (TD).  TI0 or TI1 are the two classes of Tool Impact. TI0 is chosen when there is an argument that there is no possibility that the malfunctioning software tool can violate a safety requirement. For all other cases, TI1 is chosen.  The Tool Error Detection is classified as TD1 through TD3. TD1 is chosen if there is a high degree of confidence, TD2 for medium, and TD3 for a low degree of confidence.  The different TCL levels for a testing tool change the extra burden placed on the user.

Figure 8.  The Tool Confidence Level in the ISO 26262 standard is calculated by evaluating the tool’s error detection ability and overall tool impact.

In summary, if there is more confidence in a tool’s reliability, then there is less work that needs to be put in to prove that the tool is working as expected.  As a result, this means that anyone can take advantage of a commercial-off-the-shelf test automation tool like NI TestStand for their validation of functional safety projects and still have the confidence that is necessary to use this tool for requirement verification.

One of the biggest challenges in adopting new processes is dealing with the impact on the overall organization, especially when there are current processes already in place.  Companies attempting to develop electronic systems that will be in compliance with functional safety standards are a perfect example of this.  When they want to adopt a new process that is in accordance with a functional safety standard like ISO 26262, it is an organizational decision and philosophy that must be adhered to.  There are usually new individual roles introduced, specific project management processes, and new ways needed to synchronize the knowledge of key components.  Not to mention the fact that these companies already have their own development processes like automotive SPICE they have previously followed and now they are looking to have a process on top of a process.  As we’ve discussed throughout this paper, the verification components of the process are critical and must be considered in all of these aspects, so making sure that the integration of test into everything else involved is imperative. 

In order to address these challenges, application lifecycle management tools such as the offering from IBM Rational can be powerful enablers for helping.  Utilities incorporated into these offerings can help on both the front end of overall project management along with the back end knowledge synchronization and storage.  National Instruments has partnered with IBM to make this offering specifically valuable for the verification and validation space.  Through the integration of National Instruments testing tools with the IBM Rational offering, test components can be more easily managed and checked into the overall application lifecycle management to help with consistency, efficiency, and reuse.  In addition, test results from National Instruments products are seamlessly incorporated into IBM tools to ensure that consistent documentation is created and stored while also automating the process at the same time. This type of project management coupled with documentation and test results management can greatly increase the overall collaboration within an organization and ensure that there is a maximum amount of knowledge synchronization throughout the development process. 

Figure 9.  In the past, test has typically been thought of separately, but by integrating it into the other aspects of application lifecycle management, it is easier to have an end-to-end quality management system.

The introduction of functional safety standards provides companies with many challenges. Completeness of documentation, collection and validation of system-level requirements, organizational change and the efficient implementation of processes can be supported with the appropriate methodologies and approaches. Special attention shoud be applied to the reusability of test components and the automated traceability of requirements. Life-cycle management tools help you to keep track and prototyping consolidates the requirements early. Tools used for automated verification help organizations by use of qualification kits to adhere to safety standards.  National Instruments, also in conjunction with partner companies, provides a validation and verification platform, to support companies in these challenges

• Learn more about hardware-in-the-loop (HIL) testing for transportation applications
• Learn more about HIL testing for aerospace applications
• What is the ISO 26262 Functional Safety Standard?
• Watch an introductory webcast on DO-254 qualification