Advanced Vehicle Electronics Test with a Software-Defined Automated Test System

Author(s):

Tomohiko Adachi - Mazda Motor Corporation
Hideyuki OKADA - Mazda Motor Corporation
Noriaki Kittaka - Mazda Motor Corporation
Masaya Taniguchi - Mazda Motor Corporation
Yasuhisa Okada - MAC Systems Corporation

Background

It is common knowledge that vehicles are rapidly incorporating more and more electronics. Beginning with automated windshield wipers and door locks, electronics are now a part of many vehicular components, including lighting, air-conditioning, powertrain, infotainment, and even various kinds of safety systems. Initially, vehicles were equipped with only a few CPUs. Nowadays, the number of CPUs in a vehicle is close to hundred.

To deliver high quality products to customers, Mazda's Electronics Testing and Research Group evaluates the 'logic' and 'robustness' of all electronic components. 'Logic' refers to the functionality of each electronic component. To understand the concept of ‘robustness’, one must first understand that operating environments of electronic components are not always in the ideal state. For example, components may be exposed to extreme conditions, such as a fluctuating power supply voltage, high levels of noise, or the application of input signals of undesirable quality. ‘Robustness’ refers to the components’ ability to operate with the correct logic in extreme environments. In other words, we wanted to evaluate the extent to which each electronic component could withstand these difficult conditions.

The Challenge

The logic and robustness of electronic components have always been evaluated. In the days when there were only a few kinds of electronic components performing simple functions, components were tested individually in specially prepared environments. However, as the variety and functional complexity of electronic components increased, several issues surfaced. Nowadays, there is communication between multiple electronic component systems, and increasingly, one system's operation is dependent on results from other systems. In addition to testing systems individually, multi-system tests must be conducted to meaningfully evaluate the function of such systems. Furthermore, it is imperative that a systems’ robustness also be evaluated. Yet as the variety of components and units continue to increase, the number of items to be evaluated increase exponentially. Thus, it became clear to us that the evaluation system would need to be automated.

Mazda had been aware of these needs for roughly a decade. However, an evaluation system meeting all needs did not exist. Given the situation, we decided to tackle this gap head-on. In other words, we decided to build—and automate—a system to verify the logic and evaluate the robustness of electronic components operating cooperatively from the ground up.

The Solution/Benefits

The system we needed to build was going to be extremely large scale and complex. Accordingly, the development work was forecasted to take several years and to be conducted in stages. Figure 1 is a schematic diagram of the first stage. The Stage-1 system was comprised of the following elements: a HILS (Hardware-in-the-Loop Simulation) engine, a robot, and an image processing system. For the HILS engine, a NI HILS system comprised of PXI (PCI eXtensions for Instrumentation) products and RIO (reconfigurable I/O) module was used. The software operating on these hardware products was developed using the LabVIEW system design platform.

The reasons for incorporating HILS into this system are as follows. First is Mazda’s strong drive to develop and accomplish world-firsts. As one example of this mindset, Mazda is focused on advancing the development and practical use of model-based solutions to stay ahead of competition. Given this culture of innovation, it was only natural that, where possible, we wanted to evaluate electronic components utilizing models. However, we understood that certain components are simply impossible to model. Although alternative systems are typically used for parts unsuitable for modeling, Mazda decided to expand the functionality of the HILS system instead. Since the NI PXI platform is suitable for building various test systems, we were able to build both the HILS portion and the expanded portion on a single system.

There are certain components that cannot be converted to models, and the interface between human and vehicles is extremely challenging. Of those components that cannot be converted to a model-based format, the speedometer probably provides the simplest example. Imagine a speedometer displaying '50 km/h' as the value of a vehicle's speed. In this case, the command to display ‘50 km/hr’ is issued from the controller as an electrical signal. Such a signal can be evaluated during a simulation and can also be confirmed on the actual vehicle. As long as the system is functioning correctly, the speedometer would be expected to display '50 km/h' as a result of the receiving the signal. However, to check if '50 km/h' is actually being displayed, a human is needed to visually confirm the results. In other words, the process by which a driver perceives information from the vehicle cannot be converted into a model. Similarly, operations performed by the driver to communicate information to the vehicle also cannot be modeled. For example, a driver can press a button to turn the air-conditioning on/off, or tap the touch panel to operate the navigation system. Building a model that can exactly replicate the subtle states resulting from such operations is simply not possible.

Although it is extremely challenging to validate systems where we do not have models, we decided to put an extra effort to develop test engineering strategy and methodology for those challenging areas with Mazda’s motto “Be a driver”. As described above, it is extremely challenging to model interactions between the human driver and vehicle. Thinking about this as simply as possible, for a driver to communicate information to the vehicle (electronic components), a button or some other kind of instrumentation would need to be manipulated. And the performance of such manipulations would require human hands. Indeed, performing these kinds of manual tests have been the reality. However, manual testing requires an enormous amount of time and labor. Thus, it was critical that we develop a mechanism for automating evaluation. To meet this need, we added a robot to operate the electronic components. This computer-controlled robot pushed buttons and tapped the touch panel in place of a person. Similarly, we also needed to consider how information would be communicated from the car (electronic components) to the driver. Returning to the speedometer example, the traditional testing process consisted of a human visually checking the display to determine if '50 km/h' was actually displayed. To automate this portion of the evaluation, an image processing system was added. Specifically, the automated process involved photographing the speedometer’s display with a camera, and then processing the obtained image to determine whether the result was correct. If, for example, the speed was displayed using seven-segment LED display, the camera would photograph the LED display and process the obtained image to identify numbers and confirm the displayed speed. Alternatively, if the speed was represented by a needle display, image processing was used to measure the angle of the needle, and use that value to calculate the speed in kilometers per hour. By monitoring and comparing the signals from both the control unit and the display, the system can determine whether or not the speed is displayed correctly.

Under this system, it is also possible to substitute each electronic component using a virtual system (virtual electronic components) through software. The inability to proceed with evaluations until all electronic components are completed is a huge limitation. Since we wanted to begin testing and get results as quickly as possible, we substituted virtual electronic components for the actual components whenever possible. Not only do these virtual components operate similarly to the real thing, they also have a look and feel that closely resembles the actual component. This capability to utilize virtual components enables flexible testing. Depending on the contents of the test, actual parts can be used when deemed necessary; otherwise, virtual substitutes can be used.

The content thus far describes the logic verifying components of the automated testing system. In addition to this, adding functionality to evaluate robustness was necessary. Mazda places great importance on verifying robustness; we do not stop at simply determining whether the logic is correct. At Mazda, the evaluation of robustness involves first identifying the conditions at which logically correct operations approach their limits, and then determining the amount of margin. The amount of margin necessary for a pass result is determined according to independent, in-house standards. This evaluation process enables the company to provide an excellent user experience while also providing precise feedback to the design divisions of Mazda and its suppliers.

Of all the conditions used to test robustness, the most representative conditions would be a fluctuating power supply voltage and a high-noise environment. For example, the power supply voltage could be varied to ascertain the points at which the evaluated electronic components will stop operating correctly. To evaluate robustness during high-noise conditions, a noise simulator was added for our Second Stage system (see Figure 1).

However, the performance of the logic during unfavorable conditions is not the only element subject to robustness evaluation. For example, vehicle functionality includes the use voice commands to operate the vehicle. This too is subject to robustness evaluation. To achieve this, a speech synthesis system was added to the system. This system was compatible with both Japanese and English languages, and would issue voice commands in a variety of different voices, including male or female, young or old, and with differing intensities and articulations. We needed to evaluate the robustness of the system to identify the extent to which instructions could be correctly recognized for these different variations.

In the third stage, a GPS simulator was added. This GPS simulator was used to generate mock radio signals for GPS coordinates of various locations throughout Japan. This enabled us to conduct mock evaluations without actually travelling to the various locations. We wanted to be able to evaluate the system’s ability to function correctly despite varying the strengths of radio waves from the GPS simulator. In other words, this was another element to be evaluated for robustness. Note that for this third stage, the operational robot was updated to a type capable of touching or tapping multiple locations at once.

In the fourth stage, the GPS simulator was upgraded for worldwide use. In addition, Spanish language support was added for the speech synthesis system. Furthermore, we added a Bluetooth signal analyzer and a fuzz testing tool for detecting security vulnerabilities (see Figure 2.) However, a system with all the functionalities discussed above required considerable floor space. To make testing more convenient, we split the system into several more compact systems, each containing select functionalities of the original system (see Figure 3 and 4.)

In this manner, we succeeded in building the world's first system capable of automated evaluation of both the logic and robustness of multiple, cooperatively operating electronic components. As previously mentioned, the system we developed had not previously existed, and was based on entirely new concepts. Furthermore, it was an extremely complex, large-scale system. There are several factors that contributed to this achievement.

First and foremost was the existence of Mazda's Electronics Testing & Research Group. In other words, the fact that Mazda had its own specialized in-house test engineering team was instrumental in this success. As evaluation techniques for electronic components start to reach their limits, test engineers may wonder how to go about reforming them. With such a complex problem, nothing could possibly be resolved if the bulk of testing related matters were outsourced externally. The presence of this highly-specialized in-house team meant that possible solutions and corresponding implementation methods could be fully explored. In other words, it was critical that Mazda had its own internal organization capable of taking a leading role in resolving such issues.

Another contributing factor was the NI product family. One of NI's strengths is the NI platform-centric ecosystem, which includes compatible products from partners and related companies. For example, the system we built incorporated a robot, image processing system, speech synthesis system, and various other elements. Obtaining all of these elements from a single corporation would be difficult. Instead, we selected the optimal elements from various companies’ offerings and integrated them with NI's HIL system using LabVIEW and other solutions. This became a key factor in our success, and the NI ecosystem was a huge supporting factor. Of course, the incorporation, where applicable, of widely-used turnkey solutions was also an option. However, given our aim of building a first-of-its-kind system, NI solutions were most aligned with our needs.

Furthermore, the high performance and high level of programming freedom offered by NI products were perfect for developing this system. In terms of hardware performance, the high sampling speed (time resolution) was an important factor. In our system, logic verification required a time resolution in the order of milliseconds. And the effects of noise, on the other hand, could not be evaluated unless sampling was done in the order of microseconds. NI hardware were the only products capable of sampling in the order of microseconds. In addition, NI hardware contained a built-in user-programmable FPGA. No other products offered this level of freedom. In the case of turnkey-style solutions, there is a high chance that users will have to purchase an entirely new system for each updated generation of vehicle. Conversely, NI solutions are both flexible and sustainable. Almost all NI hardware can continue to be used, with the addition or modification of select modules likely to be the only adjustment needed. The ability to adapt to future needs is another huge advantage of this system.

Our newly-developed system enables logic verification and robustness evaluation of a multitude of cooperatively operating electronic components. The fact that no such system had previously existed makes the results that we achieved all the more incredible. In addition, we were able to automate the related operations and result judgements for various tests. This dramatically reduced workload and was a huge advantage. In the case of a single electronic component, testing time was reduced by 90% compared to manual testing. Furthermore, the required man-hours were reduced by 90% compared to old methods by using a camera to photograph the display of instruments—such as a speedometer—and then processing the image with our system’s automated evaluation functionality.

As of 2019, the following business impacts have been obtained.

1. Less time spent on troubleshooting with suppliers, and a better quality of system prototypes

• Reduced troubleshooting time through by sharing the detailed test condition using a common evaluation platform.

• Shortened the time to replicate phenomenon with low or infrequent occurrence.

• Improved the quality of the prototypes that the suppliers develop thanks to a more comprehensive test coverage.

   

2. 90% reduction in costs to upgrade

Mazda is now able to make it very smooth system upgrades by replacing PXI boards (Ex. I/O, communication interface boards) and PXI controllers, as opposed to having to reconfigure the whole test system or entirely buying a new turn-key. As a result, Mazda added ECU evaluation to the existing IVI evaluation functionality. From the total cost of test point of view, Mazda estimates that they saved about 90% of cost by using the NI platform.

3. Reduced the time to market

Traditionally, the time on the schedule to perform validation test is shorter than the time for the R&D phase, and of course the launch date for new car developments are fixed. Therefore, test engineers are required to build a validation environment and improve test coverage in a shorter period of time. Mazda has solved this challenge by taking full advantage of the NI platform.

Future Developments

Our recently developed system will continue to gradually evolve. Currently, we are aiming to be able to evaluate all electronic components with this NI product-based system. Thus, the Electronics Testing & Research Group is currently targeting all electronic components, including those related to powertrains. Prior to this newly-developed system, powertrain-related components were evaluated using a turnkey HIL system. These components were therefore excluded from target items for the present system.

Whether it be an engine or an electronic component, Mazda will continue to engineer world-firsts. And, as innovations in vehicle components also call for innovations in test, Mazda will continue to evolve its evaluation processes.

Author Information:

Tomohiko Adachi
Mazda Motor Corporation
広島県安芸郡府中町
Japan