From Saturday, Nov 23rd 7:00 PM CST - Sunday, Nov 24th 7:45 AM CST, ni.com will undergo system upgrades that may result in temporary service interruption.
We appreciate your patience as we improve our online experience.
From Saturday, Nov 23rd 7:00 PM CST - Sunday, Nov 24th 7:45 AM CST, ni.com will undergo system upgrades that may result in temporary service interruption.
We appreciate your patience as we improve our online experience.
Kirill Zuev, MZTA Research Center
We wanted to build an automated test system for new freely programmable industrial automation controllers to use in Industrial Internet of Things (IIoT) applications.
We leveraged modular instruments and multilevel LabVIEW, TestStand, and Switch Executive software to quickly design and implement a professional test system with minimal investments and human resources.
Kirill Zuev - MZTA Research Center
Anton Arkhipov - MZTA Research Center
Moscow Factory of Thermal Automatics (MZTA) Research Center, part of the MZTA Group of Companies, develops hardware and software for modern solutions in industrial automation and monitoring of assets for energy infrastructure, home automation, and high-tech production floors.
The KOMEGA controller series is a new product line that aims to help companies upgrade their assets to meet the challenges of industry digitization. The task closely aligns to the Industrial Internet of Things (IIoT) concept. The KOMEGA series controls and monitors objects in home automation and common industrial automation applications. The key feature of KOMEGA controllers is that customers can configure them while placing an order, before production begins. Customers can choose from a set of predefined configurations or create their own. Through an online advisor, customers can choose the number of analog and digital I/O, place additional relays and signal conditioning circuits (such as photocouplers or RC circuits), and configure the type and amount of controller memory. They can then submit the desired configuration to the digital production line where all components are mounted onto the controller PCBs according to the configuration. The minimum order size for any configuration is a single item.
Our test procedure includes firmware installation, functional and parametric verification of digital and analog I/O channels, and power consumption control. Controller configuration flexibility leads to special challenges for the test system. Traditional test systems assume large quantity orders. They are designed to quickly test identical items and do not expect frequent reconfigurations. These controllers, on the other hand, differ drastically from item to item due to their flexibility. Like a production line, which places components per individual configuration, the test system must be prepared to test these individually built controllers.
We did not initially consider the NI platform for this project. We planned to purchase a turnkey solution to save time and avoid making amateur solutions from scratch. However, available options were too costly. We discovered the NI platform and found that we could build a professional test system ourselves, save more than half the cost, and benefit from a scalable solution for future modifications and upgrades.
We can install PXI modules like digital multimeters, source measure units, analog I/O, as well as digital interfaces for RS485, CAN, Modbus, SPI, I2C, Ethernet, USB, and others, in a single PXI chassis. Some tricky digital protocols, such as SWD, are unavailable in a PXI form factor. However, they can be relatively easy to build with the help of universal FPGAs and a special software, without the need to dig into traditional FPGA programming difficulties.
Overall, the NI platform appealed to us because of its integration and because it was built to create fully automated test systems by small teams of developers quickly. In our case, one software engineer would perform all the programming.
We started to develop the software as soon as we agreed on a hardware configuration with NI representatives, while waiting for the hardware shipment and delivery. We simultaneously began learning LabVIEW and TestStand. By the time we first touched the hardware, we had already gained experience working with it in simulation mode.
We effectively used the modular instrument simulation feature in NI Measurement & Automation Explorer (MAX) and the virtual switches approach in Switch Executive. First, we recreated the entire hardware configuration in NI MAX in the form of simulated DMMs, scopes, DAQ boards, and more. Then we thoroughly configured all the switching routes into Switch Executive virtual switches and routes.
The Switch Executive proved to be a very useful tool. The intuitive graphical representation of all our complicated connections across multiple switching devices and virtual switches helped us significantly. Once we configured the routes, we used the achieved level of abstraction without the need to address individual relays. We also found the tool to be instrumental during final debugging by showing all the switching routes graphically while the tests system connected them to perform test algorithms.
We developed two test systems with the same measurement hardware: one for functional verification of the controllers through their connectors, and the other for fully automated production test with bed of nails test fixtures and parallel test of four controllers simultaneously within a multiboard PCB. We learned LabVIEW and TestStand while building the first one, and used the experience gained to develop the second one. The main sources of knowledge we used were online NI trainings, as well as open online resources and developer forums.
Having a decent C++ development background, we met the idea of graphical LabVIEW programming skeptically, but soon realized that LabVIEW and TestStand were a powerful tool set for automated test challenges. We benefited from multi-threading, object-oriented programming out of the box, which led to many ideas that we could implement in a more elegant way than we could with the traditional text languages we were familiar with. The graphical LabVIEW programming helps users understand the code structure quickly, which saves time when reusing old code or code made by other developers.
We used object-oriented programming in LabVIEW and TestStand to separate levels of abstraction and build a transparent software architecture. Thanks to instrument simulation, we had a large part of the software architecture, LabVIEW classes, and functions ready by the time hardware was available.
Every controller is marked with a QR code, which encodes the controller’s configuration with an implementation code. The test system must use the implementation code to test the controller accordingly. The test system also delivers functional and parametric verification of any possible configuration on both the software and hardware levels. After the test system reads the QR code, it installs the required firmware to the controller and creates an array of peripheral components according to the implementation code. After that, actual test begins channel by channel, with proper test sequence per channel.
We must perform functional verification through a controller’s standard connectors. This is necessary for debugging and modification purposes and is also the only way to characterize prototypes before the bed of nails fixture and its proprietary cables are in place. This is also the only feasible way to verify and characterize controllers in use that need calibration or repair. On the production floor, the test system uses specialized fixtures that accept multiboards containing four controllers.
The controllers have been designed in such a manner that we can connect them to a test system quickly with the help of a fixture with a bed of nails, which must contact the predefined test points of the controller. This helps make the test procedure faster and increases the test coverage compared to testing with standard connectors. For example, we can read digital signals at the processor’s pins after ADCs or read voltage coming to ADCs after input circuits. This ability is important in helping us locate the root of a possible malfunction and gathering measurement information beyond what is necessary for the verification itself.
The test fixture accepts four controllers mounted within a multiboard. Each of the four controllers include 51 analog and 71 discrete pins, two power supply pins, four RS485 pins, and four common point pins. The whole fixture utilizes almost 500 needles or nails. Several specialized fixtures can test different modules. We can easily swap the fixtures in less than a minute. We also implemented special features such as protection against using the wrong fixture and automatic test sequence selection for the fixture installed.
As long as controller configurations are different, the fixture must be flexible as well. The same pin of the fixture can be analog input for one configuration and become a digital output in another. A switch subsystem therefore reconfigures its routes and connects the required instruments to the pins as needed. Switch Executive virtual switches play a vital role in this task as the software loads the switches dynamically during the test sequence execution.
We use multiple digital protocols, from simple I2C and SPI to sophisticated SWD, to install a controller’s firmware and perform other operations. We did not find a proper protocol for this task among NI modules. Solutions available on the market did not meet our requirements due to cost and offering too much functionality for the task. So, we had to implement the protocols ourselves. We picked the PXIe-7820 FPGA module for the task. It delivered 128 extra lines in 3.3 V logic, so we also used this module to implement most of our digital logic.
Our team had no experience in FPGA programming, which caused concerns about such an FPGA use. However, we used LabVIEW FPGA with its ready-to-run examples and available libraries, such as OpenOCD, to quickly implement the required protocols with only a basic understanding of FPGA principles and only a few months of LabVIEW experience. The embedded FPGA simulator helped us debug the code even without having actual hardware in hand.
In a week, we implemented a simple version with passing pre-built bit sequences through the module. We then upgraded it to a full-functioning solution in which commands are built on the FPGA. Our team is thankful to NI technical support for explaining the best practices of implementing data processing on FPGAs.
Requirements for the test system in our digital production and IIoT vision included automated data accumulation and analysis. Failure statistics, failure pattern recognition, failure root cause investigation, and test method and procedure improvement all start with report generation and database connectivity to collect data. The test system generates an operator report and stores detailed data to the company’s database for further analysis. The operator report also contains advanced measurement results that can help the operator recognize singularities that might indirectly point to parameter degradation, even though the measurements are not currently part of the test procedure and should not affect the test/fail decision.
The calibration report of 16- to 24-bit analog inputs and 8- to 12-bit analog outputs is particularly important, as the resulting coefficients are stored in the controller’s memory and must also be stored in the database. The calibration report ships with the controller to the customer.
The test of one controller through connectors takes 370 unique measurements, while production test of four controllers within multiboards takes 2,536 measurements. This multiboard test takes approximately three minutes. Most of the time is for lengthy procedures that we cannot reduce further, such as firmware installation, power circuit transient process settlement, and more. All the switching and measurements are very fast compared to these necessary delays.
Development of the simplified test system connecting through a controller’s connectors took us approximately 50 days from the hardware purchase order. We expect the fully functioning system to be ready in about eight months. This development time perfectly meets our product schedule. Furthermore, it is mostly defined by time needed for special fixture cable manufacture and production floor schedule. A single engineer, who first used LabVIEW and TestStand during this project, developed all software for the test system.
In a short time and with minimal resources, we built a high-quality, fast, and accurate automated test system for the individually configured automation controllers. Graphical LabVIEW programming, test management with TestStand, and the interactive switching configuration through Switch Executive made it easier to create this solution.
Basing our solution on the NI platform saved us half the cost compared to our initial turnkey option. Our positive experience with quick system development during this project means we can consider building test systems in-house as an effective option for other projects.
Kirill Zuev
MZTA Research Center
Tel: +74957205444
zuev@mzta.ru