SystemLink vs. Homegrown Test Operations Solutions

Manufacturers and product engineers using fleets of test equipment must efficiently manage all their assets and the data they produce. Without a robust solution with advanced data analysis capabilities, your test data is unlikely to be used to its fullest potential.

Today, most organizations use some sort of homegrown test operations solution built in Excel, Crystal Reports, or something similar. While you can achieve some powerful and complex functions with simple spreadsheets, your time to insight is greatly impeded by relying on these outdated methods of data analysis. These legacy solutions simply cannot effectively analyze the vast amounts of data produced by modern test systems.

Alternatively, you may try to develop a more robust homegrown software solution that you can completely customize to match your organization’s needs. However, tackling this challenge requires committing expensive development resources for months or years at a time. In the meantime, you may miss the opportunity to improve your product design and processes with an existing solution. Beyond that, patching and upgrading your solution is an ongoing time commitment.

Most homegrown solutions also create data silos, making it hard for teams across your organization to share test data. In addition, these solutions may not be able to index data and make it searchable. Many solutions also are incompatible with common test hardware and don’t monitor for uncalibrated equipment or test errors.

To remain competitive, organizations must invest in test operations solutions that reduce time to insight and make tangible improvements. An ideal testing solution will help you be more successful in the following key tasks:

• Collecting and store test data
• Making your test data accessible throughout your organization
• Analyzing large amounts of data to effectively produce useful insights 
• Managing your test equipment to ensure proper calibration and utilization

To address several challenges with test operations, NI developed SystemLink software—an automated, feature-rich solution that enables organizations to manage system health and improve testing workflows. SystemLink is designed specifically for engineering and manufacturing use cases as a comprehensive, customizable, and cost-effective test operations platform that streamlines and increases the speed and efficiency of testing processes, equipment, results analysis, and reporting.

SystemLink Server and Web Interface

SystemLink starts with the SystemLink Server, the base of operations for your testing environment and where your data is securely stored. Here, you can manage roles, permissions, and access for different users according to your team’s security protocols. You can also view and monitor the health status of each test system.

The core functionality of SystemLink is contained within the four powerful test management modules, all accessible through an intuitive web interface:

• Asset Module—The Asset Module helps teams ensure compliance by monitoring performance and system health data in real time. Teams can also generate utilization reports while managing software, calibration, and configuration workflows from a single dashboard. These functions enable you to obtain more accurate test results and gain more value from your underutilized test assets.
• Test Module—The Test Module makes it easy to collect test data, including parametric and waveform data, and store it in a central location where it can be viewed and analyzed. It automates complex analysis of test results while providing a variety of useful reports and data insights such as first pass yield, failure paretos, statistical process control, and other key metrics. The Test Module reduces time to insight by automating many time-consuming test analysis tasks and providing customizable visualizations of test results. This module empowers teams to quickly perform root cause analysis, fix bottlenecks, and automatically monitor and improve process efficiency.
• TDM DataFinder Module—The TDM DataFinder Module simplifies test data management by allowing teams to access, manage, and search test data from anywhere. With this tool, your team can automate the cleaning, organization and validation of new test data. After that, the Technical Data Management Streaming (TDMS) file format allows that data to be indexed and searchable across your entire organization, so you can use that data in software like LabVIEW, DIAdem, TestStand, and more. Thanks to these advanced search functions and a networked data repository, SystemLink makes test data silos a thing of the past.
• Software Configuration Module—The Software Configuration Module ensures all test equipment stays up-to-date with the right software. With this tool, you can replicate software configurations and automate remote deployments to globally distributed test assets. With automatic installation, your team won’t need to physically access each device to complete updates anymore. Instead, they can be scheduled for a time of low activity to avoid disrupting your testing workflows.

When it comes to product testing, consider how your software solution helps you get the job done. To help you make the best decision for your team, let’s look at how SystemLink differs from common homegrown solutions in terms of user experience, maintaining system health, data management, analysis, and total ROI.

Where homegrown solutions may need to make use of multiple programs or platforms to achieve similar functionality, SystemLink creates a centralized, user-friendly experience for all of these powerful tools. This eliminates the massive inefficiencies and steep learning curve of spreading your test operations management across multiple tools.

Build Custom Dashboards

Part of the power of SystemLink is the ability for your team to easily build custom dashboards to meet a wide range of unique testing needs. Even as you introduce new test equipment, software tools, or testing workflows to your operation, you’ll be able to easily access the most relevant data, leading to greater innovation and faster operational improvements.

Access and Manage Systems Remotely 

With SystemLink, access to test data and systems isn’t limited to any single building or lab. Instead, team members will be able to manage test systems from one central hub anywhere on the network. Operators can remotely monitor test equipment, access test data, push mass software updates and configurations, and access all relevant system details including asset utilization data and alerts from a common web interface. This interface reduces the need to physically access globally distributed test equipment or rerun the same tests at each site. Instead, SystemLink helps you to access, visualize, and analyze test data from other labs around the world for greater operational efficiency.

Support for NI and Third-Party Hardware/Software

By using the versatile TDMS file format, SystemLink achieves tight interoperability with NI products such as LabVIEW, TestStand, DIAdem, and more. However, many validation labs and manufacturing lines contain hardware and software from a variety of vendors. SystemLink includes tools for incorporating a wide range of third-party software and hardware technologies, offering both manual data entry options and a number of powerful APIs (G, HTTP, Python, and .NET) that allow you to programmatically access SystemLink data services outside of the SystemLink web application. As for hardware, SystemLink works directly out of the box with all NI products, and it is simple to integrate with SCPI-compliant third-party equipment.

Simplified User Experience 

In terms of user experience, homegrown solutions can’t usually deliver the same level of sophistication as SystemLink while remaining quick or easy to use. Even as you extend your fleet of test equipment and integrate other software tools, you’ll still be able to manage all your test data via SystemLink’s simple user interface.

If you are using spreadsheets to manage your test data and equipment, visibility into your operation’s bigger picture is limited. With SystemLink, unified dashboards provide greater visibility into the high-level analytics regarding system health and aggregate test data to better monitor KPIs. Combined with powerful tools to drill down into data, such as advanced search and automated analysis functions, SystemLink has everything you need to gain a clear overview of your entire testing operation.

While a homegrown solution is likely to be customized to fit your unique equipment needs, managing your test equipment and monitoring the health status of your entire fleet can be challenging. The primary obstacles are low visibility, manual maintenance workflows, and a high error rate.

Manual Maintenance and Calibration

When managing any fleet of test systems, maintenance and calibration are unavoidable. However, with many homegrown solutions, maintenance and calibration rely on costly and time-consuming manual processes. Test operators must spend time calibrating machines and performing repetitive maintenance tasks. Instead, these valuable team members could be using that time to collect and analyze test data, producing insights that can help drive your organization forward.

Without networking, operators can’t connect different systems to monitor and manage data from a single location. As a result, team members must often physically move from one test asset or system to another to verify health status, calibrate the machine, or update the software. 

Deploying Software

Without automated control or a suitable network environment, deploying software updates can be error prone. Furthermore, with team members installing software on geographically distributed machines using USB drives or similar hardware, software deployments can take weeks or months to complete. This rollout can delay essential testing activities or even halt production because of process errors.

That said, laborious software deployment isn’t the only issue to consider here. Teams also need to be able to easily verify that the correct versions of software and hardware are being used. Most homegrown solutions have no way of ensuring this sort of configuration compliance. What’s more, manual software deployment may accidentally create inconsistencies across machines. If two testers are analyzing the same data on different versions of the software, you may have errors in your test data that lead to product safety issues and other problems.

Without proper system health monitoring, the potential for this type of overlooked error is greatly increased and may take a significant amount of time to be noticed. Beyond that, after a systematic error is caught, the process of filtering out bad data can be wildly inefficient. 

With SystemLink, teams can closely monitor system health and calibration statuses, as well as any notifications or alerts that require immediate attention. SystemLink automates maintenance and calibration workflows, reducing downtime and schedule maintenance for optimal productivity. At the same time, SystemLink streamlines new equipment setup, reduces errors, and boosts efficiency by allowing for automatic software deployment.

Calibration

Just like sweeping your warehouse floor, calibrating your test assets is a recurring process—though perhaps a bit more complicated and time-consuming. To reduce downtime, SystemLink’s Asset Module can help teams schedule and automate calibration workflows and perform diagnostic and configuration tasks remotely. 

In many cases, teams using homegrown solutions discover that their equipment isn’t calibrated correctly only after an asset shows errors in the test data. With SystemLink, you can prevent costly retests through dynamic system health monitoring. You’ll gain greater visibility into the performance of your test assets and be notified ahead of time if a machine requires calibration. You can even have your equipment self-calibrate and test itself for accuracy. Beyond that, a history auditing feature enables users to track, log, and save various maintenance directives for future reference.

Software Deployment

Through the Software Configuration Module, SystemLink enables administrators to remotely deploy software updates to multiple test assets without ever traveling to the machines. Remote deployment means team members spend less time performing maintenance tasks and can instead be doing valuable work in other areas. Teams can also schedule these updates to automatically deploy at opportune times; for example, overnight or during a low productivity season.

SystemLink does more than increase efficiency. It also ensures complete configuration compliance. After an update, the Software Configuration Module checks the status of all test equipment to verify that each asset has the latest software updates and is operating according to preset compliance standards. This tool reduces errors with software configuration and test data across your entire fleet of test systems. It also helps your team address issues before they have a greater impact on your organization.

How Homegrown Solutions Handle Data Management

Most large organizations today generate vast amounts of data that could solve huge problems and increase efficiency, if only it were properly managed and analyzed. Unfortunately, many companies lack a robust test data management system that can effectively handle large datasets.

Furthermore, most data held in homegrown solutions is saved in inconsistent formats, which may lack the ability to add metadata and descriptive tags or not be indexed and searchable. Combine this with the prevalence of departmental or office-specific data silos, and you are left with a graveyard of information which severely limits comparison between datasets and interdepartmental collaboration. If the person who developed the homegrown solution is no longer with the organization, as is often the case, long-term sustainability also becomes an issue.

Many homegrown solutions also require manual data input and store the data in Excel spreadsheets or database software. While teams may be able to store data this way, it’s unlikely a solution with these limitations will be scalable as the needs of the organization expand. In addition, it’s unlikely the data will remain transferable and compatible between different types of testing equipment. 

Combined, these limitations can make it nearly impossible to develop valuable insights from your testing and asset data. This problem may in turn delay decision making, process improvement, and production across departments.

The SystemLink Approach to Data Management and Accessibility

Data is useless unless you can analyze it effectively. While big data from test systems is a challenge to manage and analyze with homegrown solutions, SystemLink is built for big data. In addition to cleaning and preparing complex datasets, SystemLink streamlines searching and transferring data for analysis and is scalable to suit any testing operation.

While homegrown solutions can be built on commercial databases, such as SQL Server, and visualization tools like Tableau, this approach requires significant development efforts to support test-specific analytics. SystemLink provides out-of-the-box data management and analytics specific to test organizations.

Using powerful tools to automate data acquisition and entry, test data can be automatically stored in a central repository immediately after a test cycle is completed. The platform helps teams easily clean and validate datasets before analysis and make copies of those datasets for later use. Because both LabVIEW and TestStand work with SystemLink out of the box, you can set them up to automatically store test data, measurement data, and associated steps to the SystemLink server. The Test Module stores this data using PostgreSQL file format, which can easily be searched later using metadata and other custom properties—including product, serial number, system, operator, and other information. Waveform data can be stored in a three-tiered hierarchy with the TDMS file format. By breaking down each file by dataset, group, and channel, you can make your waveform data easier to understand and distinguish across all test equipment.

With the TDM DataFinder Module, teams can index all data for easy, universal access. Unlike traditional siloed systems, SystemLink enables different departments to access to the same datasets and insights according to the permissions you set. In addition, the DataFinder Module supports scaling for local and global searches. By linking multiple servers in different locations, teams can easily search for and access test and measurement data from anywhere on the network.

With the SystemLink Test Module, you can create custom dashboards with a variety of different visualizations to gain a quick overview of the most important metrics for your project, across any time period or component. From there, you can use test result queries to filter results based on your chosen criteria, view more specific details about individual tests, and compare results over time. This process allows for more effective root cause and first pass yield analysis. Users can also visualize test data using highly customizable interactive reports, making it easier to uncover trends and insights.

The Inefficiencies of Analyzing Data with Homegrown Solutions

The structure and design of a spreadsheet-based interface limits the ability to effectively analyze test and measurement data. Rather than providing a streamlined workflow to collect, clean, and analyze data, many teams are forced to use the native data management and analysis tools found within programs like Excel and Crystal Reports. While these simple reporting functions may be enough for some operations, it often takes a long time to reach valuable insights when analyzing data from large fleets of test systems.

Most spreadsheet-based solutions do not account for user error or uncalibrated test assets. Lack of error control wastes time analyzing bad data, ties up assets that could be used elsewhere, and leads to expensive retests. Similarly, because most homegrown solutions can’t automatically convert data from various sources into a common format, manual data preparation is likely to be time consuming and error-prone. Even then, some datasets are too large for these tools to handle at all.

In many homegrown solutions, the ability to share analysis scripts across users is limited. What’s more, triggering these analysis scripts is often a manual process, which not only creates a lack of consistency and introduces the potential for errors from recreating new analysis scripts, but it is also guaranteed to detract from time spent improving product designs or manufacturing processes.

For any analysis process to be efficient, you need a set of standard workflows that ensure consistency in data analysis. Otherwise, with limited analysis and visualization capabilities, mining for valuable engineering insights may prove to be impossible—or inefficient at best.

How SystemLink Shortens Time to Insight

Using the Analysis add-on through SystemLink’s TDM DataFinder module, teams can easily configure and analyze test data in a number of ways:

• While homegrown solutions are often limited to working with data from one source at a time, SystemLink can reconfigure and consolidate data from a variety of test equipment and multiple disparate sources.
• Rather than manually cleaning new test data, teams can automatically clean and validate data for analysis within SystemLink.
• Working closely with DIAdem, TestStand, Python, and .NET, teams can create, save, share, and reconfigure analysis scripts, ensuring consistent analysis across datasets. Teams can then run these scripts on any individual file or entire datasets with custom logic and generate PDF reports immediately after analysis.
• Teams can choose to execute analysis scripts manually, trigger them on specific criteria once new data is received, or run them automatically on a specific schedule. With the ability to automate analysis, teams can run scripts at optimal times, reducing time to insight and better utilizing their test equipment.

Beyond that, the reports possible with SystemLink more than rival the capabilities of homegrown solutions. With SystemLink, teams can analyze big data and uncover hidden opportunities for optimization.

Cree Lighting, a market leader in lighting products, provides a great example of how SystemLink can improve an existing organization at multiple levels. At the beginning of 2019, Cree Lighting faced challenges in managing and analyzing their test data that prevented them from maximizing their returns on their testing investments. To remain competitive, they needed a solution to help them gain greater visibility into their testing processes. After achieving 100% user adoption of SystemLink, Cree Lightning saw the following successes:

• Increased efficiency by repairing technological issues in their logistics process.
• Increased first-pass yield on their production floor by more than 120%.
• Reduced downtime by 13% as a direct result of implementing SystemLink.

When deciding what sort of test management system to implement, consider the potential return on investment for each option. By considering the features your team needs, the timeline to implementation, and maintenance requirements, you’ll be able to make the right choice for your organization.

The primary advantage of developing your own test operations solution is the ability to completely control the design of the features and user interface. However, keep in mind that building a robust test operations solution from scratch requires a heavy software development workload. If you don’t have the development resources necessary to achieve the needed functionality, pursuing a homegrown solution could be a costly mistake. Even if you have the development capabilities to produce your own test operations system, it will take many iterations to develop data management and analysis capabilities on par with SystemLink. Instead, you could benefit from NI’s decades of test systems experience, utilizing these functions immediately while maintaining the ability to customize the software to meet your organization’s evolving needs.

In any testing or production environment, time is critical. Unexpected delays could mean lengthy downtime and significant financial losses. Building a complex software solution like this is bound to take years. If you have plenty of time and resources to spend on developing a custom solution, you may prefer this option. However, beyond that, building a custom solution requires ongoing support, maintenance, feature updates, and security patches. If you can’t commit development resources beyond the initial build, creating your own software is not a good long-term solution. However, SystemLink is easy to maintain, and the team at NI can provide support and consistent updates throughout your subscription.

If your current solution is failing to uncover optimization opportunities or resulting in failure to meet production or quality standards, it is actively costing you money. It’s important to consider whether your organization can continue to live with a suboptimal solution while you develop a test operations system that can track test assets and manage system health, as well as store and analyze test data. During that time, you could have implemented SystemLink to improve efficiency, reduce time to insight, and secure a more immediate return on investment. 

No matter how powerful or well-designed your software solution, it can only impact your operation if it works well with your existing systems and is fully adopted. Even after you’re certain that a new solution will provide a positive long-term ROI, introducing new software into the company can be a challenge. Even after extensive training and onboarding, employees may resist a change to the status quo and feel uncomfortable learning a new system. Unless everyone on the team clearly understands how this new tool will improve workflows, you may encounter skepticism and resistance.

Because of the complexity of the various tools involved in test operations, it’s important to have an intuitive, unified dashboard from which you can manage all aspects of your test operation. This unified experience eases the learning curve and helps drive adoption. Furthermore, NI offers white-glove onboarding services to help take the burden of training off your shoulders and ensure everything is set up to suit the unique needs of your organization.

Hidden inside big data is the potential to help engineering teams improve products and enhance operational procedures through powerful insights. Unfortunately, the manual methods of test operations and data management most companies rely on are not only laborious, but also costly and prone to errors.

SystemLink, a comprehensive test data and asset management solution, streamlines data collection, analysis, system health monitoring, and maintenance workflows. It also provides increased visibility and control over test systems and unrivaled data analysis capabilities with reports that can be shared across your entire organization. If you want to help your organization shorten your time to insight and make better data-driven decisions, SystemLink is for you. 

Contact us today to learn how SystemLink can help you get the most out of your testing investments.