Automate Measurement Data Standardization, Enrichment, and Validation with SystemLink TDM Data Preparation Add-on

If we start with a raw data set that includes data from multiple sources we typically run into 4 problems:

• Multiple Sources and Formats 
• Different Metadata Names and Values
• Different Engineering Units
• Data Might Contain Errors

If these issues exist in the raw data, they ultimately result in inconsistent analysis for the following reasons.

• Consolidating File Formats Manually Leads to Errors

Multiple file formats and sources are used in a test setup for a variety of reasons, but is most often attributed to different groups within the company testing subsystems or components. These different groups could use different hardware vendors or different processes. Unless an automated process is used, consolidating data from multiple file formats is a tedious and manual process that leaves room for possible human error and wasted time. 

• Can’t Find the Data 

Data querying tools, like SystemLink TDM DataFinder Module, can reduce the time and effort it takes to find the data you need to analyze. Creating a query requires metadata names and values to be used. If these metadata names and values don't match and extra queries aren't used to expand the search results, the data won't be found and analyzed.

• Analysis Results are Incorrect

Incorrect engineering units and erroneous raw data can lead to drastic changes in the results of analysis procedures resulting in invalid results. Different engineering units can be a serious issue for teams located in different sites where a standardized unit hasn't been agreed upon. The problem also tends to happen when testing is completed by other groups outside of the company. Sensor failures, data corruption, or human error can also have the same effect. These problems are typically only realized after a critical error occurs. 

The Data Preprocessor, included with the SystemLink Data Preparation Add-on, automatically standardizes, enriches, and validates data before using it for other purposes. By using the Data Preprocessor, you eliminate time wasted upfront in preparing your data files and save yourself from making costlier mistakes further downstream.

The Data Preprocessor has six configurable steps:

• Metadata Names—Map metadata, or property, names that have slight variations to a single, standardized name. For example, channel names “Engine,” “Engine Type,” and “Fuel” can be mapped to “Engine_Fuel_Type.”
• Metadata Values—Similar to the step above, map property values with slight differences in representation to a single, standardized representation. For example, values “Diesel,” “D,” and “Fuel_Diesel” can be mapped to “Diesel.”
• Engineering Units—Automatically convert measurements to standardized units using a comprehensive, built-in unit library. For example, convert displacement measurements in feet and inches to meters.
• Signal Statistics—Calculate characteristic values for the data and store as properties. For example, calculate mean for one channel of measurement data.
• Final Validation—Check that actual data is within an expected range. For example, if 1,000 data values are expected but only 250 are present in the file, exclude this data and flag for error condition.
• Store Data—Store the standardized, enriched, and validated data as a “master” copy in another disk location on the server machine or over the network. This “master” copy is saved into a high-performance, low-footprint file format for optimized loading and storage. The original data files are never deleted or modified.

The 64-bit versions of DIAdem Advanced and Professional provide the interface to configure the above Data Preprocessor steps. At the end of the configuration, DIAdem saves a configuration file (*.DPP), which you upload to the DataFinder Server using the Server Manager web client. Afterward, the Server Manager becomes your interface for starting and stopping the preprocessor, as well as pointing it to the location of raw data to process.

The 64-bit versions of DIAdem Advanced and Professional provide the interface to configure the above Data Preprocessor steps. At the end of the configuration, DIAdem saves a configuration file (*.DPP), which you upload to the SystemLink Server using the SystemLink web client. Afterward, the web client becomes your interface for starting and stopping the preprocessor, as well as pointing it to the location of raw data to process.

The SystemLink Analysis add-on  makes it possible for you to extract the maximum value from large amounts of data in minimal time by harnessing the power of server technology to analyze large amounts of data and generate reports. When used together, SystemLink Data Preparation Add-on ensures data is standardized and verified with a Data Preprocessor before analysis. The SystemLink TDM DataFinder Module, the core of the SystemLink measurement data management solution, builds and maintains an automated index of your data that enables the SystemLink Analysis Add-on to perform complex search queries on data distributed throughout the organization for automatically triggered analysis routines.

^{Figure 3: SystemLink TDM DataFinder Module and Add-ons work together to create a fully automated data management workflow.} 

SystemLink delivers measurable improvements in operational efficiency and productivity by providing you with a centralized web-based management interface for connected devices, software, and data. Although aligned with NI products such as LabVIEW, TestStand, and hardware systems, SystemLink also offers an open architecture for incorporating a wide range of third-party software and hardware technologies. SystemLink TDM DataFinder Module and Add-ons are components of the SystemLink product family. To learn more about the other capabilities of SystemLink, visit ni.com/systemlink.

• Contact Sales to learn more about SystemLink TDM DataFinder Module and TDM Data Preparation Add-on for your application.
  
• Download an evaluation of SystemLink TDM DataFinder Module and Add-ons
• Learn more about the SystemLink TDM DataFinder Module and Add-ons
• Learn more about DIAdem