Best Practices for Target File IO with LabVIEW Real-Time

There are reasons to avoid file input and output (IO) on the real-time (RT) target in a LabVIEW RT application.

• File IO is non-deterministic; it should never be used in a time-critical process. The jitter introduced by file IO is often on the order of milliseconds, but it’s technically unbounded.
• File IO is a shared resource and introduces the risk of priority inversion. When the system grants control of the disk system to a thread, it protects the system with a mutex (an inter-thread semaphore). If a time-critical thread requests the disk while it’s protected, LabVIEW temporarily promotes the thread that owns the mutex to time-critical priority to release the mutex as soon as possible.
• Disk operations are frequently the speed bottleneck in an RT application. Reading and writing to disk is typically orders of magnitude slower than reading and writing the same data to RAM.
• Embedded systems typically have finite disk resources and are deployed for extended periods of time. Disk writes on the target consume these resources and may limit the time the program can run.
• FAT32 file systems are not fault-safe. If the RT system disk is FAT32, file IO to the disk has the potential to corrupt the disk if power is lost during IO. Some National Instruments real-time targets use the Reliance file system that helps guard against file system corruption.
• Some RT targets use flash memory for disk storage, and flash memory has erase-write lifecycle limitations. You can maximize the lifespan of your embedded system by writing to it less often.

This document presents decisions and methods that can be used to eliminate and reduce file IO in an RT application. The document is divided into four focuses: Evaluate, Offload, Reduce and Optimize.

It may be possible to reduce file IO by thoroughly evaluating the need for the data. Eliminate disk operations that aren’t truly necessary. The following questions may help ascertain the need for archived data.

• Is there a system requirement for archived data?
  • What are minimum data sets that must be archived?
  • Are individual samples necessary or will trends/averages suffice?
  • Is it reasonable to archive alarm conditions only?
  • Is it reasonable to discard outliers?
• Does data really need to span system restarts?
• Does the quantity of data exceed the amount of RAM available for data storage?
  • Can the RT system post-process the data without storing it to disk?
• Is file IO used to pass values or tokens within an application?
• What would happen if the archived data were lost?

Scrutinize each disk operation to identify unnecessary file IO that can be removed.

Determine which system is best-suited to perform the file IO.

Offload to Host

If writing data to disk is necessary, the host system may be better suited to perform the read/write operation.

• There is typically much more disk space available on the host.
• Access to OS-specific technologies.
  • ActiveX
  • NET
  • 3rd party applications
  • Standardized database engines
• Saves processor time and code space on the target.
• IDE and SATA drives typically do not have a finite number of read/write cycles.

There are seven network communication methods outlined by LabVIEW 2009 Real-Time Module Help: Exploring Remote Communication Methods (RT). These methods may be used to transfer data from the real-time target to the host computer where it can be logged. Refer to the help file for more information each method.

1. Network-Published Shared Variable
2. Time-Triggered Shared Variable
3. TCP
4. UDP
5. VI Server
6. SMTP
7. Logos

There are also APIs based on the above methods that may simplify the transfer overhead. An example is the Simple Messaging Reference Library (STM).

File IO on the Target

There are, however, caveats to offloading data from the target to the host.

• Requires a reliable network connection.
• Network connection may be a bottleneck.

Consider the following points when data must be archived on the real-time target.

• Only log data to the target system when the network connection fails. Use the communication method’s error codes to determine if the network timed out.
• Always perform file IO in non-deterministic threads.
  • A rule of thumb is that the complete application should consume less than 80% of the CPU resources to avoid starvation.
• Consider writing to a non-system disk.
  • Ideally, use a separate physical IDE/SATA drive rather than a non-system partition.
  • USB disks may be used for data storage but should be considered carefully due to possible jitter.
  • Consider the NI 9802 Secure Digital Removable Storage Module for CompactRIO.

Consider techniques to reduce the amount of data that must be archived. The amount of reduction possible depends on the system requirements.

Loop Rate

Code that runs faster than necessary reduces the resources available for future expansion and may generate superfluous data.

• Run loops only as fast as necessary to fulfill the system requirements.
• Provide the appropriate execution timing to allow other processes to execute.
  • Consider allowing additional CPU time for future expansion.
  • See LabVIEW 2009 Real-Time Module Help: Timing Deterministic Applications (RT Module) for more information.
• Consider the rates in the entire system.
  • For example, a cRIO 9074 reading one channel of thermocouple (TC) data from a 9211 module through the default scan engine. The RT code can execute in a few microseconds, the scan engine updates every 10ms, but the 9211 has a scan rate of 14 scans/second. Archiving TC samples to disk faster than 14Hz results in duplicate data.

Data Size and Type

When a small amount of data is used to represent a data point, less data must be written to disk to archive the point.

• Consider archiving RAW data instead of calibrated and scaled data.
  • Raw data is typically 12, 16, 20 or 32-bits per sample.
  • Scaled and calibrated data is typically 64 bits.
  • Properly packing and writing raw data may require 2-4 times less disk access (see Write Less Often in the Optimization section of this document).
• Consider the actual data resolution.
  • All digitizers have a finite resolution.
  • Use the smallest numeric representation possible.

File Type

You can choose to write to several types of files from LabVIEW.

• Binary*
• Datalog*
• TDMS*
• Text
• Spreadsheet
• Configuration
• XML

Items marked with an asterisk write data in binary format. This data is not human readable but often consumes less space on disk. The other formats are human readable, but often require more space on disk.

For example, a DBL uses 64 bits in a Binary file. The same DBL uses 8 bits per digit in a Text file (up to 120 bits depending on the number). See LabVIEW 2009 Help: Numeric Data Types Table for more information.

Use a binary file format to reduce read/write time on the real-time system.

Decimate

Reduce the number of data points archived to disk. There are built-in LabVIEW functions to decimate data.

• Sample Compression
• Decimate 1D Array
• Decimate (single shot)
• Decimate (continuous)

You can also build your own decimation algorithms. The decimation factors in the following examples are arbitrary. The following considerations are made.

• Data is always processed.
• Data is not always archived to disk.
• Data history – how do we know how much data was not logged?
  • In these examples, there is a constant loop rate and a file timestamp.
  • A sample counter could have been used.
• How will the next file-write be determined?
  • We apply the algorithm to the last point written to file.
  • The algorithm could be applied to a constant set point.

Decimate by Sample

Log every N’th sample (Figure 1). A simple rollover counter can be used to log every N’th sample. None of the samples between N, 2N, 3N… are logged to file.

Figure 1

Averaging N Samples

Log the average of the last N points to file (Figure 2). An additional Shift Register or Feedback Node is used to store the running sum, and the average is calculated and written to disk.

Figure 2

Decimate by Time

Log one point of data every M units of time (Figure 3). When configured to reset, the Elapsed Time Express VI is suited for this task. None of the samples between t = M, 2M, 3M… are logged to file.

Figure 3

Average Samples over Time

Average samples over time (Figure 4). One Shift Register/Feedback Node is used to store the running sum and another is used to store the number of iterations since the last file write. The average is calculated and written to disk.

Figure 4

Delta Filter

A strict delta filter logs data only if the value changes (Figure 5).

Figure 5

Absolute Deadband

Extend delta filtering to absolute deadbanding. An offset from the current data point must be exceeded to qualify for a disk write (Figure 6). The In Range and Coerce function is suited for the comparison.

Figure 6

Absolute deadbanding can be modified to implement alarming and bandpass filtering. The min and max values are provided to the In Range and Coerce function to identify the window.

Percentage Deadband

Rather than use an offset to define the deadband, it may be more meaningful to define the deadband as a percent change (Figure 7). For example, if the new value is within 5% of the last value written, do not write the value to disk.

Figure 7

Statistical Profile

The nature of the data and system requirements will determine what statistical parameters are of interest. There are built-in LabVIEW functions to calculate statistics.

• Min & Max Function
• Statistics Express VI

A simple data set may include the following.

• Number of samples
• MIN
• MAX
• Average
• RMS

Combination Algorithms
Using these simple building blocks, a robust algorithm may be implemented. Consider the following combination.

1. Use a Deadbanding algorithm to determine if a threshold has been exceeded.
2. Use Statistical Profiling to build up a picture of the data over time.
3. If the deadband has not been exceeded, write and reset the statistical profile every hour.
4. If the deadband has been exceeded, write and reset the statistical profile as well as the offending data point.

Compression Algorithms

Reduce the size of the data archived to disk.

Delta Compression

Write an initial value and delta (either as a scalar value or percentage) to disk. A smaller data type may be used to represent the delta than the data point itself. This technique is mostly applicable to continuous phenomena (temperature in an incubator for example).

Data Specific Compression

There are a variety of compression algorithms that have been written for specific data sets (audio, video, image, etc.). These algorithms may be used to reduce the amount of data that gets written to disk. A decompression algorithm is used to extract the data when it’s needed. These algorithms are beyond the scope of this document, but an Internet search for “Compression Algorithms” will get you started.

There is also a 3rd party API called OpenG Zip Tools that can be used to compress files on LabVIEW RT targets. This API is not supported by National Instruments.

Optimization techniques are aimed at making file IO more efficient on the embedded system.

Write Less Often

Store data in RAM to buffer disk operations rather than writing each data point (Figure 8).

Figure 8

Buffering data in RAM can also be used to store data on the RT system for post-processing without ever writing to disk. Add more RAM to the system to increase the RAM buffer.

Write According to Disk Parameters

The diagram in Figure 8 can be used to optimize the disk operation to the sector size of the disk. See DeveloperZone Tutorial: Optimizing File I/O in LabVIEW and LabVIEW RT for more information.

If file IO is necessary in a real-time application, a robust process can be derived from the following chart.

Benchmark Project.lvproj. (2009). LabVIEW 2009 > Help > Find Examples > Browse by Task > Toolkits and Modules > Real-Time > Benchmarking.

Attribution 3.0. (2009, September 5). Oglib lvzip. Retrieved October 21, 2009, from OpenG > Viewing Oglib lvzip: https://www.vipm.io/package/oglib_lvzip/

Datalight Incorporated. (2009). High-Integrity Transactional File System with Dynamic Transaction Point™ Technology. Retrieved October 21, 2009, from Datalight Home > Products > File Systems > Reliance Family > Reliance: https://www.tuxera.com/products/

National Instruments. (2009, June 1). Decimate (single shot) VI . Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/vi-lib/analysis/2dsp-llb/decimate-single-shot-vi.html

National Instruments. (2009, June 1). Decimate 1D Array Function . Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/functions/decimate-1d-array.html

National Instruments. (2009, June 1). Exploring Remote Communication Methods (RT Module). Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Real-Time Module Help: https://www.ni.com/docs/en-US/bundle/370622r/page/download.html

National Instruments. (2009, June 1). File IO VIs and Functions. Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/menus/categories/programming/file-mnu.html

National Instruments. (2005, December 21). Is DAQmx Raw Data Calibrated and/or Scaled? Retrieved October 21, 2009, from NI Home > Support > KnowledgeBase: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019YLoSAM&l=en-US

National Instruments. (2007, August 1). LabVIEW Execution Trace Toolkit 2.0 Help. Retrieved October 22, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW Execution Trace Toolkit 2.0 Help :
 https://www.ni.com/docs/en-US/bundle/370622r/page/download.html

National Instruments. (2009, June 1). Max & Min Function . Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help:  https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/functions/max-min.html

National Instruments. (2009, June 1). Numeric Data Types Table. Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview/page/numeric-data-types-table.html

National Instruments. (2006, September 6). Optimizing File I/O in LabVIEW and LabVIEW RT. Retrieved October 21, 2009, from NI DeveloperZone > Tutorial. https://www.ni.com/en/support/documentation/supplemental/09/best-practices-for-target-file-io-with-labview-real-time.html

National Instruments. (2009, June 1). Sample Compression Express VI. Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/vi-lib/express/express-signal-manip/samplecompressionblock-llb/ex-inst-sample-compression-vi.html

National Instruments. (2009, April 22). Simple Messaging Reference Library (STM). Retrieved October 21, 2009, from NI Developer Zone > Example Program: https://www.ni.com/en/support/documentation/supplemental/21/simple-messaging-stm-communication-library-for-labview.html

National Instruments. (2009, June 1). Statistics Express VI. Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Help: https://www.ni.com/docs/en-US/bundle/labview-api-ref/page/vi-lib/express/express-analysis/statisticsblock-llb/ex-inst-statistics-vi.html

National Instruments. (2009, June 1). Timing Deterministic Applications (RT Module). Retrieved October 21, 2009, from NI Home > Support > Product Reference > Manuals > LabVIEW 2009 Real-Time Module Help:  https://www.ni.com/docs/en-US/bundle/370622r/page/download.html