Archived: Debugging Multicore Applications with the Real-Time Trace Viewer

Multithreading has been natively supported by NI LabVIEW since the mid 1990s. With the introduction of multicore CPUs, developers can now take full advantage of the capabilities of this new technology with the help of LabVIEW. Parallel programming poses new challenges when developing applications for multicore CPUs such as synchronizing concurrent access to shared memory by multiple threads, and processor affinity. LabVIEW handles most multithreading tasks automatically while giving flexibility to users to assign threads to CPUs (or cores of the same CPU) of their choice. The LabVIEW 2014 Real-Time Module includes support for CPUs with up to 12 cores on Phar Lap ETS targets. Previously, the module supported only eight CPU cores even if the target had more available. 

If you are developing a real-time application, the best way to monitor detailed CPU usage and other events is by capturing execution traces from real-time targets. With the help of the Real-Time Trace Viewer you can view and analyze the execution traces of real-time tasks including Virtual Instruments (VIs) and Operating System (OS) threads. The Real-Time Trace Viewer consists of two parts: the Instrumentation VIs and the Trace Viewing Utility. The Instrumentation VIs need to be added around the code whose execution you would like to trace. The Trace Viewing Utility is used to view the captured execution trace. In 2013 and prior releases of the LabVIEW Real-Time Module and LabWindows/CVI Real-Time Module the Real-Time Trace Viewer is packaged as a separate toolkit (Real-Time Execution Trace Toolkit).

Accessing low level execution information is critical for optimizing and debugging real-time applications because you can easily identify sources of jitter such as processor affinity, memory allocations, priority inheritance, or race conditions.

Using shared resources by multiple threads of different priority level can lead to unexpected behavior of the application. Table 1 shows some shared resources and the potential problems that may appear in an application.

Table 1.Shared resources and related potential problems.

In the next two paragraphs we will discuss how you can assign and trace processor affinity and how we can debug potential problems with shared resources.

The following two screenshots represent execution traces of a program which was run on an embedded multicore real-time target. Parallel programming in NI LabVIEW was implemented by using two Timed Loop (TL) structures. Each of the TLs was assigned to a different CPU. All code within a given TL will execute on the same CPU. Processor affinity is preserved until the program completes execution. Figures 1 and 2 clearly show the CPU affinity of each thread. Threads associated with a given CPU are highlighted while the rest of the threads are grayed out. Also note the parallel execution of threads running on separate CPUs.

Figure 1. This execution trace shows threads associated with CPU 0. The rest of the threads are grayed out.

Figure 2. This execution trace shows threads associated with CPU 1.

See also:
Multitasking in LabVIEW

Using shared resources by a time-critical thread (a thread that needs to execute within a deterministic amount of time) can introduce extra jitter to your real-time application. One such shared resource is the LabVIEW Memory Manager. It is responsible for dynamically allocating memory.

When a normal priority program is in possession of the Memory Manager, the rest of the threads, including the time-critical thread, must wait for the shared resource to become available. In such cases jitter is inevitably introduced in the time-critical thread. To resolve this issue, the thread scheduler temporary boosts the normal-priority application to run in the time-critical thread so that it can finish up quicker and release the Memory Manager. This phenomenon is known as priority inheritance or priority inversion. To avoid such situations NI advises to avoid using shared resources. A solution in the case of the Memory Manager would be to preallocate memory for arrays.

Priority inheritance can be sometimes enhanced when mixing two priority scheduling schemes – at the VI level and at the Timed Loop level. For example, in Figure 3 we can see a time-critical subVI (red icon) and a Timed Loop fighting for the same memory resources. The race for shared resources between the time-critical subVI and the Timed Loop can lead to priority inheritance.

Figure 3. Example of using two different priority assignment schemes.

Figure 4 is the trace of another program which has a normal priority subVI and a time-critical subVI sharing a common resource – the Memory Manager. The green flags show dynamic memory allocation, and the orange flag shows priority inheritance. The execution of the time-critical thread was interrupted so that the normal priority subVI can be boosted up to run in the time-critical thread, and therefore release the shared resource quicker. This eventually affects the determinism of the time-critical subVI.

Figure 4. The green flags show dynamic memory allocation(accessing the Memory Manager). 

See also:
Avoiding Contiguous Memory Conflicts (RT Module)

• Debug run-time problems in single processor and multiprocessor applications
• Identify shared resources and memory allocation
• Verify expected timing behavior
• Monitor CPU utilization and multicore interaction
• Learn about the LabVIEW execution model

Along with the Real-Time Trace Viewer we strongly advise the use of other standard debugging tools to assure expected application performance.

VI-level debugging tools

• Probes, breakpoints, execution highlighting
• RT Debug String VI
• User Code

System-level debugging tools

• NI MAX or Distributed System Manager (On-Screen CPU Utilization)
• VI Profiler