Resolving SystemLink Client Failures to Forward Data When Store and Forward is Enabled

The SystemLink Client Store and Forward mechanism caches data on the local disc of the client between when it is acquired and when it is sent to the server. This mechanism is enabled in TestStand and is primarily used in the following ways:

• Store and Forward (Only When Disconnected from Server) – if the client is unable to reach the server, store data and forward when a connection is re-established
• Store and Forward (Always) - always store data on the client and send it later through the forwarding service to improve throughput and performance of test execution

NI has identified issues that fall into the following four categories:

• Incompatibilities between newer versions of SystemLink Client and older versions of SystemLink Server - NI does not support backwards compatibility of SystemLink Client with SystemLink Server
• Processing transactions out of order due to timestamp conflicts
• Forwarding service entering bad state after failure to forward transactions
• Failures to attach reports and files to test results

The primary user impact is incomplete data upload to the SystemLink Server. Missing data on the SystemLink Server can impact accuracy of reporting, data analysis and visualization. 

For example, a user may observe: 

• Test results uploaded to SystemLink Server but have missing steps, file attachments or reports 
• Test results and steps that were executed but not uploaded to SystemLink Server 
• Test results uploaded to SystemLink Server that completed but status reads Timed Out 

Data that has not been forwarded to the server is cached on the local disc of the client and is largely recoverable following the actions required by this notification. Transactions that previously failed to forward and have been quarantined may require additional intervention to recover. Please contact NI support if you require assistance recovering quarantined transactions.

Please contact NI support if you are uncertain how to proceed, would like to consult with an NI Technical Support Engineer regarding fix deployment, SystemLink Server upgrade or any other step of the process.

After completing the Action Required sections of this document, if you are still experiencing incorrect behaviors with Store and Forward enabled and/or backlogged data has not been cleared from patched SystemLink Clients, please contact NI Support for further assistance. 

To determine if a SystemLink Client is impacted by these issues complete the following checks on each SystemLink Client.

Determine if Store and Forward is Enabled from TestStand

Complete these steps on each SystemLink Client to determine if it is impacted by the store and forward issues.

1. Open TestStand on the SystemLink Client.
2. Select Configure > Result Processing.
   
3. For NI SystemLink Test Monitor Client, select Options .
4. Note whether Enable Store and Forward is checked. If it is and Always or Only When Disconnected is also checked, this client is impacted.

If none of your SystemLink Clients are impacted, no further action is required.

(Optional) Determine if Data Is Backlogged on the SystemLink Client

Complete these steps to determine if data is backlogged on the SystemLink Clients. It is not necessary to complete this check on every client. After deploying updates in the SystemLink Client Updates section of this document, you will have visibility into this on each client.

1. Open a Windows File Explorer on the SystemLink Client.
2. Navigate to C:\ProgramData\National Instruments\Skyline\Data\Store.
3. Explore each of the folders in this directory.
   1. If there are files in these directories other than _CACHE_ and _ID_ it is because the client has data cached that has not been forwarded.

4. Explore the quarantine folder under each directory.
   1. If there are files in these directories it is because the client has data quarantined that failed to forward.

Note: If you would like to have a complete picture of how much data is backlogged on all clients before taking further action, you may choose to install the Store and Forward Salt Beacons now. See "SystemLink Client Updates" for installation instructions. Use a Package Repository feed and System State to deploy the beacons ONLY at this time.

Addressing these issues may require multiple steps. Carefully read and identify the required steps for your SystemLink deployment as outlined in this section.

Identify SystemLink Clients That are Not Version Compatible with the SystemLink Server

It is possible that some of your SystemLink Clients may not be version compatible with your SystemLink Server. SystemLink Clients ARE NOT backward compatible with SystemLink Server. The SystemLink Client version must be less than or equal to that of the SystemLink Server version.

1. Open NI Package Manager on the SystemLink Server.
2. Select the Installed tab, search for “NI SystemLink Server” and note the version installed.
3. Open the SystemLink web application.
4. From the navigation tree, select Systems Management > Systems.
5. Add a column to the grid for the SystemLink Version property.
   1. Click the Edit Grid  button.
   2. In the Edit Grid slideout, click + Add.
   3. Add a property column.
   4. Using the dropdown selector choose SystemLink Version.
   5. Save the grid view to use in a later step.

6. Observe the SystemLink Client version installed on each system and record it in the table below.
7. Take note of any SystemLink Clients that are versioned higher than the SystemLink Server.

Installed SystemLink Server Version: ________________________________________________

NI recommends one of two approaches for resolving these incompatibilities:

1. Upgrade the SystemLink Server to a version at least as high as the highest SystemLink Client version
   • Preferred if many clients are not version compatible with the server
   • Note: Additional considerations on version selection are included in the SystemLink Server Upgrade section of this document.
2. Re-image incompatible Clients and install a version compatible software set
   • Preferred if a small number of clients are not version compatible with the server
   • Prior to reimaging a client, zip and archive the store directory  C:\ProgramData\National Instruments\Skyline\Data\Store to a safe location off the client
     • After reimaging and reinstalling software, unzip the archived store directory back into the original path on the client

Note: Downgrading NI software through NI Package Manager is NOT supported.

Determine if a SystemLink Server Upgrade is Required

There are two factors in determining if a SystemLink Server upgrade is required.

1. Is a server upgrade required to resolve client incompatibility? (Refer to the previous section.)
2. Will applying the SystemLink Client patches necessitate a server upgrade?

All affected SystemLink Clients will require a patch and may require a version upgrade to do so. NI is providing a patched version of the following SystemLink Clients:

• SystemLink Client 2022 Q1 (21.5.2)
• SystemLink Client 2021 R3 (21.3.3)
• SystemLink Client 2020 R4.4

Record the intended patch version for each client in the table above. You may have clients with different versions so long as the version of each is less than or equal to that of the server.

If you are upgrading or patching your SystemLink Clients and that will result in any SystemLink Clients being versioned higher than the currently installed version of SystemLink Server, a SystemLink Server upgrade is also required.

Is a SystemLink Server Upgrade required? YES / NO

What SystemLink Server version will you upgrade to? __________________________________

If you determined above that a SystemLink Server upgrade is required, complete the steps in this section of the document. If you determined that a SystemLink Server upgrade is not required, skip to SystemLink Client Updates. 

NOTE: Upgrading SystemLink Server typically requires a minimum of 1-2 hours of Server Downtime. During this time period, clients will be unable to connect to or publish data to the server.

NOTE: Depending on which version of SystemLink Server you currently have installed and which version you are upgrading to, additional downtime (sometimes significant) may be incurred due to data migrations triggered during the upgrade process. (Ex. Upgrading to SystemLink Server 2022 Q1 triggers a data migration for the Test Monitor which can take days – weeks to execute on very large databases).

There are two approaches for executing a SystemLink Server upgrade. Please read this section carefully and determine which path is appropriate for your deployment.

Redundant Server Upgrade

For mission critical deployments, NI recommends executing a redundant server upgrade. This process involves backing up the SystemLink Server, migrating data, provisioning a mirror server with the new SystemLink Server version and executing a seamless switchover.

Follow the upgrade instructions in the SystemLink Operations Handbook and upgrade the SystemLink Server to the version you identified in Establish a Plan.

In-Place Server Upgrade

For deployments where downtime is less of a concern, it is possible to execute an in-place upgrade of the server. This approach exposes you to additional downtime for recovery in the event of a problem during upgrade.

If you choose to execute an in-place SystemLink Server upgrade, follow the instructions below:

1. Shutdown the NI Web Server.
   1. Open NI Web Server Configuration on the SystemLink Server.
   2. Select the Control tab.
   3. Select the radio button Disable the web server.
   4. Click the Apply button.
   5. Confirm the Server Status reads “the web server is disabled”.
   6. Confirm a banner is present that reads “The web server is disabled and will not start automatically.”
       
2. Stop the SystemLink Service Manager.
   1. From the start menu, open Services.
   2. Find “NI SystemLink Service Manager” in the services list.
   3. Right click and select Stop.
       
3. Backup your SystemLink Server.
   1. This process varies depending on how the server is hosted (physical machine, VM, cloud hosted, etc.) Consult with your system administrator or NI support.
       
4. Upgrade SystemLink Server.
   1. Open NI Package Manager on the SystemLink Server
   2. Install the SystemLink Server version you identified in Establish a Plan, and any add-ons you require.
      Wait for the installation to complete and reboot when prompted.
       
5. Server Bring-Up.
   1. Launch NILM or VLM and confirm the products are licensed.
      1. Re-activation may be required. Resolve any unlicensed products. Contact NI Support if you need assistance with licensing.
   2. Restart the NI Web Server.
      1. Open NI Web Server Configuration on the SystemLink Server
      2. Select the Control tab.
      3. Select the radio button for Enable the web server.
      4. Click the Apply and Restart button.
   3. Ensure all SystemLink Services are running, and no issues are detected.
      1. Open NI SystemLink Server Configuration on the SystemLink Server.
      2. Confirm 0 issues are detected on the Troubleshooting tab.
      3. Confirm all services are running on the NI SystemLink Service Manager tab.
   4. Check Client Connection and Activation Status
      1. Launch the SystemLink web application.
      2. From the navigation menu, select Systems Management > Systems.
      3. Confirm all clients are activated and have re-stablished connection to the server.
           1. Resolve any clients that are not activated.
           2. Resolve any clients that are not fully connected.  

Each SystemLink Client that is impacted requires an upgrade to the patched version of SystemLink Client you identified in Establish a Plan.

There are two approaches to applying SystemLink Client updates:

1. Manually apply updates to each SystemLink Client
2. Use SystemLink Systems Management tools to push updates to SystemLink Clients in batches

NOTE: NI recommends testing the following updates on a single SystemLink Client prior to rolling out updates to your entire fleet.

NOTE: NI has developed a collection of Salt Beacons to improve visibility into the health of the Store & Forward mechanism on SystemLink Clients. These beacons monitor the Store & Forward directories and publish information to the SystemLink Server as tags. NI recommends installing the beacons on any SystemLink Client that has Store and Forward enabled (as described in SystemLink Client Updates).

Manually Update an Individual SystemLink Client

To manually update a single SystemLink Client, complete the following procedures:

1. Install the Store and Forward Beacons.
   1. On the SystemLink Client, download the package “ni-systemlink-storeandforward-beacon_21.0.0.0_windows_all.nipkg” for the Store and Forward Beacons.
   2. From Windows File Explorer, right click on the package and select Install. This will launch NI Package Manager.
   3. Follow the prompts in the NI Package Manager dialog to start the installation.
   4. Wait for the installation to complete.
   5. Reboot the SystemLink Client.
   6. Open the SystemLink web application.
   7. From the navigation tree, select Utilities > Tags.
   8. Filter for “*.TestMonitor.StoreandForward.*” to view the tags.
   9. Take note of the volume of files / transactions currently pending or in quarantine.
       
2. Install the SystemLink Client Patch.
   1. On the SystemLink Client, open NI Package Manager.
   2. Search for “SystemLink Client” and select the store item.
   3. Select the patched version determined in Establish a Plan for this particular client.
   4. Click the Install button and follow the prompts in the NI Package Manager dialog to start the installation.
   5. Wait for the installation to complete and reboot when prompted.
   6. Open the SystemLink web application.
   7. From the navigation tree, select Utilities > Tags.
   8. Filter for “*.TestMonitor.StoreandForward.*” to view the tags.
   9. Confirm the number of pending files is reducing as the tags are updated.

Push Updates to SystemLink Clients Using Systems Management Tools

Using SystemLink’s Systems Management tools, you can streamline deployment of fixes and update as many Clients as you like in a batch.

NOTE: You will need to create a separate Feed and System State for each distinct SystemLink Client version you want to apply and each workspace the feed and state needs to be available to. NI does not recommend putting multiple versions of the SystemLink Client into the same feed.

1. Create a Feed and Add Packages.
   1. Open the SystemLink web application.
   2. From the navigation tree select Systems Management > Package Repository.
   3. Click the Replicate button.
   4. Select Replicate ni.com Download.
   5. Search for “SystemLink Client” + the version determined in Establish a Plan.
   6. Select the SystemLink Client version and click Next.
   7. Give the feed a distinct name, assign it to a workspace and give it a clear description.
   8. Click Replicate and wait for the replication to complete.
   9. Download the package “ni-systemlink-storeandforward-beacon_21.0.0.0_windows_all.nipkg” for the Store and Forward Beacons.
   10. From the Package Repository click the Feed Settings button.
   11. In the Packages section click the Add button.
   12. Select Upload Packages.
   13. Click the Browse button and select the package “ni-systemlink-storeandforward-beacon_21.0.0.0_windows_all.nipkg”.
   14. Click the Back button.
   15. Confirm the ni-systemlink-storeandforward-beacon package is in the package list.
        
2. Create a System State.
   1. Open the SystemLink web application.
   2. From the navigation tree select Systems Management > States.
   3. Click the Create button.
   4. Give the state a distinct name, description and assign it to a workspace.
   5. In the Feeds section, click the Add button.
   6. Select Add from Package Repository.
   7. Select the feed you created in the last section.
   8. Click the Add button.
   9. In the Packages section, Click the Add button.
   10. Uncheck Products Only.
   11. Select all the packages from the feed.
   12. Click the Add button.
   13. In the Packages section, ensure the Install Recommends checkbox is checked.
   14. At the top of the page, click the Create button.
   15. Add a system restart command to the system state.
       1. In the menu bar, select the checkbox for Show Raw State.
       2. Add the following lines at the end of the Raw View
          reboot:
          system.reboot:
          - message: 'System is rebooting now'
          - timeout: 10
          - in_seconds: true
          - only_on_pending_reboot: False
       3. In the menu bar click Save, enter a change description and click the Save button on the dialog.
3. 
   
4. Deploy the System State.
   1. Open the SystemLink web application.
   2. From the navigation tree select Systems Management > Systems.
   3. In the grid, select all clients to which you would like to deploy the System State.
   4. Click the Software button in the menu bar.
   5. In the left pane, select the States tab.
   6. Find the state you would like to deploy and select Install.
   7. Click the Next button.
   8. On the next page, click the Apply button.
   9. Wait for the System State to finish deploying and updating the clients.
   10. Confirm that the systems have been updated to the desired version of SystemLink Client using the previously saved Systems grid view.
       1. Click the view dropdown selector on the Systems Grid menu bar.
       2. Choose the previously saved Systems Grid view.
   11. Confirm that the Store and Forward beacons are publishing tags and backlogged data is being processed.
       1. Open the SystemLink web application.
       2. From the navigation tree, select Utilities > Tags.
       3. Filter for “*.TestMonitor.StoreandForward.*” to view the tags.
       4. Confirm the number of pending files is reducing as the tags are updated.

Recovering Backlogged Data

Using the salt beacons & corresponding tags installed in SystemLink Client Updates, you can monitor the Store and Forward directories on your clients from the SystemLink Web Application. After the SystemLink Clients have beenupgraded, the forwarding service should begin processing any data and files that are backlogged in the store.

Monitor the Store and Forward beacon tags and consider setting up a dashboard and alarms for better visibility. Depending on the volume of data, it may take anywhere from several minutes to several days for the forwarding service to process the backlogged transactions. Clients are in a healthy state when there are no files left in the store and quarantine directories.

Note: Files in the quarantine will not be automatically recovered. If you are not concerned about recovering the previously failed transactions, you may delete the files in the client's quarantine directories. If you would like to recover transactions that previously failed to forward, please contact NI support.

• Storing and Forwarding Test Data