3D Imaging with NI LabVIEW

There are several ways to calculate depth information using 2D camera sensors or other optical sensing technologies.  Below are brief descriptions of the most common approaches:

Starting with LabVIEW 2012, Vision Development Module now includes binocular stereo vision algorithms to calculate depth information from multiple cameras.  By using calibration information between two cameras, the new algorithms can generate depth images, providing richer data to identify objects, detect defects, and guide robotic arms on how to move and respond.

A binocular stereo vision system uses exactly two cameras. Ideally, the two cameras are separated by a short distance and are mounted almost parallel to one another. In the example shown in Figure 1, a box spherical chocolates is used to demonstrate the benefits of 3D imaging for automated inspection.  After calibrating the two cameras to know the 3D spatial relationship, such as separation and tilt, two different images are acquired to locate potential defects in the chocolate.  Using the new 3D Stereo Vision algorithms in the Vision Development Module, the two images can be combined to calculate depth information and visualize a depth image.

^{Figure 1:  Example of depth image generated from left and right images using Stereo Vision}

While it’s less apparent in the 2-dimensional images, the 3D depth image shows that two of the chocolates are not spherical enough to pass the high-quality standards.  The image in Figure 2 shows a white box around the defects that have been identified.

^{Figure 2: 3D Depth Image with White Boxes Around the Defective Chocolates}

One important consideration when using stereo vision is that the computation of the disparity is based on locating a feature from a line of the left image and the same line of the right image. To be able to locate and differentiate the features, the images need to have sufficient texture, and to obtain better results, you may need to add texture by illuminating the scene with structured lighting.

Finally, binocular stereo vision can be used to calculate the 3D coordinates (X,Y,Z) of points on the surface of an object being inspected. These points are often referred to as point clouds or cloud of points. Point clouds are very useful in visualizing the 3D shape of objects are can also be used by other 3D analysis software.  The AQSense 3D Shape Analysis Library (SAL3D), for example, is now available in the LabVIEW Tools Network, and uses a cloud of points for further image processing and visualization.

To better illustrate how binocular stereo vision works, Figure 3 shows the diagram of a simplified stereo vision setup, where both cameras are mounted perfectly parallel to each other, and have the exact same focal length.

^{Figure 3: Simplified Stereo Vision System}

The variables in Figure 3 are:

b is the baseline, or distance between the two cameras

f is the focal length of a camera

X_A is the X-axis of a camera

Z_A is the optical axis of a camera

P is a real-world point defined by the coordinates X, Y, and Z

u_L is the projection of the real-world point P in an image acquired by the left camera

u_R is the projection of the real-world point P in an image acquired by the right camera

Since the two cameras are separated by distance “b”, both cameras view the same real-world point P in a different location on the 2-dimensional images acquired.  The X-coordinates of points u_L and u_R are given by:

u_L = f * X/Z

and

u_R = f * (X-b)/Z

Distance between those two projected points is known as “disparity” and we can use the disparity value to calculate depth information, which is the distance between real-world point “P” and the stereo vision system. 

disparity  =   u_L – u_R   =  f * b/z

depth = f * b/disparity

In reality, an actual stereo vision set-up is more complex, would look more like the typical system shown in Figure 4, but all of the same fundamental principles still apply.

^{Figure 4: Typical Stereo Vision System}

The ideal assumptions made for the simplified stereo vision system cannot be made for real-world stereo vision applications. Even the best cameras and lenses will introduce some level of distortion to the image acquired, and in order to compensate, a typical stereo vision system also requires calibration.  The calibration process involves using a calibration grid, acquired at different angles to calculate image distortion as well as the exact spatial relationship between the two cameras.  Figure 5 shows the calibration grid included with the Vision Development Module.

^{Figure 5:  A calibration grid is included as a PDF file with the Vision Development Module}

The Vision Development Module includes functions and LabVIEW examples that walk you through the stereo vision calibration process to generate several calibration matrices that are used in further computations to calculate disparity and depth information.  You can then visualize 3D images as shown earlier in Figure 1, as well as perform different types of analysis for defect detection, object tracking, and motion control.

Stereo vision systems are best suited for applications in which the cameras settings and locations are fixed, and won’t experience large disturbances. Common applications include navigation, industrial robotics, automated inspection and surveillance.

The stereo vision features in the LabVIEW Vision Development Module bring new 3D vision capabilities to engineers in a variety of industries and applications areas.  Through the openness of LabVIEW, engineers can also use third -party hardware and software 3D vision tools for additional advanced capabilities, including the SICK 3D Ranger Camera for laser triangulation imaging and the AQSense 3D Shape Analysis Library for 3D image processing. The LabVIEW Vision Development Module makes 3D vision accessible to engineers in one graphical development environment.

Next Steps:

• Download and evaluate Vision Development Module
• Learn more about Vision Development Module
• Acquire 3D and MultiScan Data from SICK 3D Cameras
• Access certified, third-party add-ons with the LabVIEW Tools Network

Reference Material:

• Vision Development Module Concepts Help