# Improved least-squares method for phase-to-height relationship in fringe projection profilometry

- 397 Downloads

## Abstract

### Background

The height estimating function is very important for three-dimensional (3-D) measurement systems based on a digital light processing (DLP) projector and a camera. Sinusoidal fringe patterns of the projector are projected onto an object, and the phase of the measuring point is calculated from the camera image. Generally, the least-squares method (LSM) and look-up table(LUT) method are typically used to obtain the phase-to-height relationship.

### Results

The merit of LSM is that one equation is obtained from geometric analysis, but it has difficulty incorporating lens distortion. The LUT method can obtain an exact model that includes lens distortion, but the amount of memory needed increases with the number of pixels because each model is obtained at each pixel. This paper compares for these two methods and proposes an improved LSM using LUT.

### Conclusions

The proposed method has one equation like the case of LSM, but the modeling result is better than LSM because lens distortion is fully considered.

### Keywords

Shape measurement Fringe pattern projection Phase-height relationship Lens distortion### Abbreviations

- 3-D
Three-dimensional

- CCD
Charge-coupled device

- DLP
Digital light processing

- FPP
Fringe pattern projection

- LSM
Least-squares method

- LUT
Look-up table

## Background

Three-dimensional (3-D) measurement using optical sensors has been studied extensively for applications because of the intrinsic noncontact nature and its high speed of the method [1]. A typical 3-D measurement system based on a fringe pattern projection (FPP) system [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] is constructed using a white light projector and a charge-coupled device (CCD) camera. The projector projects sinusoidal fringe patterns onto an object, and the CCD camera acquires the patterns that are deformed by the object’s shape. The height information of the object is encoded into the deformed fringe pattern recorded by the CCD camera. In this measurement system, it is very important to obtain the phase-to-height relationship, for which there are three principal calibration methods [4, 10].

The first method is based on measuring geometric parameters. This method relies on a certain measurement setup in which the geometric parameters and the intrinsic parameters must be precisely determined in advance. Approaches for deriving the phase-to-height model usually employ various simplifying assumptions, such as simple optical arrangements, small angles, or large distances between the projector and object compared to the illuminated area [3, 4, 5]. Although such approaches are simple, they would be impossible to use for complex optical arrangements because it is very hard to measure the position of the focus and the direction of the center line for the camera. Particularly, even if the related parameters can be precisely measured, it is absolutely impossible to consider the error from lens distortion. Therefore, this method will be excluded in this paper because the effect of lens distortion has to be considered in the phase-to-height model.

The second method is based on the least squares method (LSM) to determine the phase-to-height relationship presented by a mathematical description [6, 7, 8, 9, 10]. This method is more flexible in terms of implementation since it allows the system to be arranged arbitrarily and avoids the problems of accurate geometric parameter determination. Du et al. proposed an equation for the phase-to-height relationship using the coordinate transformation matrix with an arbitrarily arranged fringe projection profilometry system [8]. The equation is a fractional expression with 11 parameters and 3 variables: the phase value, the horizontal coordinate, and the vertical coordinate of the camera. This method uses precise height information such as standard gauge blocks to obtain the related parameters. However, it is hard to measure an object precisely because the equation does not consider lens distortion. Lens distortion consists of radial and tangential components. Huang et al. [10] considered the radial lens distortion to improve the model by Du et al. [8], and the number of the parameters in the equation was increased to 27 for the same variables. The modeling errors were greatly improved even though, only the radial lens distortion was considered. However, it is no longer possible to improve the modeling errors using one modeling equation for the entire working volume.

The third method uses a simple look-up table (LUT) containing the relationship between phase values and heights for each camera pixel [11, 12, 13, 14]. The LUT method obtains the phase-to-height relationship for each pixel without any geometric analysis and then stores the information in the LUT, which it uses to measure the height. It is possible to obtain the exact model including all lens distortion of the camera or the projector with this method, but there are as many modeling functions as there are camera pixels. Liu et al. [11] expressed the height as a fifth order polynomial function of the phase at each pixel to consider all lens distortion and measured the related phases for 21 different heights. When the parameters were determined by the LSM, the modeling results were very good. Guo et al. [12] presented a rational function instead of a polynomial function at each pixel. Because the rational function is derived from the geometry of the measurement system, it achieved higher accuracy than the polynomial function in the measurement noise produced by the inherent divergent illumination of a projector. Jia et al. [13] formulated linear and nonlinear equations for the mapping relationship between the phase and height of the object surface. They showed that the nonlinear calibration model expressed as the rational function was superior to the linear one in terms of measurement accuracy. Most of the LUT methods use a precise z stage for the calibration of the phase-to-height model. However, Li et al. [14] proposed a checker board(a plane board with calibration squares) instead of a precise linear z stage, which made it was possible to estimate the height from the size of the squares.

Most research on the phase-to-height relationship focuses on how the modeling error can be reduced in spite of lens distortion. The LSM model uses one modeling equation for all the pixels, while the LUT method uses pixel-based models to reduce the error by lens distortion. This paper compares the modeling errors between the LSM and the LUT methods and proposes one equation for the phase-to-height relationships with the accuracy of the LUT method. The proposed method is a fusion method that obtains the phase-to-height relationships for each pixel using the LUT method and merges them into one equation to apply to all of the pixel points.

The rest of the paper is organized as follows. In section Phase-to-height relationship by LSM, two representative LSMs are introduced and compared with each other. One method neglects lens distortion, and the other considers partly it. Section Phase-to-height relationship by LUT introduces the conventional LUT method and its modeling results are compared with that of the LSM. Section Improved LSM using LUT presents the improved LSM based on the LUT, which is expressed as one fractional equation like the LSM. Conclusions are given in section Conclusions.

## Methods

### Phase-to-height relationship by LSM

#### Geometric analysis

*Oxy*, camera imaging plane

*O’x’y’*, and projection plane

*O”x”y”*are arbitrarily arranged.

*P*represents an arbitrary point on the object,

*B*is the imaging point of

*P*,

*D*is the original fringe point projected at

*P*, and

*A*and

*C*are the lens centers of the camera and the projector, respectively. For convenience and clarification, the coordinates of a point in a coordinate system are denoted by the corresponding coordinate symbols, and the symbol of the point is chosen as the subscript. For example, point

*P*is denoted as (

*x*

_{ p },

*y*

_{ p },

*z*

_{ p }), (

*x*

_{ p }’ ,

*y*

_{ p }’,

*z*

_{ p }’), and (

*x*

_{ p }”,

*y*

_{ p }”,

*z*

_{ p }”) in coordinate systems

*Oxyz*,

*O’x’y’z’*, and

*O”x”y”z”*, respectively. Based on the coordinate relations among points

*P*,

*A*, and

*B*in the system

*Oxyz*, we obtain:

*B*from system

*O’x’y’z’*to system

*Oxyz*is as follows:

*Rot*(

*z*,

*γ*),

*Rot*(

*y*,

*β*), and

*Rot*(

*x*,

*α*) are the coordinate transformation matrices, and

*α*,

*β*, and

*γ*are the rotation angles of the

*x*’ ,

*y*’ , and

*z*’ axes based on the reference coordinate system

*Oxyz*, respectively. Finally, the height of the object based on the least squares method is as follows [8]:

*z*

_{ p }is the out-of-reference-plane height at point (

*x,y,z*) on the object, coefficients

*c*

_{1}to

*c*

_{5}and

*d*

_{0}to

*d*

_{5}are constant coefficients that need to be determined from geometric information such as the position and direction of the camera and the projector, and

*ϕ*is the fringe phase at the same point. Eq. (3) cannot consider the lens distortion because it is derived from geometric analysis. Generally, if the lens distortion is not considered, it is not difficult to reduce the modeling error for the phase-to-height relationship. Because the lens distortion consists of the radial lens distortion and the tangential lens distortion, the new normalized point coordinate (

*x*

_{ d },

*y*

_{ d }) is defined as follows [10]:

*dx*

_{ r }and

*dy*

_{ r }are the position error caused by the radial lens distortion, while

*dx*

_{ t }and

*dy*

_{ t }are the position error from the tangential lens distortion, which are defined as:

*k*

_{ c1},

*k*

_{ c2},

*k*

_{ c3}, and

*k*

_{ c4}are the constant coefficients of camera lens distortion, and

*R*is defined as \( {R}^2={x^{\prime}}_B^2+{y^{\prime}}_B^2 \). Since the radial distortion is usually much larger than the tangential distortion in modern optics,

*k*

_{ c3}and

*k*

_{ c4}can be reasonably discarded. It is very difficult to calculate the inverse function for the variables

*x’*

_{ B }and

*y’*

_{ B }because the tangential distortion includes the term of the multiplication of

*x’*

_{ B }and

*y’*

_{ B }. Therefore, if the tangential distortion (

*dx*

_{ t }and

*dy*

_{ t }) is ignored, the point coordinate is simplified as:

*c*

_{0}to

*c*

_{13}and

*d*

_{0}to

*d*

_{13}are constants. The coefficients should be determined in advance using the least-squares method based on the phase data for the several reference planes for which the heights are known.

#### Experimental results

*x*-

*y*-

*z*axis because the focal depth of the

*z*-axis is relatively sensitive to the

*x*- and

*y*-axes.

A training set is necessary for the training process of the coefficients in the LSM and is formed by training pair vectors. In the phase-to-height relationship, 3-D points are used as the training pair vectors. To obtain the related coefficients, the LSM needs more training pair vectors than the number of the coefficients. Eqs. (3) and (12) need at least 11 and 27 training pair vectors, respectively. If the training pair vectors use 5 different heights for the z-axis (every 12.5 mm from -25 to +25 mm) and 5 different positions with 200 pixel intervals for each axis of the camera, the number of the training members *N* is 125 (5 × 5 × 5). Using the training set, each coefficient is adjusted to minimize the modeling error by the gradient descent method. However, it is very easy to obtain the coefficients if the pseudo-inverse matrix in Matlab is used for the 125 points.

*x*

_{ d },

*y*

_{ d }) are obtained by normalizing the camera pixel coordinates (

*u*,

*v*) for the camera center. Figure 3 shows the modeling errors of the horizontal line (v = 500) in the phase-to-height relationship obtained using Eq. (3). The modeling error is the difference between the real height and the modeling height. Because lens distortion is neglected in Eq. (3), the errors greatly change according to the

*u*-coordinate. Eq. (3) is not suitable as a phase-to-height model because the maximum errors are as large as ±2 mm. Eq. (12) considers the radial lens distortion. Figure 4 shows the modeling errors for when 27 coefficients of Eq. (12) are obtained using 125 reference points and Matlab functions, and the errors are fairly reduced to ±0.15 mm. However, even with 6,237 (63 × 9 × 11) more reference points than 125, the modeling errors do not improve any further. For less error, it is necessary to derive an equation that includes the tangential lens distortion, which is very difficult to achieve using geometric analysis.

## Phase-to-height relationship by LUT

*x*

_{ d },

*y*

_{ d }) is constant at each camera pixel. The LUT contains the related coefficients that appear in the phase-to-height relationship for each pixel. In this method, the height for each pixel is obtained as a fractional equation or a polynomial equation of the phase value [11, 12, 13, 14]:

## Results and discussion

### Improved LSM using LUT

*u*and

*v*coordinates. If the

*y’*-axis of camera imaging plane

*O’x’y’*is adjusted to be parallel to the

*y”*-axis of projection plane

*O”x”y”*, the camera image can be captured as shown in Fig. 9. The image shows that the fringe pattern is generally parallel to the vertical axis of the camera but is a little bent on the edge by lens distortion. Because each pair of fringe pattern means a phase difference of 2π, the phase value greatly changes for the horizontal axis while it hardly changes for the vertical axis. To combine the coefficients of the LUT, they must monotonically increase or decrease. Figure 10 shows the variation of the coefficients for the horizontal center axis when the model is obtained by the third-order equation with four coefficients. The constant and first-order terms among the four coefficients monotonically increase or decrease, while the second and third-order terms are not. Because the coefficients include rapidly changing terms, it is nearly impossible to express them as polynomial functions of the

*u*value.

*H*=

*c*

_{1}/

*d*

_{1},

*a*=

*d*

_{0}/

*d*

_{1},

*b*= 1/

*d*

_{1}-(

*c*

_{1}/

*d*

_{1})(

*d*

_{0}/

*d*

_{1}). Figure 11 shows the variation of three coefficients for the horizontal center axis when the LUT is obtained by the fractional model of Eq. (14). Because

*a*,

*b*, and

*H*have no rapid change unlike the cases of Fig. 10, the LUT values at

*v*= 500 can be combined into a polynomial function of the

*u*value.

*a*,

*b*, and

*H*. Even though each coefficient (

*a*,

*b*, and

*H*) is expressed by the fifth-order polynomial functions, the modeling errors are very big and worse than those of the LSM shown in Fig. 4. To find the cause for the modeling errors, we investigate the sensitivity of the parameters

*a*,

*b*, and

*H*for the height. The sensitivity is characterized by ∂

*z*

_{ p }/∂

*H*= 1, ∂

*z*

_{ p }/∂

*a*=

*b*/(

*a*+

*ϕ*)

^{2}, and ∂

*z*

_{ p }/∂

*b*= 1/(

*a*+

*ϕ*). The sensitivity ∂

*z*

_{ p }/∂

*H*= 1 means that the error of

*H*directly affects the height

*z*

_{ p }, while the errors of

*a*and

*b*are reduced by

*b*/(

*a*+

*ϕ*)

^{2}and 1/(

*a*+

*ϕ*), respectively. Therefore, it is very important to reduce the modeling error of

*H*to measure the height precisely. The coefficient

*H*shown in Fig. 11 is nearly constant, but there are many fluctuations upon closer examination. It is very difficult to represent the fluctuation as a fifth-order polynomial function. In geometric analysis,

*H*is the height from the reference plane to the lens focus of the projector [15]. If lens distortion is ignored,

*H*is constant, regardless of the camera coordinates as shown in Eq. (3), but it may change if lens distortion is considered. Because

*H*is nearly constant and varies only slightly with lens distortion, Eq. (14) can be modified as an equation with a constant

*H*. Using the average value of

*H*, the LUT for the remaining coefficients

*a*and

*b*is calculated at each pixel. The coefficients

*a*and

*b*are then expressed by the following polynomial functions for the horizontal value

*u*:

*a*

_{0}to

*a*

_{n}and

*b*

_{0}to

*b*

_{n}are the coefficients of the polynomial functions of

*a*and

*b*, respectively. Figure 13 shows the modeling errors for when the fifth-order function is used for the coefficients

*a*and

*b*, and the errors are very similar to those of the LUT in Fig. 8. Compared with the LSM, the error improvement is more than doubled. Thus, 2,560(2×1280) array elements of memory in the LUT are reduced to 12 coefficients for

*a*

_{0}to

*a*

_{5}and

*b*

_{0}to

*b*

_{5}at the horizontal line of

*v*= 500.

*a*and

*b*for the different horizontal lines. However, we can expect the models to be slightly different because the phase values on the vertical axis are influenced by lens distortion, as shown in Fig. 9. Table 1 shows the coefficients of the quintic modeling functions for five different vertical values (

*v*= 100, 300, 500, 700, 900). As expected, most of the coefficients are similar to each other, as shown in the table. The coefficients of the constant and the first-order term are very similar, regardless of the vertical values. Therefore, the 12 coefficients for

*a*

_{0}to

*a*

_{5}and

*b*

_{0}to

*b*

_{5}can also be presented as a polynomial function for the vertical value

*v*. When each coefficient in Table 1 is presented as a third-order polynomial function of

*v*, the 12 equations are as follows:

Modeling coefficients for sampled x-z planes

| | | | | | |
---|---|---|---|---|---|---|

100 | 53.274/-54449 | -216.63/23488 | 46.962/-6035.1 | -33.211/-4008.9 | 16.705/-2824.6 | -13.417/3008.8 |

300 | 52.611/-54412 | -257.74/23081 | 42.901/-5166.2 | -25.114/-2098.2 | 13.512/-2209.5 | -19.354/4707.0 |

500 | 52.128/-54433 | -255.82/22857 | 40.049/-4513.0 | -21.537/-1539.1 | 14.513/-2548.9 | -21.383/4904.7 |

700 | 51.910/-54540 | -255.69/22839 | 38.860/-4171.5 | -22.115/-1841.5 | 13.480/-2357.2 | -13.304/2684.3 |

900 | 51.646/-54639 | -256.95/22947 | 39.330/-4266.2 | -23.079/-2004.2 | 13.295/-2362.3 | -11.644/2423.1 |

*d*

_{0}to

*d*

_{3}and

*e*

_{0}to

*e*

_{3}are calculated by LSM using the values of Table 1. Table 2 shows the coefficients presented in Eqs. (17) and (18) when the values for

*a*

_{0}to

*a*

_{5}and

*b*

_{0}to

*b*

_{5}are presented as third-order polynomial functions of

*v*. Finally, the phase-to-height relationship including the camera coordinates is as follows:

Coefficients of *a* and *b* by LST

| | | | | | |
---|---|---|---|---|---|---|

| 52.16/-54440 | -255.7/22840 | 40.11/-4514 | -23.87/2118 | 13.83/-2415 | -14.26/3100 |

| -1.461/-385.5 | 6.100/-873.7 | -10.56/2584 | -1.596/1643 | 5.875/-1989 | 36.82/-10470 |

| 1.752/-598.7 | -21.90/2303 | 19.02/-4036 | -35.14/7552 | 6.855/-887.3 | 31.12/-7316 |

| -3.488/898.8 | -2.250/1418 | 7.822/-2712 | 97.30/-28280 | -68.62/17340 | -231/65170 |

Eq. (19) is based on the LUT method and includes the camera coordinates for the phase-to-height relationship as in Eq. (12) of the LSM. Because the equation is obtained using only five different horizontal lines, the modeling errors are not guaranteed for the other horizontal lines. To check the modeling errors for the other horizontal lines, they are examined for the line of *v* = 400. Figure 14 shows the modeling results obtained using Eq. (19) for the line of *v* = 400. Although this line was not used to derive Eq. (19), the modeling errors are similar to the results for *v* = 500 used in the training, as shown in Fig. 13.

*v*= 800. The modeling errors are also similar to the results of Figs. 13 and 14. Most of modeling errors from Eq. (19) are within ±0.05 mm except for some of the boundary area. However, the amount of memory for LUT is dramatically reduced from 2.6 million (2×1280×1024) to 48(2×4×6) as shown in Table 2. Compared with the errors of Fig. 4 obtained by the LSM, the modeling error improvements are still more than doubled, even though the number of coefficients is increased a little. Therefore, Eq. (19) considers both the tangential lens distortion and the radial lens distortion like the LUT method.

In summary, the proposed method can obtain the phase-to-height relationship and includes the image coordinates. First, the phase-to-height relationship for each pixel is obtained using the LUT method. Next, the relationships are combined into one equation for the pixels on the same horizontal line. Then, the equation includes the *u*-coordinate the phase for every horizontal line. Finally, the equations for the horizontal lines are combined into Eq. (19) including the *u*- and *v*-coordinates. The proposed method is a fusion of the LSM and LUT methods because it has one mapping function but the same accuracy as the LUT method for 3-D measurement.

## Conclusions

The phase-to-height relationship in a 3-D measurement system is generally obtained by the LSM or LUT methods, which were compared and combined into an improved LSM method based on the LUT method. The LSM can express the phase-to-height relationship as one equation but it is difficult to fully consider lens distortion. LSM consists of two different models, one that neglects lens distortion and another that considers radial distortion. The latter method has ten times less modeling error.

The LUT method eliminates the effect of the distortion because each model is obtained at each pixel, but too much memory is needed. The LUT method shows better performance than the LSM. This study combined the methods into one based on the LUT method but expressed as one equation for the phase-to-height relationship. The proposed equation is expressed as one fractional equation like the case of LSM, but the modeling result is better because it considers tangential lens distortion as well as radial lens distortion.

## Notes

### Acknowledgements

This work was supported by the 2015 Yeungnam University Research Grant.

### Competing interests

The author declares that he has no competing interests.

### Ethics approval and consent to participate

This is not applicable.

### References

- 1.Chen, F, Brown, GM, Song, M: Overview of three-dimensional shape measurement using optical methods. Opt. Eng.
**39**, 10–22 (2000)ADSCrossRefGoogle Scholar - 2.Hu, Y, Xi, J, Li, E, Chicharo, J, Yang, Z: Three-dimensional profilometry based on shift estimation of projected fringe patterns. Appl. Optics
**45**, 678–687 (2006)ADSCrossRefGoogle Scholar - 3.Takeda, M, Mutoh, K: Fourier transform profilometry for the automatic measurement of 3-D object shapes. Appl. Optics
**22**, 3977–3982 (1983)ADSCrossRefGoogle Scholar - 4.Rajoub, B, Lalor, M, Burtom, D, Karout, S: A new model for measuring object shape using non-collimated fringe-pattern projections. J. Optics A: Pure Appl. Optics
**9**, S66–S75 (2007)ADSCrossRefGoogle Scholar - 5.Maurel, A, Cobelli, P, Pagneux, V, Petitjeans, P: Experimental and theoretical inspection of the phase-to-height relation in Fourier transform profilometry. Appl. Optics
**48**, 380–392 (2009)ADSCrossRefGoogle Scholar - 6.Sansoni, G, Carocci, M, Rodella, R: 3D vision based on the combination of gray code and phase shift light projection. Appl. Optics
**38**, 6565–6573 (1999)ADSCrossRefGoogle Scholar - 7.Hu, Q, Huang, P, Fu, Q, Chiang, F: Calibration of a three-dimensional shape measurement system. Opt. Eng.
**42**, 487–493 (2003)ADSCrossRefGoogle Scholar - 8.Du, H, Wang, Z: Three-dimensional shape measurement with an arbitrarily arranged fringe projection profilometry system. Opt. Letters
**32**, 2438–2440 (2007)ADSCrossRefGoogle Scholar - 9.Da, F, Gai, S: Flexible three-dimensional technique based on a digital light processing projector. Appl. Optics
**47**, 377–385 (2008)ADSCrossRefGoogle Scholar - 10.Huang, L, Chua, P, Asundi, A: Least-squares calibration method for fringe projection profilometry considering camera lens distortion. Appl. Optics
**49**, 1539–1548 (2010)ADSCrossRefGoogle Scholar - 11.Liu, H, Su, W, Reichard, K, Yin, S: Calibration-based phase-shifting projected fringe profilometry for accurate absolute 3D surface profile measurement. Optics Commun
**216**, 65–80 (2003)ADSCrossRefGoogle Scholar - 12.Guo, H, He, H, Yu, Y, Chen, M: Least-squares calibration method for fringe projection profilometry. Opt. Eng.
**44**, 033603 (2005)ADSCrossRefGoogle Scholar - 13.Jia, P, Kofman, J, English, C: Comparison of linear and nonlinear calibration methods for phase-measuring profilometry. Opt. Eng.
**46**, 043601 (2007)ADSCrossRefGoogle Scholar - 14.Li, W, Fang, S, Duan, S: 3D shape measurement based on structured light projection applying polynomial interpolation technique. Optik
**124**, 20–27 (2013)ADSCrossRefGoogle Scholar - 15.Chung, B, Park, Y: Hybrid method for Phase-to-height relationship in 3D shape measurement using fringe pattern projection. J. Precision Eng. Manuf
**15**, 407–413 (2014)CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.