Background

The accurate knowledge of the refractive index of air is very important in length measurements. The wavelength of a laser (which is commonly used for length measurements) propagating in air changes depending on the air refractive index. A lower refractive index of air corresponds to a higher speed of light, and as the speed of light changes, its wavelength also changes. When measuring linear displacements in air with a laser, it is particularly necessary to perform wavelength compensation. For most applications, the refractive index of air is directly calculated by the Edlén equation (for e.g., [1,2,3,4]).

Here, we consider the case of a length measurement based on the use of the wavelength of a single-mode laser. Firstly, we determine values for air pressure, temperature, and humidity using sensors. We next calculate the phase refractive index of air (PRA) as np. Secondly, we know the laser vacuum wavelength to be λvac, and therefore, using np and λvac, we calculate the laser wavelength in air, λair. For a linear displacement L, we measure L as Lair = λair × (M + N)/2. Here, M and N denote the integer and fraction parts, respectively, of the length measurement. Next, it is required to compensate for the influence of np on Lair. By calculating Lvac = Lair × np, we can convert the length measured in air (Lair) to the length in vacuum, Lvac. Without the influence of np, the values of lengths in vacuum are comparable with each other.

Probably the most exciting recent advancement in the field of length measurement is the development of the distance ranging technique based on the femtosecond optical frequency comb (FOFC) [5]. This technique enables us to measure length based on not only wavelength (e.g., [6,7,8,9]) but also the adjacent pulse repetition interval length (APRIL) (e.g., [10,11,12,13,14,15,16,17,18,19,20]), where APRIL denotes the physical length between two adjacent pulses.

In the study, we consider the case of a length measurement based on the use of the APRIL of an FOFC laser. We acquire values for air pressure, temperature, and humidity, and we calculate the group refractive index of air (GRA), ng, using the following equation [21].

$$ {\mathrm{n}}_{\mathrm{g}}\left({\lambda}_{\mathrm{cen}\_\mathrm{vac}}\right)={\mathrm{n}}_{\mathrm{p}}\left({\lambda}_{\mathrm{cen}\_\mathrm{vac}}\right)-{\lambda}_{\mathrm{cen}\_\mathrm{vac}}\times {\left(\mathrm{d}\;{\mathrm{n}}_{\mathrm{p}}\left({\lambda}_{\mathrm{vac}}\right)/\mathrm{d}{\lambda}_{\mathrm{vac}}\right)}_{\lambda_{\mathrm{cen}\_\mathrm{vac}}}, $$
(1)

where λcen_vac represents the central wavelength of the FOFC. Further, \( {\left({\mathrm{dn}}_{\mathrm{p}}\left({\lambda}_{\mathrm{vac}}\right)/\mathrm{d}{\lambda}_{\mathrm{vac}}\right)}_{\lambda_{\mathrm{cen}\_\mathrm{vac}}} \) represents the derivative of the function y = np(λvac) at λvac = λcen_vac.

We know that the vacuum value of APRIL is Λvac. Using np andΛvac, we calculate the FOFC’s APRIL in air, Λair. For a linear displacement L, we represent L as LΛair = Λair × (MΛ + NΛ)/2. Here, MΛ and NΛ denote the integer and fraction parts, respectively, of the length measurement, and they are required when LΛair is measured in units ofΛair. Next, we compensate for the influence of ng on LΛair. By calculating LΛvac = LΛair × ng, we can convert the length measured in air LΛair to the length in vacuum LΛvac. Without the influence of ng, the values of the lengths in vacuum are comparable with each other.

In essence, for APRIL-based distance evaluation, the GRA is required, which can be calculated via the derivative of the PRA with respect to the wavelength.

The expression of the PRA given by the empirical formula is a complicated function, and mistakes in differentiation can lead to transformation and calculation mistakes. One way to avoid this problem is to use the difference approximation. The differential approximation is a method of approximating derivatives with algebraic operation of discrete values. In a previous research [22], we determined that by means of the two-point central difference with a step size of 10 nm, the GRA could approximated by an order of 60 × 10− 9 for both the visible light range (380-770 nm) and the optical fiber light transmittance range (1260-1625 nm). In addition [23], we found that the GRA computation can be easily approximated by the four-point central difference approximation method and that approximation accuracies of less than 30 × 10− 9 can be achieved over the entire wavelength range (320-1680 nm) with a step size of 10 nm.

The question remains as to whether the applications of the two-point difference approximation could be sufficiently accurate to cater to non-expert users over the entire abovementioned wavelength range. When compared with the four-point central difference approximation method, the calculation of two-point difference approximation method is very simple. In previous researches [22, 23], we calculated the PRA with nine-digit accuracy. In this study, we verify the achievable accuracy of the two-point difference approximation method for different step sizes and number of significant digits (NSD) of the PRA.

The rest of the paper is organized as follows. Section 2 presents an overview of the definition of two-point difference approximation. Section 3 presents our numerical calculations. Section 4 presents the results of our numerical calculations and discussions. Finally, Section 5 presents our conclusions.

Methods

Having understood the importance of environmental compensation in length measurement, we next focus our attention on differential calculations by difference approximations (e.g., [24, 25]). The complexity of the calculation of the two-point difference methods (namely, two-point forward, two-point backward, and two-point central difference methods) is the same. As can been inferred from previous researches [22, 23], when compared with the two-point central difference method, no significant improvement in the calculation accuracy can be expected with the two-point forward and two-point backward difference methods. Based on these two facts, we only consider the two-point central difference method in the following discussion. We first present the difference approximation equations for the convenience of users. The expression for the GRA calculated by means of the two-point central np_2cent(λvac) difference method is presented as Eqs. (2).

$$ {\mathrm{n}}_{g\_\mathrm{cent}}\left({\lambda}_{\mathrm{vac}}\right)={\mathrm{n}}_{\mathrm{p}}\left({\lambda}_{\mathrm{vac}}\right)-{\lambda}_{\mathrm{vac}}\times \left[\frac{{\mathrm{n}}_{\mathrm{p}}\left({\lambda}_{\mathrm{vac}}+\Delta \lambda \right)-{\mathrm{n}}_{\mathrm{p}}\left({\lambda}_{\mathrm{vac}}-\Delta \lambda \right)}{2\times \Delta \lambda}\right] $$
(2)

With this understanding of the calculation of ng for environmental compensation and the concept of the difference approximation, we turn to the numerical calculations.

The goal of this study is to determine a specific step size and NSD of np that can be applied to most wavelengths, thereby resulting in an approximated calculation of ng with a negligible difference from the actual value. The Edlén formula affords an accuracy of approximately 3~ 5 × 10− 8 [26]. We set a difference of < 5 × 10− 9 between differential calculations and the differential approximation as an acceptable level of difference.

Due to length constraints of this paper, we only discuss the Edlén empirical equations [2,3,4] in which the PRA can be derived as a function of the wavelength λvac, temperature T, barometric pressure P, and humidity H. For the phase refractive index, we use the equations given in Ref. [4].

Numerical experiments

The absolute value of (the) difference (between the theoretical and approximated values) is a function of the approximation method, step size, differential wavelength point, and NSD of np. The calculable range of the empirical formula [4] is 300 to 1700 nm, which means that λvac + Δλ and/or λvac − Δλ must lie within this range. In other words, for the two-point central difference method, the calculable ranges of the GRA is [λmin, λmax] (λmax + Δλ ≤ 1700, λmin − Δλ ≥ 300). Since our objective is the determination of a specific step size that can be applied to almost all wavelengths, as a test, we apply step sizes that are multiples of 10 and less than 10 nm.

In the study, the first numerical calculations were performed under the following environmental conditions: T = 20 °C, P = 101.325 kPa, and H = 50%. The theoretical value of n g (λvac) (λvac : 300 ≤ λvac ≤ 1700 nm) was calculated with the use of Eq. (1). The approximate values of ng_2cent(λvac) were calculated based on Eqs. (2), for different λvac values. Subsequently, by using these values, we calculated the absolute value of the difference between the theoretical and difference approximation values as |n g (λvac) − ng_2cent(λvac)| for the two-point central difference method. The approximate values change when the step sizes and NSD of the PRA change.

Results and discussion

Figure 1 shows the variation in the absolute value of difference with the calculable wavelength with Δλ = 1 nm upon calculating the PRA with 12-digit accuracy. In Fig. 1, we note that the maximum absolute value of the two-point central difference method is obtained at around 1700 nm. As mentioned in a previous study [23], the “zigzag” nature of the curves is due to the round-off error in calculating the PRA. The round-off error originates from numerical calculations that use a limited number of digits in a computer. In the case of this combination (namely, Δλ = 1 nm and np with 12 digits), the absolute value of the difference of the two-point central difference method is mainly affected by the round-off error. The reason underlying the high round-off error occurring at larger wavelengths is that the curve of PRA at larger wavelengths is “gentler” than that at smaller wavelengths. A gentle curve means the difference between these two values used for calculation is too small. If we do not consider more digits, we cannot accurately evaluate their difference.

Fig. 1
figure 1

Variation in the absolute value of difference as a function of wavelength for Δλ = 1 nm upon calculating the phase refractive index of air (PRA) with 12-digit accuracy

Full size image

We next change the step size Δλ and NSD of the PRA to observe the change in the maximum absolute value of difference. For example, Fig. 2 shows the variation in the absolute value of difference with Δλ = 0.1 nm upon calculating the PRA with 16-digit accuracy. From the figure, we note that the maximum absolute value of difference obtained with the two-point central difference method exhibits a corresponding change, with the corresponding position shifting to around 300 nm. The absolute values of difference obtained at smaller wavelengths are mainly affected by the truncation error. Truncation error exists because the difference approximation is calculated based on the linear Taylor expansion. This shift of the maximum value indicates that the main error contributor is now the truncation error and not the round-off error. In the case of this combination (namely, Δλ = 0.1 nm and np with 16 digits), the maximum absolute value of difference for the two-point central difference method is mainly affected by the truncation error. The reason underlying high truncation error at smaller wavelengths is that the curve of the PRA at smaller wavelengths is “steeper” than that at larger wavelengths. Since the difference approximation is a linear approximation, its precision is suitable for a gentle curve, and its accuracy is poor for a steep curve.

Fig. 2
figure 2

Variation in the absolute value of difference as a function of wavelength for Δλ = 0.1 nm upon calculating the phase refractive index of air (PRA) with 16-digit accuracy

Full size image

When the combination of Δλ and the NSD of PRA is changed, the main contributor of the error in the maximum absolute value of difference also changes. It is difficult to estimate the absolute value of difference based on the law of propagation of uncertainty. For this reason, we perform numerical simulations to evaluate the maximum absolute value of difference without considering the main cause of error. Here, we remark that the PRA is a continuous function, and therefore, we only test the range around the upper and lower limits of the variation range of the environmental parameters. Since the maximum absolute value of the difference is obtained around 300 nm and 1700 nm, we address only these two cases in the following discussion.

Figures 1 and 2 indicate that there is an optimal combination of Δλ and NSD of PRA that can minimize the total error. We can determine a suitable estimate of this combination by balancing the truncation error and the round-off error of the PRA; this forms the basis of our strategy. We test different pairs of Δλ and PRA digits. Figure 3 shows a candidate result that can be optimized to almost all wavelengths with Δλ = 1.5 nm and 12-digit PRA accuracy. We note that the maximum absolute value of difference is 2.7 × 10− 9. Since the purpose of this research is to determine a combination of the step size Δλ and NSD of the PRA satisfying the error conditions, we only consider this combination in the following discussion.

Fig. 3
figure 3

Variation in the absolute value of difference obtained with the two-point central difference method as a function of wavelength for Δλ = 1.5 nm upon calculating the phase refractive index of air (PRA) with 12-digit accuracy

Full size image

Figures 4 and 5 show the variation in the absolute value of difference as a function of temperature. The maximum absolute values of difference in Figs. 4 and 5 are 3.5 × 10− 9 and 2.8 × 10− 9, respectively. The maximum absolute value of difference with λvac = 301.5 nm is larger than that of λvac = 1698.5 nm. In the following discussion, we only present the results for λvac = 301.5 nm.

Fig. 4
figure 4

Variation in the absolute value of difference obtained the with two-point central difference method as a function of temperature (λvac = 301.5 nm, P = 101.325 kPa, H = 50%)

Full size image
Fig. 5
figure 5

Variation in the absolute value of difference obtained with two-point central difference method as a function of temperature (λvac = 1698.5 nm, P = 101.325 kPa, H = 50%)

Full size image

Figures 6 and 7 depict the variation in the absolute value of difference with pressure. The maximum absolute values of difference in Figs. 6 and 7 are 3.8 × 10− 9 and 3.7 × 10− 9, respectively. The maximum absolute value of difference at T = 10 °C is larger than that at T = 30 °C, and therefore, in the following discussion, we subsequently address only the case of T = 10 °C.

Fig. 6
figure 6

Variation in the absolute value of difference obtained with the two-point central difference method as a function of pressure (λvac = 301.5 nm, T = 10 °C, H = 50%)

Full size image
Fig. 7
figure 7

Variation in the absolute value of difference obtained with the two-point central difference method as a function of pressure (λvac = 301.5 nm, T = 30 °C, H = 50%)

Full size image

Figure 8 shows the variation in the absolute value of difference as a function of humidity. The maximum absolute value of difference in Fig. 8 is 4.1 × 10− 9.

Fig. 8
figure 8

Variation in the absolute value of difference obtained with the two-point central difference method as a function of humidity (λvac = 301.5 nm, T = 10 °C, P = 120 kPa)

Full size image

Here, we again recall that the PRA is a continuous function. The maximum or minimum value of the function appears at the upper and lower bounds of the variable range. The round-off error leads to the zigzagging of the curve. As can be observed in the above figures, there is no significant deviation in the curve. Based on these facts, we can conclude that under environmental conditions of T : 10 ≤ T ≤ 30 °C, P : 60 ≤ P ≤ 120 kPa, and H : 5 ≤ H ≤ 85 %, the maximum absolute value of difference is less than 5 × 10− 9 over the entire wavelength range (301.5-1698.5 nm). This result proves the feasibility of our approach.

Conclusion

Under environmental conditions of T : 10 ≤ T ≤ 30 °C, P : 60 ≤ P ≤ 120 kPa, and H : 5 ≤ H ≤ 85 %, we confirmed the accuracy of the two-point central difference approximation method via numerical calculations. Our strategy was to determine a suitable combination of the step size Δλ and NSD of PRA by balancing the truncation error and the round-off error. We determined that the GRA computation can be easily approximated by the two-point central difference approximation method and that approximation accuracies of less than 5 × 10− 9 can be achieved over the entire wavelength range (301.5-1698.5 nm) with Δλ = 1.5 nm and 12-digit PRA accuracy. Our method can be used by non-expert users to obtain the GRA with sufficient accuracy for environmental compensation in length measurements.