Keywords

1 Introduction

Integral imaging (II) as an attractive three-dimensional (3D) technology can reconstruct the autostereoscopic 3D images without glasses and provide both horizontal and vertical parallaxes with continuous views [1–3]. Basically, the conventional II system consists of the pickup and display sections. In the pickup section, however, there are still some problems such as the limitations of 3D resolution, parallax range, and scene size, which delay the practical application of the II. In the past decades, many researchers have focused on solving these problems, and many technologies have been proposed, including optical pickup method, computer graphic technology, and depth camera based technology.

The conventional optical pickup method using a micro-lens array (MLA) is limited by the scene size, unnecessary beams, and aberrations [4–7]. It is difficult to capture the 3D information on a real and large-sized 3D scene in practice. The quality of the reconstructed 3D image is reduced because of the limitations imposed by the manufacturing technique used for the MLA. Some researchers replace the MLA with a certain camera array in the pickup part to collect the full-color and high-resolution 3D information of a large-sized real 3D scene [8–11]. Then the elemental image array (EIA) is generated with the 3D information by pixel mapping algorithms [11, 12]. Although, those methods can be applied for the large-sized real scene, they require some complex optical and mechanical structures. Furthermore, the calibration of the camera array is a difficulty when the camera array contains a large number of cameras [9, 11].

In recent years, with the development of the computer graphic technology, computer-generated integral imaging (CGII) has been proposed [12–16]. In the CGII, however, the limitations of computer graphic technology itself make the capturing of 3D information on real scenes difficult. For the simplification of the II pickup, a great contribution has been made by Chungbuk National University to collect the EIA of the real scene with a depth camera [17]. This method simplifies the pickup process, but it is limited by the accuracy and resolution of the depth camera. The occlusion and the holes in the depth map degrade the quality of the EIA seriously. Some researches combined the optical and computer-generated methods. The dual-camera enabled II pickup system has been proposed [18]. Obviously, the generated parallaxes are limited by the two-view stereoscopic camera and stereo matching algorithms [19].

In this paper, we propose a system to achieve a high-precision II pickup for the real scene. The dual-projection optical pickup (DPOP) method is used to capture the 3D information with no need of complex calibration. This method obtains the more complete reconstructed 3D shape. Then sub-images are generated based on the color texture and depth data of the real scene. The EIA for the II display are generated by interleaving the obtained sub-images. Experimental results verify the usefulness of the proposed system.

2 Principle of the Proposed II Pickup System

We achieve the high-precision II pickup system for real scene based on the DPOP method. In the proposed system, a more complete 3D shape of the real scene can be obtained. The continuous and tunable parallaxes, the sub-images, are extracted from the color texture and depth data. Interleaving the sub-images, the high-precision EIA can be obtained for the II display.

The architecture of the proposed system is composed of four parts: the input part including the parameters of the DPOP method and the II display input, the reconstruction part including the reconstruction of the more complete 3D shape and the capture of the corresponding color texture, the EIA generation part including the generation of the sub-images and the pixel mapping for the EIA, and the 3D display part showing the EIA through the MLA for the viewers.

2.1 Comparison of the Conventional and Proposed II Pickup System

In the conventional II pickup system, as shown in Fig. 1(a) and (b), the EIA is generated by the MLA or the camera array. In Fig. 1(a), the real scene is captured as elemental images through each micro-lens in the MLA. The size of the scene and the accuracy of the EIA are limited by the parameters of the MLA. The cross-talk effect between neighboring micro-lenses also reduces the quality of reconstructed 3D images [20]. In Fig. 1(b), the camera array is arranged to pick up the real scene. But the camera array needs accurate calibration, operation, and synchronism. Besides, the latency and bandwidth are both the limitations.

Fig. 1.
figure 1

Comparison of EIA generation in the II system (a) by the MLA, (b) by the camera array, and (c) by the proposed DPOP method.

Full size image

In our proposed system, as shown in Fig. 1(c), we utilize two digital light processing projectors (DLPs) to project the structured light [21–24]. The utilization of the dual-projection, DLP1 and DLP2, avoids the error accumulation with the occlusion of a single DLP [25]. In the proposed DPOP method, DLP1 and DLP2 are used for projecting the grating patterns on the real scene, and the deformed patterns are captured by the charge coupled device (CCD). The 3D shape of the real scene is modulated in the deformed patterns. With the phase unwrapping algorithms and the mapping algorithms, the 3D shape can be extracted from the deformed patterns [26–29]. But the phase unwrapping algorithms are not suitable for the blind areas, which can cause the error accumulation in the reconstructed 3D shape. So the reconstructed 3D shape may be not complete based on single DLP projection [25]. In the proposed DPOP method, two DLPs project grating patterns from different directions, and the 3D shape is reconstructed with each of the DLPs, respectively. These 3D shapes have some imperfection, and we introduce the fusion and stitching algorithm to obtain the more complete 3D shape.

2.2 Reconstruction of 3D Shape for Real Scene by DPOP Method

In this paper, the DPOP method is proposed to obtain the complete 3D shape of the real scene. Two DLPs are utilized to project grating patterns to avoid the blind areas of single DLP. The reconstructed 3D shapes can be fused together completely.

As shown in Fig. 2, the DLP1 and DLP2 are arranged in the front of the real scene and project N grating patterns. N grating patterns are arranged by a sinusoidal rule. There is an equal 2π/N phase-shifting between the adjacent grating patterns. The CCD captures the j-th deformed patterns from the DLP i (i = 1, 2), and the intensity of the captured deformed pattern is denoted as I i (x, y, j):

$$ I_{i} (x,y,j) = R_{i} (x,y)\left\{ {A_{i} (x,y) + B_{i} (x,y)\cos [\varphi_{i} (x,y) + \sigma_{j} ]} \right\} $$
(1)

where j = 1, 2,…, and N, and x, y are the pixel coordinates in the captured deformed patterns, R i (x, y) is the surface reflectance of the real scene, A i (x, y) represents the background light intensity, B i (x, y) is the fringe contrast, φ i (x, y) indicates the deformed phase modified by the real scene, and σ j is the phase-shifting of the j-th deformed pattern.

Fig. 2.
figure 2

Principle of the reconstruction of the 3D shape by the proposed DPOP method.

Full size image

As the structured light illumination by single DLP [22, 29], the truncated phase φ′ i (x, y) of the deformed phase φ i (x, y) can be deduced as:

$$ \varphi '_{i} (x,y) = \arctan \frac{{\sum\limits_{n = 1}^{N} {I_{i} (x,y,n)\sin (\sigma_{n} )} }}{{\sum\limits_{n = 1}^{N} {I_{i} (x,y,n)\sin (\sigma_{n} )} }}. $$
(2)

According to the inverse trigonometric functions, φ′ i (x, y) has a value in [−π, π). For the continuous phase distributions, the truncated phase φ′ i (x, y) needs to be unwrapped by the phase unwrapping algorithm [22, 28], and the unwrapped phase is denoted as Ψ i (x, y). Then the phase-changing ΔΨ i (x, y) between the real scene and the reference plane can be calculated. And according to the phase-to-height mapping algorithm, the height Δh i (x, y) of the captured real scene can be calculated as follows:

$$ \frac{1}{{\Delta h_{i} (x,y)}} = a_{i} (x,y) + \frac{{b_{i} (x,y)}}{{\Delta \varphi_{i} (x,y)}} + \frac{{c_{i} (x,y)}}{{\Delta \varphi_{i}^{2} (x,y)}}, $$
(3)

where a i (x, y), b i (x, y) and c i (x, y) are the mapping parameters, which can be acquired by plane calibrations. After dealing with the deformed patterns information, we can get the height and contour information of the real scene. The height Δh i (x, y) obtained by the single DLP i maybe not complete because of the blind areas. However, the obtained height Δh i (x, y) is simply determined by the real scene, not the measurement system. In other words, the height Δh i (x, y) is independent of the parameters in DPOP method. So the different height Δh i (x, y) can be fused and stitched together to obtain the more complete 3D shape. The fused height ΔH(x, y) can be obtained as:

$$ \Delta H(x,y) = \sum\limits_{i = 1}^{M} {\Delta h_{i} (x_{i} ,y_{i} )} ,\quad (x_{i} ,y_{i} ){ \in \varOmega }_{i} , $$
(4)
$$ \sum\limits_{i = 1}^{M} {\Omega _{i} =\Omega } , $$
(5)

where Ω i represents the pixel regions in which the reconstructed height Δh i (x, y) has no accumulate errors in the phase unwrapping algorithm, and Ω represents the whole pixel region.

2.3 Generation Method of Sub-images and EIA for II Display

For the high-precision EIA, we generate the sub-images firstly. The sub-image, which is a collection of the pixels at the same position in every elemental image, has the orthographic projection geometry. In the II display, the sub-images represent a series of directional images. As shown in Fig. 3, the real scene imaged on the EIA plane by the MLA. The parallel rays with the same directional angle θ can be extracted to form an orthographic sub-image [16, 30]. The Fig. 3(a) and (b) show the generation geometries of the sub-images and EIAs with the different central depth planes. The pixel information of the sub-images is extracted from the color texture. And the pixel coordinates are decided by the central depth plane and the depth data. The depth data ΔD(x, y) can be transformed from the fused height ΔH(x, y):

$$ \Delta D(x,y) =\Delta H(x,y)\frac{W}{{R_{w} }} =\Delta H(x,y)\frac{H}{{R_{h} }}, $$
(6)

where the W and H are the real width and height of the real scene, and the R w  × R h is the resolution of captured deformed pattern. The Eq. 6 converts the height ΔH(x, y) from the true height to the pixel coordinates, as the depth data ΔD(x, y). In the sub-image, as shown in Fig. 3, the pixel information K is mapping to the pixel coordinate G, and the pixel shifting between K and G is denoted as Δq. According to the geometry shown in Fig. 3, the sub-image I θ (x, y) for the projecting angle θ can be deduced as:

$$ I_{\theta } (x,y) = T(x +\Delta q_{x} ,y +\Delta q_{y} ), $$
(7)

where T(x, y) is the pixel information of the color texture, and Δq x and Δq y are the components along the x and y axes of the pixel shifting Δq, respectively. The pixel shifting Δq is depend on the depth data and central depth plane, and can be calculated as:

$$ \Delta q = \left( {\Delta D(x,y) - d_{c} } \right)\tan \theta , $$
(8)

where d c , described by pixel coordinate, is the distance between the zero plane of the depth data (z = 0) and the central depth plane.

Fig. 3.
figure 3

Geometry of generations for the sub-images and EIAs in the proposed system: (a) and (b) with the different central depth plane.

Full size image

In the proposed system, the projecting angle θ can be deduced by the parameters of the II display. As shown in Fig. 3, the gap between the MLA and the EIA is g and the interval between the elemental image’s pixel and the centre is Δr. The projecting angle θ can be decided by:

$$ \theta = \arctan \frac{{\Delta r}}{g}. $$
(9)

For the different intervals, the projecting angle θ is different. So the parallaxes are continuous and tunable.

With the sub-images obtained for all projecting angle, the EIA can be generated by interleaving the sub-images based on the viewpoint vector rendering method efficiently [30].

3 Experiments and Results

In our experiments, we use two projectors (CB-X24) as the DLP1 and the DLP2 to project N grating patterns. In our experiments, N = 4, so the phase-shifting σ j  = 2π/4. The CCD (GM501-H) captures the deformed patterns in 640 × 480 pixels. The generated EIA is displayed on the II pad [15]. The experimental setup is shown in Fig. 4.

Fig. 4.
figure 4

Experimental setup of the proposed II pickup system.

Full size image

The proposed II pickup system is configured with the specification in Table 1. The distance between the CCD and the DLPs is 0.331 m, and the center depth plane is located at the d c  = 0 pixel plane and d c  = 130 pixel plane.

Table 1. Configuration parameters and experiment environment of the proposed II system
Full size table

In our experiments, a “man head” is used as the real scene. We reconstruct the 3D shapes of the head with the deformed patterns (Fig. 5(a) and (c)) by each of DLPs, respectively. As shown in Fig. 5(d) and (e), the 3D shapes are not complete by single DLP. The Fig. 5(e) shows the fused 3D shape in our proposed system. From the profile shown in Fig. 5(d), we can see that the fused 3D shape is complete and no error accumulation.

Fig. 5.
figure 5

Captured deformed patterns and reconstructed 3D shapes in experiments (a) and (c) the deformed patterns by DLP1 and DLP2, (b) and (d) the 3D shapes reconstructed with (a) and (c), (e) the fused 3D shape in the proposed system, and (f) the profile of the fused 3D shape (y = 230).

Full size image

We generate the sub-images by the proposed method from the depth data and color texture of the head as shown in Fig. 6(a)–(c). The projecting angle θ is continuous and tunable. The EIAs generated with the different central depth planes are shown in Figs. 6(d) and (e).

Fig. 6.
figure 6

Sub-images and generated EIAs by the proposed method (a), (b), and (c) the sub-images, (d) and (e) the EIAs with the different central depth planes.

Full size image

When the viewer moves in the front of the II display, the reconstructed images from different positions are captured, as shown in Fig. 7.

Fig. 7.
figure 7

Different views of the reconstructed 3D images (a) top view, (b) left view, (c) front view, (d) right view, and (e) bottom view.

Full size image

4 Conclusion

A dual-projection based high-precision II pickup system for the real scene is proposed in this paper. The proposed system takes advantage of the high accuracy of the structured light illumination, and simplifies the optical and mechanical structure for the capturing of real scene. With the high-precision depth data and color texture, the continuous and tunable parallaxes are generated in the experiments. By the proposed II pickup system, the high-quality EIA can be generated efficiently.