Keywords

1 Introduction

In recent years, with the development of 3D display technology, great numbers of researchers paid more and more attention to light field technology, which is an effective mean achieving real-time rendering of 3D scene. The main research contents of the light field technology include the theory and technology of light filed rendering, the construction and application of light field array and the acquisition and reconstruction of signal [1]. And the acquisition and reconstruction of light field signal is one of the core problems need to be solved in the light field. As the light field consists of large-scale camera arrays, the system will have hundreds of images per second to store and transmit in order to meet the real-time needs of 3D scene interaction.

At present, light field acquisition and reconstruction technology can be divided into compressed and non-compressed light field technology. The former captures light field signals through dense camera array [1, 2], generate high precision light field images without compression and reconstruction. However, the efficiency of this light field acquisition system reconstruction is not high due to the large transmission and storage of data. And the latter implements the compression coding of the light field by installing a microlens array or optical mask and other devices in the camera [3]. The light field images are restored by a linear or nonlinear algorithm during the reconstruction phase. Sparse light field arrays can improve transmission and reconstruction efficiency in this way, but with high cost of such a light field acquisition system and complex structure.

Now, the method of light field acquisition and reconstruction mainly face the contradiction between reconstruction precision and efficiency. In this paper, a distributed compressive sensing field reconstruction method using structured measurement matrix are proposed to improve the accuracy and efficiency of reconstructing the images using distributed compression sensing. In the signal acquisition stage, the K-SVK algorithm is used to train the adaptive dictionary of the light field, which makes light field signal sparse coding. In the reconstruction phase of the image, since the slices of the 3D image matrix formed by the light field images sequence have the characteristics of the stepped stripes, the slices are rearranged to obtain vertical stripe and then reconstruct the light field image sequences by combining structured measurement matrix. The experimental results represented that the reconstructed light field images is more effective in subjective visual and objective evaluation compared with other compressive sensing algorithms.

2 Related Work

Light field theory and application is the core research content of computational photography, which is studied for nearly a hundred years. At the beginning, Lippmann put forward the concept of integral photography, and then Gershun proposed the use of light to describe the radiation properties of light and formed a prototype of the light field. Adelson et al. [4] of MIT proposed a seven-dimensional plenoptic function \(P(x,y,z,\theta ,\varphi ,\lambda ,t)\) by using a formal description of the light in space, where (x, y, z) is the position of the receiving light side in the three-dimensional space, \((\theta ,\varphi )\) is the azimuth and inclination of the light, \(\lambda \) represent the wavelength of the light and t indicates the moment. On this basis, Levoy et al. [5] of Stanford proposed a biplane parametric representation of the light field, that is, the light field information is described by the coordinates (s, t) and (u, v) of the intersection of a ray with the spatial plane and the angle plane, the light field information briefly and intuitively through four-dimensional light field function L(s, t, u, v). And Debevec et al. [5] of USC collected the dynamic field of the periodic motion successfully. These series of work set off a wave of light field research.

These days in the field of image processing, the demand for fast and efficient algorithms and low hardware requirements is becoming more and more prominent, and the use of compressive sensing to capture and reconstruct light fields is attracting more and more researchers. Compressive sensing (CS) was proposed by Donoho and Candes et al. [6, 7], which using a measurement matrix to project a sparsible or compressible high-dimensional signal onto a low-dimensional space, and the measured values obtained after projection can be reconstructed by a certain linear or non-linear decoding model [7, 8]. The compressive sensing has loog been applied to the image acquisition. Derin Babacan et al. [8] have proposed a new camera sampling model and reconstructed the light field image using the Bayesian method combined with compressive sensing. Shu and Ahuja [9] put forward an idea of three-dimensional scene and used it in reducing the sampling density of the light field camera array. Gan et al. [10] proposed the method utilizing block compressive sensing to collect and reconstruct image signals, which is use the independent measurement matrix on the same size of the image block for projection and reconstruction, which improves the efficiency of image reconstruction to a certain extent.

3 Proposed Approach

The image sequences of light field camera array are not much different, resulting in a large number of common parts, thus making full use of the correlation between the light field images can effectively improve the efficiency of light field reconstruction. In this paper, a distributed compressive sensing field reconstruction method using structured measurement matrix is proposed. Firstly, all images are superimposed as 3D image matric. Since the slices of the 3D image matrix formed by the sequence of the light field images have the characteristics of the stepped stripes, the slices are rearranged to obtain 2D vertical stripe image, and then the measurement matrix is modified into a Structured Random Matrix (SRM) by using a compressive sensing reconstruction algorithm. Finally, the light field images sequence is recovered from the 3D image matrix. The sketch is shown in Fig. 1.

Fig. 1.
figure 1

Proposed algorithm for acquisition and reconstruction of light field images

Full size image

3.1 Problem Formulation

Compressive sensing is a new type of signal processing method for reconstructing the original signal from sparse signals. The sparse representation of the signal is the primary problem of compressive sensing. Since the compressive sense is based on the original signal that can be sparse or compressed, it is necessary to sparse and transform the original signal on some transform domains, where the original signal \(x\in \mathbb {R}^{n}\) can be represented by the sparse matrix \(\varPsi \in \mathbb {R}^{n\times m}\) as \(x=\varPsi \alpha \), \(\varPsi \) is an orthogonal matrix, \(\alpha \) is the coefficient vector of the original signal. If there are only s non-zero elements in \(\alpha \), then \(\alpha \) is the sparse representation of the s-sparse of the signal x [11].

Distributed compressive sensing aims at the correlation between distributed signals to explore the correlation between signals, and achieve multi-signal distributed compression sampling and joint reconstruction. Distributed compressive sensing divides the signals collected by distributed sources into common component and innovation [12], where the common components are the same and the innovation components are different. When the signal is sparse and reconstructed, the common only need to be processed once. For the distributed source system of the light field array, the distributed compressive sensing technique can reduce the sampling rate by accurately reconstructing the image signal and improving light field image reconstruction efficiency (Fig. 2).

Fig. 2.
figure 2

Light field image capture used DCS method sketch

Full size image

Assuming that the light field camera array is equipped with k camera, the measured value of the image signal capture by the camera \(p(p=1,2,...,k)\) can be expressed as \(y_{p}=\varPhi _{p}x_{p}\), where \(y_{p}\in \mathbb {R}^{m}\) is the vector of the measured value of the camera p, \(\varPhi _{p}\in \mathbb {R}^{m\times n}(m\ll n)\) is the sensing matrix of the camera, m is called the sampling volume, \(R=m/n\) represents the sampling rate, \(x_{p}\in \mathbb {R}^{n}\) is a vectorized representation of the captured image for camera p, so the camera array signal and the set of measured values can be expressed as:

$$\begin{aligned} X=\left[ \begin{array}{c} x_{1}\\ x_{2}\\ \vdots \\ x_{k} \end{array} \right] , Y=\left[ \begin{array}{c} y_{1}\\ y_{2}\\ \vdots \\ y_{k} \end{array} \right] , \varPhi =\left[ \begin{array}{cccc} \varPhi _{1}&{} 0 &{} \dots &{}0\\ 0&{} \varPhi _{2} &{}\dots &{}0\\ \vdots &{}\vdots &{}\ddots &{}\vdots \\ 0&{} 0 &{}\dots &{}\varPhi _{k} \end{array} \right] \end{aligned}$$
(1)

Where \(X\in \mathbb {R}^{kn}\), \(Y\in \mathbb {R}^{km}\), \(\varPhi \in \mathbb {R}^{kn\times km}\), then the measured values of camera arrays can be expressed as \(Y=\varPhi X\). In order to reconstruct the s-sparse coefficient signal \(\alpha \) from the measured value \(y_{p}\), where the measurement matrix \(\varPhi \) and sparse matrix \(\varPsi \) need to satisfy the RIP criterion [9], which is used to limit the column vectors of these two matrices to irrelevant. For any s-sparse signal \(\alpha \) and fitting error threshold \(\delta \subset (0,1)\) is satisfied as:

$$\begin{aligned} (1-\delta )\Vert \alpha _{p}\Vert _{2}^{2}\le \Vert \varPhi _{p}x_{p}\Vert _{2}^{2}\le (1+\delta )\Vert \alpha _{p}\Vert _{2}^{2} \end{aligned}$$
(2)

After completing the design of the measurement matrix, it is necessary to design the reconstruction algorithm to solve the problem of reconstructing the original signal x from the measured value y. The essence of the problem is to solve an under-determined equation, which is theoretically infinite solution, but because of the RIP standard, the problem can be transformed into a convex optimization problem:

$$\begin{aligned} arg\ min\Vert \alpha \Vert _{1}\ s{.}t.\ \Vert Y-\varPhi \varPsi \alpha \Vert _{2}\le \varepsilon \end{aligned}$$
(3)

Where \(\varPhi \) is the distributed compression sensing matrix, \(\varepsilon \) is the noise value of the measured value, which determines the accuracy of the reconstructed image, and the image sequence X of light field camera array is reconstructed by \(X=\varPsi \alpha \).

3.2 Light Field Images Acquisition and Sparse

The sparse representation of the light field signal is the basis and premise of the compressive sensing application in the reconstruction of the light field. The sparse representation of the signal is a linear combination of the primitive signals as a few radians through the sparse matrix, which transforms the signal over a sparse domain to reduce redundant information in the original signal, it can also save storage space and improve signal processing efficiency. Dictionary learning solute the problem of the atomic library can find the best linear combination of m atoms to represent the signal. The core issue is to optimize the following questions:

$$\begin{aligned} min\ \Vert x-D\alpha \Vert _{2}^{2}\ s{.}t.\ \Vert _{0}\le s \end{aligned}$$
(4)

The light field image sequence has a particularity, that is, a collection object is generally fixed, so the gap between the collected images is small. In the sparse representation of the light field signal, the adaptive dictionary is obtained by training the light field signal. The proposed method uses the K-SVK algorithm [13, 14, 16] to train the adaptive dictionary of the light field image sequence named Lego Knights provided by the Stanford University Computer Graphics Laboratory signal acquisition phase.

The K-SVK algorithm is a commonly used algorithm for dictionary learning, and the resulting dictionary has a lower average representation error and a faster convergence rate. The K-SVK dictionary learning algorithm produces a dictionary of segmented smooth texture atoms, the visualization results of the adaptive dictionary obtained by training the light field are shown in Figure 3, which is capable of effectively sparse representation of other light field images.

Fig. 3.
figure 3

Light field dictionary learning visualization results

Full size image

3.3 Light Field Images Reconstruction

The light field images is a sequence of images obtained by photographing the same scene from different views through the camera array. There is great correlation between images. Compared with traditional compressive sensing, the distributed compressive sensing makes full use of the correlation between the light field images to reconstruct the image. According to the characteristics of camera parallax fixation in the light field array, all the images are superimposed into 3D image matrix and the image matrix is cut along each line to get the same stepped stripes with the same inclination angle. And the slice image pixels are rearranged along the inclination angle \(\theta \) of the stepped stripes, then the rearranged image is maked to form into a vertical stripe image slice. Finally, these images are reconstructed by compressive sensing [15]. Figure 4 illustrates the light field image sequence rearrangement: (a) The 3D image matrix consists of a sequence of light field images and is cut along each row to obtain slices of stepped fringe features. (b) According to the characteristics of the slice image, the inclination angle \(\theta \) of the reorder line \(L_{\theta }\) is solved. (c) The 2D vertical stripe image is rearranged along the \(L_{\theta }\).

Another key problem to reconstruct light field images using DCS is to select a measurement matrix \(\varPhi \). At present non-structured measurement matrix (NSRM) such as Gaussian matrix, Bernoulli matrix and Hadamard matrix were generally selected to reconstruct image when use CS. This kind of measurement matrix satisfies the RIP criterion and is simple to construct, but it has the drawbacks of its unfixed inherent elements and large required storage space, which is difficult to meet in a large scale of data transmission and real-time reconstruction.

Fig. 4.
figure 4

Light field images sequence reconstruction using DCS

Full size image

In this paper, a structured random matrix [16] (SRM) with high computational speed and storage efficiency is selected for the measurement matrix according to the mass data feature of light field, which has following advantages: (1) SRM is almost uncorrelated to all orthogonal matrices and multiple sparse signals and satisfy RIP criteria; (2) SRM accurately reconstructs the original signal with fewer number of measurements than NSRM; (3) SRM can be decomposed into the product of many structured sub-matrices of block diagonalized matrices, facilitating block processing and linear filtering, with low complexity and fast computational properties. SRM is defined as:

$$\begin{aligned} \varPhi _{B}=\sqrt{N/M}\cdot D\cdot F\cdot R \end{aligned}$$
(5)

Where \(R\in \mathbb {R}^{n\times n}\) is a random permutation matrix or diagonal random matrix, which randomly scrambles the order of the target signal, the diagonal element \(R_{ii}\) is Bernoulli random variables in the same distribution. \(F\in {\mathbb {R}}^{n \times n}\) is a standard orthogonal matrix, the role of which is to disperse the sampling signal information to all measurement points, usually with Fast Fourier Transform (FFT), Discrete Cosine Transform (DCT), Walsh Hadamar Transform (WHT). \(D\in {\mathbb {R}}^{n\times n}\) is called the subsampling matrix, which randomly selects the M-row subset of the matrix FR. The coefficient \(\sqrt{N/M}\) is in order to make the energy of the measurement vector consistent with the original signal, also known as the compression sampling rate. Using the SRM as the measurement matrix to carry out the signal reconstruction consists of three steps. First, the target signal is pre-randomized by R; Then, the target orthogonal matrix F is applied to the random signal of the previous step; Finally, randomly select the N column from FR as the measurement matrix.

In this paper, taking advantage of SRM can be used for large-size images directly observe the characteristics of sampling, the SRM-DCS algorithm, which is suitable for light field signal acquisition, is proposed by using SRM as the measurement matrix (Table 1).

Table 1. Proposed SRM-DCS algorithm
Full size table

4 Experiments

In order to compare the performance of SRM-DCS algorithm with other compressive sensing image reconstruction algorithms, the light field image (Fig. 5) Lego Knights provided by Stanford University Computer Graphics Laboratory [17] was selected to experiment. The experimental hardware equipment is a computer that has Intel i5-4590 CPU (3.30 GHz) and memory for 8 GB. The experiments were conducted using MATLAB 2014b on Windows.

Fig. 5.
figure 5

\(5\times 5\) Lego Knights light field images

Full size image

NSRM-DCS (Gaussian) in the experiment represents that the image is reconstructed with a random Gaussian measurement matrix, NSRM-DCS (Bernoulli) represents the use of random Bernoulli measurement matrix for image reconstruction, and NSRM-DCS (PartHadamard) represents the use of part of the Hadamard measurement matrix for image reconstruction. These three methods are commonly used algorithm implementation to do the measurement matrix compression sensing (NSRM-DCS) with non-structured measurement matrix (NSRM). SRM-DCS (FFT) represents a fast Fourier transform (FFT) as an orthogonal matrix F in a structured random matrix, SRM-DCS (DCT) denotes a discrete cosine transform (DCT) as an orthogonal matrix F in a structured random matrix, SRM-DCS (WHT) represents the use of the Walsh Hadamard transform (WHT) as the orthogonal matrix F in the structured random matrix. And these three methods are commonly used algorithm implementation to do the measurement matrix compression sensing (NSRM-DCS) with structured measurement matrix (SRM).

Equation 6 shows with the increase in sample volume, the relationship between the sampling volume and the reconstruction success rate using NSRM-DCS and SRM-DCS. The number of experimental iterations is 200 times, the image quality is excel-lent if the general image SNR is higher than 60 dB, the reconstructed image whose SNR greater than 60 dB is called reconstruction successful image in the experiment. X in this experiment is the original image, \(X_{r}\) the reconstructed image, and the SNR can be expressed as:

$$\begin{aligned} SNR=20\lg (\Vert X \Vert _{2}/\Vert X-X_{r}\Vert _{2}) \end{aligned}$$
(6)

Figure 6 represented that the success rate using SRM is better than using NSRM acted as a measurement matrix, and the former algorithm has earlier convergence. The success rate of SRR-DCS is lower than that of SRM-DCS, which has a close to the success rate using the SRM acted as the measurement matrix only when using the Part Hadamard acted as measurement matrix and the sampling volume is less than 90. In the sampling volume of 130, the use of SRM has higher success rate of about 20% compared with NSRM. The convergence time of SRM-DCS is earlier than NSRM-DCS, and Fig. 6 shows the measurement matrix algorithm using SRM start-ed convergence when the sampling volume reaches 110, but the measurement matrix algorithm using NSRM don’t begin to converge when the sampling volume is 130.

Fig. 6.
figure 6

The Relationship between the Sampling Volume and the reconstruction success rate using NSRM-DCS and SRM-DCS

Full size image

Table 2 shows the PSNR values of the Lego Knights field images sequence, when use NSRM-DCS and SRM-DCS to reconstruct the light field image. Among them, the sampling rate of R increases from 0.1 to 0.9, and the result represented that the quality of image reconstruction using SRM-DCS is better than that of NSRM-DCS. SRM acted as the measurement matrix reconstructed more efficiently than the NSRM measurement matrix, with the average quality increase by 2.01 dB, which can further show that the reconstruction accuracy of SRM-DCS algorithm is better than the NSRM-DCS algorithm.

Table 2. PSNR value (dB) of reconstructed images using NSRM-DCS and SRM-DCS
Full size table

5 Conclusion

In this paper, a distributed compressive sensing field reconstruction method (SRM-DCS) using structured measurement matrix was proposed. In the light field image acquisition stage, the K-SVD algorithm was selected to train the adaptive light field dictionary. In the light field reconstruction phase, since the slices of the 3D image matrix formed by the light field images sequence have the characteristics of the stepped stripes, slices were rearranged according to the inclination angle to obtain a 2D vertical stripe image, and the light field images sequence is reconstructed in combination with the structured measurement matrix (SRM), the experimental results indicated that the algorithm has faster reconstruction precision and reconstruction efficiency. However, the reconstructed light field images using DCS did not make full use of the a prior information of the light field images sequence, so it needs to be optimized in the reconstruction method. In the future, we can spend more attention to the reconstruction of the light field by using the prior information of the compressive sensing combined with the images sequence.