Shape from Light Field Meets Robust PCA

  • Stefan Heber
  • Thomas Pock
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8694)


In this paper we propose a new type of matching term for multi-view stereo reconstruction. Our model is based on the assumption, that if one warps the images of the various views to a common warping center and considers each warped image as one row in a matrix, then this matrix will have low rank. This also implies, that we assume a certain amount of overlap between the views after the warping has been performed. Such an assumption is obviously met in the case of light field data, which motivated us to demonstrate the proposed model for this type of data. Our final model is a large scale convex optimization problem, where the low rank minimization is relaxed via the nuclear norm. We present qualitative and quantitative experiments, where the proposed model achieves excellent results.


light field nuclear norm low rank 


  1. 1.
    Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2), 99–106 (1992)CrossRefGoogle Scholar
  2. 2.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation 15, 1373–1396 (2002)CrossRefGoogle Scholar
  3. 3.
    Bishop, T.E., Favaro, P.: The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(5), 972–986 (2012)CrossRefGoogle Scholar
  4. 4.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001)CrossRefGoogle Scholar
  5. 5.
    Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM Journal on Imaging Sciences 3(3), 492–526 (2010)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 1–37 (2011)CrossRefGoogle Scholar
  8. 8.
    Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted ℓ1 minimization (2007)Google Scholar
  9. 9.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40, 120–145 (2011)CrossRefzbMATHMathSciNetGoogle Scholar
  10. 10.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Journal on Scientific Computing 20, 33–61 (1998)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Coffey, D.F.W.: Apparatus for making a composite stereograph (December 1936)Google Scholar
  12. 12.
    Collins, R.T., Collins, R.T.: A space-sweep approach to true multi-image matching (1996)Google Scholar
  13. 13.
    Dudnikov, Y.A.: Autostereoscopy and integral photography. Optical Technology 37(3), 422–426 (1970)Google Scholar
  14. 14.
    Eckart, C., Young, G.: The approximation of one matrix by another of lower rank. Psychometrika 1, 211–218 (1936)CrossRefzbMATHGoogle Scholar
  15. 15.
    Fife, K., Gamal, A.E., Philip Wong, H.S.: A 3mpixel multi-aperture image sensor with 0.7m pixels in 0.11m cmos (February 2008)Google Scholar
  16. 16.
    Heber, S., Ranftl, R., Pock, T.: Variational Shape from Light Field. In: International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (2013)Google Scholar
  17. 17.
    Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psych. 24 (1933)Google Scholar
  18. 18.
    Isaksen, A., McMillan, L., Gortler, S.J.: Dynamically reparameterized light fields. In: SIGGRAPH, pp. 297–306 (2000)Google Scholar
  19. 19.
    Jolliffe, I.T.: Principal Component Analysis. Springer, Berlin (1986)CrossRefGoogle Scholar
  20. 20.
    Levin, A., Fergus, R., Durand, F., Freeman, W.T.: Image and depth from a conventional camera with a coded aperture. ACM Trans. Graph. 26(3) (July 2007)Google Scholar
  21. 21.
    Levoy, M., Hanrahan, P.: Light field rendering. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 31–42. ACM, New York (1996)CrossRefGoogle Scholar
  22. 22.
    Lippmann, R.: La photographie intégrale. Comptes-Rendus, Académie des Sciences 146, 446–551 (1908)Google Scholar
  23. 23.
    Ng, R.: Digital Light Field Photography. Phd thesis, Stanford University (2006),
  24. 24.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Tech. rep., Stanford University (2005)Google Scholar
  25. 25.
    Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Trans. Pattern Anal. Mach. Intell. 15(4), 353–363 (1993)CrossRefGoogle Scholar
  26. 26.
    Peng, Y., Ganesh, A., Wright, J., Xu, W., Ma, Y.: Rasl: Robust alignment by sparse and low-rank decomposition for linearly correlated images. IEEE Transactions on Pattern Analysis and Machine Intelligence 34(11), 2233–2246 (2012)CrossRefGoogle Scholar
  27. 27.
    Peng, Y., Suo, J., Dai, Q., Xu, W., Lu, S.: Robust image restoration via reweighted low-rank matrix recovery. In: Gurrin, C., Hopfgartner, F., Hurst, W., Johansen, H., Lee, H., O’Connor, N. (eds.) MMM 2014, Part I. LNCS, vol. 8325, pp. 315–326. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  28. 28.
    Perwass, C., Wietzke, L.: Single lens 3d-camera with extended depth-of-field (2012)Google Scholar
  29. 29.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: International Conference on Computer Vision (ICCV), pp. 1762–1769. IEEE (2011)Google Scholar
  30. 30.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vision 47(1-3), 7–42 (2002)CrossRefzbMATHGoogle Scholar
  31. 31.
    Stühmer, J., Gumhold, S., Cremers, D.: Real-time dense geometry from a handheld camera. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) Pattern Recognition. LNCS, vol. 6376, pp. 11–20. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  32. 32.
    Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras (December 2013)Google Scholar
  33. 33.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A Global Geometric Framework for Nonlinear Dimensionality Reduction. Science 290(5500), 2319–2323 (2000)CrossRefGoogle Scholar
  34. 34.
    Unger, M., Pock, T., Werlberger, M., Bischof, H.: A convex approach for variational super-resolution. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) Pattern Recognition. LNCS, vol. 6376, pp. 313–322. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  35. 35.
    Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J.: Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26(3) (July 2007)Google Scholar
  36. 36.
    Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D lightfields. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  37. 37.
    Wanner, S., Goldluecke, B.: Spatial and angular variational super-resolution of 4D light fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 608–621. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  38. 38.
    Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Vision, Modelling and Visualization (VMV) (2013)Google Scholar
  39. 39.
    Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An Improved Algorithm for TV-L 1 Optical Flow. In: Cremers, D., Rosenhahn, B., Yuille, A.L., Schmidt, F.R. (eds.) Statistical and Geometrical Approaches to Visual Motion Analysis. LNCS, vol. 5604, pp. 23–45. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  40. 40.
    Wilburn, B., Joshi, N., Vaish, V., Talvala, E.V., Antunez, E., Barth, A., Adams, A., Horowitz, M., Levoy, M.: High performance imaging using large camera arrays. ACM Trans. Graph. 24(3), 765–776 (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Stefan Heber
    • 1
  • Thomas Pock
    • 1
    • 2
  1. 1.Institute for Computer Graphics and VisionGraz University of TechnologyAustria
  2. 2.Safety & Security DepartmentAIT Austrian Institute of TechnologyAustria

Personalised recommendations