Depth Estimation Through a Generative Model of Light Field Synthesis

  • Mehdi S. M. Sajjadi
  • Rolf Köhler
  • Bernhard Schölkopf
  • Michael HirschEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9796)


Light field photography captures rich structural information that may facilitate a number of traditional image processing and computer vision tasks. A crucial ingredient in such endeavors is accurate depth recovery. We present a novel framework that allows the recovery of a high quality continuous depth map from light field data. To this end we propose a generative model of a light field that is fully parametrized by its corresponding depth map. The model allows for the integration of powerful regularization techniques such as a non-local means prior, facilitating accurate depth map estimation. Comparisons with previous methods show that we are able to recover faithful depth maps with much finer details. In a number of challenging real-world examples we demonstrate both the effectiveness and robustness of our approach.


Light Field Markov Random Field Depth Estimation Light Field Image Epipolar Plane Image 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    The (new) stanford light field archive (2008). Accessed 07 Apr 2016
  2. 2.
    Adelson, E.H., Wang, J.Y.A.: Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 14(2), 99–106 (1992)CrossRefGoogle Scholar
  3. 3.
    Bishop, T.E., Favaro, P.: The light field camera: extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 34(5), 972–986 (2012)CrossRefGoogle Scholar
  4. 4.
    Bolles, R.C., Baker, H.H., Marimont, D.H.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)CrossRefGoogle Scholar
  5. 5.
    Buades, A., Coll, B., Morel, J.M.: A non-local algorithm for image denoising. In: CVPR (2005)Google Scholar
  6. 6.
    Carbonetto, P.: A programming interface for L-BFGS-B in MATLAB (2014). Accessed 15 Apr 2015
  7. 7.
    Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: ACM SIGGRAPH (2000)Google Scholar
  8. 8.
    Cho, D., Kim, S., Tai, Y.-W.: Consistent matting for light field images. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part IV. LNCS, vol. 8692, pp. 90–104. Springer, Heidelberg (2014)Google Scholar
  9. 9.
    Dansereau, D.G., Bongiorno, D.L., Pizarro, O., Williams, S.B.: Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. In: IS&T/SPIE Electronic Imaging (2013)Google Scholar
  10. 10.
    Dansereau, D.G., Mahon, I., Pizarro, O., Williams, S.B.: Plenoptic flow: closed-form visual odometry for light field cameras. In: IROS (2011)Google Scholar
  11. 11.
    Dansereau, D.G., Pizarro, O., Williams, S.B.: Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: CVPR (2013)Google Scholar
  12. 12.
    Diebold, M., Goldlücke, B.: Epipolar plane image refocusing for improved depth estimation and occlusion handling. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)Google Scholar
  13. 13.
    Favaro, P.: Recovering thin structures via nonlocal-means regularization with application to depth from defocus. In: CVPR (2010)Google Scholar
  14. 14.
    Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: ICCV (2013)Google Scholar
  15. 15.
    Goldluecke, B., Wanner, S.: The variational structure of disparity and regularization of 4D light fields. In: CVPR (2013)Google Scholar
  16. 16.
    Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: ACM SIGGRAPH (1996)Google Scholar
  17. 17.
    Heber, S., Pock, T.: Shape from light field meets robust PCA. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 751–767. Springer, Heidelberg (2014)Google Scholar
  18. 18.
    Heber, S., Ranftl, R., Pock, T.: Variational shape from light field. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.-C. (eds.) EMMCVPR 2013. LNCS, vol. 8081, pp. 66–79. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Isaksen, A., McMillan, L., Gortler, S.J.: Dynamically reparameterized light fields. In: ACM SIGGRAPH. ACM (1996)Google Scholar
  20. 20.
    Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.H.: Scene reconstruction from high spatio-angular resolution light fields. ACM SIGGRAPH (2013)Google Scholar
  21. 21.
    Levoy, M., Hanrahan, P.: Light field rendering. In: ACM SIGGRAPH. ACM (1996)Google Scholar
  22. 22.
    Li, N., Ye, J., Ji, Y., Ling, H., Yu, J.: Saliency detection on light field. In: CVPR (2014)Google Scholar
  23. 23.
    Liang, C.K., Lin, T.H., Wong, B.Y., Liu, C., Chen, H.H.: Programmable aperture photography: multiplexed light field acquisition. ACM SIGGRAPH (2008)Google Scholar
  24. 24.
    Lin, H., Chen, C., Bing Kang, S., Yu, J.: Depth recovery from light field using focal stack symmetry. In: ICCV (2015)Google Scholar
  25. 25.
    Ng, R.: Digital light field photography. Ph.D. thesis, stanford university (2006). Ren Ng founded LytroGoogle Scholar
  26. 26.
    Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P.: Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2(11) (2005)Google Scholar
  27. 27.
    Park, J., Kim, H., Tai, Y.W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3D-TOF cameras. In: ICCV (2011)Google Scholar
  28. 28.
    Perwass, C., Wietzke, L.: The next generation of photography (2010). Accessed 15 Apr 2015, Perwass and Wietzke founded Raytrix
  29. 29.
    Perwass, C., Wietzke, L.: Single lens 3D-camera with extended depth-of-field. In: IS&T/SPIE Electronic Imaging (2012)Google Scholar
  30. 30.
    Sebe, I.O., Ramanathan, P., Girod, B.: Multi-view geometry estimation for light field compression. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2002)Google Scholar
  31. 31.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: CVPR (2010)Google Scholar
  32. 32.
    Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: ICCV (2013)Google Scholar
  33. 33.
    Tosic, I., Berkner, K.: Light field scale-depth space transform for dense depth estimation. In: CVPR Workshops (2014)Google Scholar
  34. 34.
    Vaish, V., Wilburn, B., Joshi, N., Levoy, M.: Using plane + parallax for calibrating dense camera arrays. In: CVPR (2004)Google Scholar
  35. 35.
    Wang, T.C., Efros, A.A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: ICCV (2015)Google Scholar
  36. 36.
    Wanner, S., Goldluecke, B.: Globally consistent depth labeling of 4D light fields. In: CVPR (2012)Google Scholar
  37. 37.
    Wanner, S., Goldluecke, B.: Spatial and angular variational super-resolution of 4D light fields. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 608–621. Springer, Heidelberg (2012)Google Scholar
  38. 38.
    Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI) 36(3), 606–619 (2014)CrossRefGoogle Scholar
  39. 39.
    Wanner, S., Meister, S., Goldluecke, B.: Datasets and benchmarks for densely sampled 4D light fields. In: Annual Workshop on Vision, Modeling and Visualization: VMV (2013)Google Scholar
  40. 40.
    Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: CVPR (2013)Google Scholar
  41. 41.
    Zhang, Z., Liu, Y., Dai, Q.: Light field from micro-baseline image pair. In: CVPR (2015)Google Scholar
  42. 42.
    Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: fortran subroutines for large-scale bound-constrained optimization. ACM TOMS 23(4), 550–560 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Mehdi S. M. Sajjadi
    • 1
  • Rolf Köhler
    • 1
  • Bernhard Schölkopf
    • 1
  • Michael Hirsch
    • 1
    Email author
  1. 1.Max-Planck-Institute for Intelligent SystemsTübingenGermany

Personalised recommendations