Advertisement

Inverse Lightfield Rendering for Shape, Reflection and Natural Illumination

  • Antonin SulcEmail author
  • Ole Johannsen
  • Bastian Goldluecke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10746)

Abstract

We propose an inverse rendering model for light fields to recover surface normals, depth, reflectance and natural illumination. Our setting is fully uncalibrated, with the reflectance modeled with a spatially-constant Blinn-Phong model and illumination as an environment map. While previous work makes strong assumptions in this difficult scenario, focusing solely on specific types of objects like faces or imposing very strong priors, our approach leverages only the light field structure, where a solution consistent across all subaperture views is sought. The optimization is based primarily on shading, which is sensitive to fine geometric details which are propagated to the initial coarse depth map. Despite the problem being inherently ill-posed, we achieve encouraging results on synthetic as well as real-world data.

Keywords

Lightfield Inverse rendering BRDF Natural illumination 

Notes

Acknowledgement

This work was supported by the ERC Starting Grant “Light Field Imaging and Analysis” (LIA 336978, FP7-2014).

References

  1. 1.
    Aldrian, O., Smith, W.A.P.: Inverse rendering of faces on a cloudy day. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 201–214. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33712-3_15 CrossRefGoogle Scholar
  2. 2.
    Aldrian, O., Smith, W.A.: Inverse rendering of faces with a 3D Morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1080–1093 (2013)CrossRefGoogle Scholar
  3. 3.
    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2015)CrossRefGoogle Scholar
  4. 4.
    Bolles, R., Baker, H., Marimont, D.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)CrossRefGoogle Scholar
  5. 5.
    Branch, M.A., Coleman, T.F., Li, Y.: A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems. SIAM J. Sci. Comput. 21(1), 1–23 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16(5), 1190–1208 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Graber, G., Balzer, J., Soatto, S., Pock, T.: Efficient minimal-surface regularization of perspective depth maps in variational stereo. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  8. 8.
    Heber, S., Pock, T.: Shape from light field meets robust PCA. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 751–767. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10599-4_48 Google Scholar
  9. 9.
    Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10113, pp. 19–34. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-54187-7_2 CrossRefGoogle Scholar
  10. 10.
    Huang, R., Smith, W.A.: Shape-from-shading under complex natural illumination. In: 2011 18th IEEE International Conference on Image Processing (ICIP), pp. 13–16. IEEE (2011)Google Scholar
  11. 11.
    Jeon, H., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y., Kweon, I.: Accurate depth map estimation from a lenslet light field camera. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  12. 12.
    Johannsen, O., Honauer, K., Goldluecke, B., Alperovich, A., Battisti, F., Bok, Y., Brizzi, M., Carli, M., Choe, G., Diebold, M., Gutsche, M., Jeon, H.G., Kweon, I.S., Park, J., Schilling, H., Sheng, H., Si, L., Strecke, M., Sulc, A., Tai, Y.W., Wang, Q., Wang, T.C., Wanner, S., Xiong, Z., Yu, J., Zhang, S., Zhu, H.: A taxonomy and evaluation of dense light field depth estimation algorithms. In: Proceedings of the 2nd Workshop on Light Fields for Computer Vision at CVPR (2017)Google Scholar
  13. 13.
    Lombardi, S., Nishino, K.: Reflectance and natural illumination from a single image. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 582–595. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33783-3_42 CrossRefGoogle Scholar
  14. 14.
    Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. ACM Trans. Graph. 22(3), 759–769 (2003)CrossRefGoogle Scholar
  15. 15.
    Oxholm, G., Nishino, K.: Shape and reflectance from natural illumination. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 528–541. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33718-5_38 CrossRefGoogle Scholar
  16. 16.
    Oxholm, G., Nishino, K.: Multiview shape and reflectance from natural illumination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2155–2162 (2014)Google Scholar
  17. 17.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: International Conference on Computer Vision (ICCV 2011) (2011)Google Scholar
  18. 18.
    Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 117–128. ACM (2001)Google Scholar
  19. 19.
    Romeiro, F., Zickler, T.: Blind reflectometry. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 45–58. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15549-9_4 CrossRefGoogle Scholar
  20. 20.
    Rusinkiewicz, S.M.: A new change of variables for efficient BRDF representation. In: Drettakis, G., Max, N. (eds.) Rendering Techniques ’98, pp. 11–22. Springer, Heidelberg (1998).  https://doi.org/10.1007/978-3-7091-6453-2_2 CrossRefGoogle Scholar
  21. 21.
    Strecke, M., Alperovich, A., Goldluecke, B.: Accurate depth and normal maps from occlusion-aware focal stack symmetry. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  22. 22.
    Tao, M., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: Proceedings of the International Conference on Computer Vision (2013)Google Scholar
  23. 23.
    Wang, T., Efros, A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3487–3495 (2015)Google Scholar
  24. 24.
    Wang, T.C., Chandraker, M., Efros, A.A., Ramamoorthi, R.: SVBRDF-invariant shape and reflectance estimation from light-field cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5451–5459 (2016)Google Scholar
  25. 25.
    Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2014)CrossRefGoogle Scholar
  26. 26.
    Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2013)Google Scholar
  27. 27.
    Zeisl, B., Zach, C., Pollefeys, M.: Variational regularization and fusion of surface normal maps. In: Proceedings of the International Conference on 3D Vision (3DV) (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Antonin Sulc
    • 1
    Email author
  • Ole Johannsen
    • 1
  • Bastian Goldluecke
    • 1
  1. 1.University of KonstanzKonstanzGermany

Personalised recommendations