Abstract
We propose an inverse rendering model for light fields to recover surface normals, depth, reflectance and natural illumination. Our setting is fully uncalibrated, with the reflectance modeled with a spatially-constant Blinn-Phong model and illumination as an environment map. While previous work makes strong assumptions in this difficult scenario, focusing solely on specific types of objects like faces or imposing very strong priors, our approach leverages only the light field structure, where a solution consistent across all subaperture views is sought. The optimization is based primarily on shading, which is sensitive to fine geometric details which are propagated to the initial coarse depth map. Despite the problem being inherently ill-posed, we achieve encouraging results on synthetic as well as real-world data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available online http://gl.ict.usc.edu/Data/HighResProbes/.
References
Aldrian, O., Smith, W.A.P.: Inverse rendering of faces on a cloudy day. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 201–214. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33712-3_15
Aldrian, O., Smith, W.A.: Inverse rendering of faces with a 3D Morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 35(5), 1080–1093 (2013)
Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2015)
Bolles, R., Baker, H., Marimont, D.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)
Branch, M.A., Coleman, T.F., Li, Y.: A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems. SIAM J. Sci. Comput. 21(1), 1–23 (1999)
Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16(5), 1190–1208 (1995)
Graber, G., Balzer, J., Soatto, S., Pock, T.: Efficient minimal-surface regularization of perspective depth maps in variational stereo. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2015)
Heber, S., Pock, T.: Shape from light field meets robust PCA. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 751–767. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_48
Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10113, pp. 19–34. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54187-7_2
Huang, R., Smith, W.A.: Shape-from-shading under complex natural illumination. In: 2011 18th IEEE International Conference on Image Processing (ICIP), pp. 13–16. IEEE (2011)
Jeon, H., Park, J., Choe, G., Park, J., Bok, Y., Tai, Y., Kweon, I.: Accurate depth map estimation from a lenslet light field camera. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2015)
Johannsen, O., Honauer, K., Goldluecke, B., Alperovich, A., Battisti, F., Bok, Y., Brizzi, M., Carli, M., Choe, G., Diebold, M., Gutsche, M., Jeon, H.G., Kweon, I.S., Park, J., Schilling, H., Sheng, H., Si, L., Strecke, M., Sulc, A., Tai, Y.W., Wang, Q., Wang, T.C., Wanner, S., Xiong, Z., Yu, J., Zhang, S., Zhu, H.: A taxonomy and evaluation of dense light field depth estimation algorithms. In: Proceedings of the 2nd Workshop on Light Fields for Computer Vision at CVPR (2017)
Lombardi, S., Nishino, K.: Reflectance and natural illumination from a single image. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 582–595. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_42
Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. ACM Trans. Graph. 22(3), 759–769 (2003)
Oxholm, G., Nishino, K.: Shape and reflectance from natural illumination. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 528–541. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33718-5_38
Oxholm, G., Nishino, K.: Multiview shape and reflectance from natural illumination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2155–2162 (2014)
Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: International Conference on Computer Vision (ICCV 2011) (2011)
Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 117–128. ACM (2001)
Romeiro, F., Zickler, T.: Blind reflectometry. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 45–58. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15549-9_4
Rusinkiewicz, S.M.: A new change of variables for efficient BRDF representation. In: Drettakis, G., Max, N. (eds.) Rendering Techniques ’98, pp. 11–22. Springer, Heidelberg (1998). https://doi.org/10.1007/978-3-7091-6453-2_2
Strecke, M., Alperovich, A., Goldluecke, B.: Accurate depth and normal maps from occlusion-aware focal stack symmetry. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2017)
Tao, M., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: Proceedings of the International Conference on Computer Vision (2013)
Wang, T., Efros, A., Ramamoorthi, R.: Occlusion-aware depth estimation using light-field cameras. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3487–3495 (2015)
Wang, T.C., Chandraker, M., Efros, A.A., Ramamoorthi, R.: SVBRDF-invariant shape and reflectance estimation from light-field cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5451–5459 (2016)
Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2014)
Wanner, S., Straehle, C., Goldluecke, B.: Globally consistent multi-label assignment on the ray space of 4D light fields. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition (2013)
Zeisl, B., Zach, C., Pollefeys, M.: Variational regularization and fusion of surface normal maps. In: Proceedings of the International Conference on 3D Vision (3DV) (2014)
Acknowledgement
This work was supported by the ERC Starting Grant “Light Field Imaging and Analysis” (LIA 336978, FP7-2014).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Sulc, A., Johannsen, O., Goldluecke, B. (2018). Inverse Lightfield Rendering for Shape, Reflection and Natural Illumination. In: Pelillo, M., Hancock, E. (eds) Energy Minimization Methods in Computer Vision and Pattern Recognition. EMMCVPR 2017. Lecture Notes in Computer Science(), vol 10746. Springer, Cham. https://doi.org/10.1007/978-3-319-78199-0_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-78199-0_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-78198-3
Online ISBN: 978-3-319-78199-0
eBook Packages: Computer ScienceComputer Science (R0)