Advertisement

Combining Shape from Shading and Stereo: A Joint Variational Method for Estimating Depth, Illumination and Albedo

  • Daniel Maurer
  • Yong Chul Ju
  • Michael Breuß
  • Andrés Bruhn
Article
  • 400 Downloads

Abstract

Shape from shading (SfS) and stereo are two fundamentally different strategies for image-based 3-D reconstruction. While approaches for SfS infer the depth solely from pixel intensities, methods for stereo are based on a matching process that establishes correspondences across images. This difference in approaching the reconstruction problem yields complementary advantages that are worthwhile being combined. So far, however, most “joint” approaches are based on an initial stereo mesh that is subsequently refined using shading information. In this paper we follow a completely different approach. We propose a joint variational method that combines both cues within a single minimisation framework. To this end, we fuse a Lambertian SfS approach with a robust stereo model and supplement the resulting energy functional with a detail-preserving anisotropic second-order smoothness term. Moreover, we extend the resulting model in such a way that it jointly estimates depth, albedo and illumination. This in turn makes the approach applicable to objects with non-uniform albedo as well as to scenes with unknown illumination. Experiments for synthetic and real-world images demonstrate the benefits of our combined approach: They not only show that our method is capable of generating very detailed reconstructions, but also that joint approaches are feasible in practice.

Keywords

Stereo reconstruction Shape from Shading Variational methods Illumination estimation Albedo estimation Joint reasoning 

Notes

Acknowledgements

This work has been partly funded by the German Research Foundation (DFG) within the joint Project BR 2245/3-1 and BR 4372/1-1. Moreover, we thank the DFG for financial support within Project B04 of SFB/Transregio 161.

References

  1. Agarwal, S., Snavely, N., Simon, I., Seitz, S. M., & Szeliski, R. (2009). Building Rome in a day. In Proceedings of IEEE International Conference on Computer Vision (pp. 72–79).Google Scholar
  2. Baillard, C., & Maître, H. (1999). 3-D reconstruction of urban scenes from aerial stereo imagery: A focusing strategy. Computer Vision and Image Understanding, 76(3), 244–258.CrossRefGoogle Scholar
  3. Basha, T., Moses, Y., & Kiryati, N. (2012). Multi-view scene flow estimation: A view centered variational approach. International Journal of Computer Vision, 101(1), 6–21.MathSciNetCrossRefzbMATHGoogle Scholar
  4. Black, M. J., & Anandan, P. (1991). Robust dynamic motion estimation over time. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 296–302).Google Scholar
  5. Blake, A., Zisserman, A., & Knowles, G. (1985). Surface descriptions from stereo and shading. Image and Vision Computing, 3(4), 183–191.CrossRefGoogle Scholar
  6. Bleyer, M., Rhemann, C., & Rother, C. (2011). PatchMatch stereo: Stereo matching with slanted support windows. In Proceedings of British Machine Vision Conference (pp. 14:1–14:11).Google Scholar
  7. Boulanger, J., Elbau, P., Pontow, C., & Scherzer, O. (2011). Non-local functionals for imaging. In Fixed-point algorithms for inverse problems in science and engineering, Springer Optimization and Its Applications (pp. 131–154). Springer.  https://doi.org/10.1007/978-1-4419-9569-8_8
  8. Bredies, K., Kunisch, K., & Pock, T. (2010). Total generalized variation. SIAM Journal on Imaging Sciences, 3(3), 492–526.MathSciNetCrossRefzbMATHGoogle Scholar
  9. Breuß, M., Cristiani, E., Dorou, J.-D., Falcone, M., & Vogel, O. (2012). Perspective shape from shading: Ambiguity analysis and numerical approximations. SIAM Journal on Imaging Science, 5(1), 311–342.MathSciNetCrossRefzbMATHGoogle Scholar
  10. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In Proceedings of European Conference on Computer Vision (pp. 25–36).Google Scholar
  11. Bruhn, A., & Weickert, J. (2005). Towards ultimate motion estimation: Combining highest accuracy with real-time performance. In Proceedings of IEEE International Conference on Computer Vision (pp. 749–755).Google Scholar
  12. Camilli, F., & Tozza, S. (2017). A unified approach to the well-posedness of some non-Lambertian models in shape-from-shading theory. SIAM Journal on Imaging Science, 10(1), 26–46.MathSciNetCrossRefzbMATHGoogle Scholar
  13. Charbonnier, P., Blanc-Féraud, L., Aubert, G., & Barlaud, M. (1997). Deterministic edge-preserving regularization in computed imaging. IEEE Transactions on Image Processing, 6(2), 298–311.CrossRefGoogle Scholar
  14. Chen, Q., & Koltun, V. (2013). A simple model for intrinsic image decomposition with depth cues. In Proceedings of IEEE International Conference on Computer Vision (pp. 241–248).Google Scholar
  15. Cryer, J. E., Tsai, P.-S., & Shah, M. (1995). Integration of shape from shading and stereo. Pattern Recognition, 28(7), 1033–1043.CrossRefGoogle Scholar
  16. Di Zenzo, S. (1986). A note on the gradient of a multi-image. Computer Vision, Graphics and Image Processing, 33, 116–125.CrossRefzbMATHGoogle Scholar
  17. Förstner, W., & Gülch, E. (1987). A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proceedings of ISPRS Intercommission Conference on Fast Processing of Photogrammetric Data (pp. 281–305).Google Scholar
  18. Fua, P., & Leclerc, Y. G. (1995). Object-centered surface reconstruction: Combining multi-image stereo and shading. International Journal of Computer Vision, 16(1), 35–56.CrossRefGoogle Scholar
  19. Galliani, S., Lasinger, K., & Schindler, K. (2015). Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of IEEE International Conference on Computer Vision (pp. 873–881).Google Scholar
  20. Gelfand, I. M., & Fomin, S. V. (2000). Calculus of variations. New York: Dover.zbMATHGoogle Scholar
  21. Graber, G., Balzer, J., Soatto, S., & Pock, T. (2015). Efficient minimal-surface regularization of perspective depth maps in variational stereo. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 511–520).Google Scholar
  22. Hafner, D., Schroers, C., & Weickert, J. (2015). Introducing maximal anisotropy into second order coupling models. In Proceedings of German Conference on Pattern Recognition (pp. 79–90).Google Scholar
  23. Haines, T. S. F., & Wilson, R. C. (2007). Integrating stereo with shape-from-shading derived orientation information. In Proceedings of British Machine Vision Conference (pp. 84:1–84:10).Google Scholar
  24. Han, Y., Lee, J.-Y., & Kweon, I. S. (2013). High quality shape from a single RGB-D image under uncalibrated natural illumination. In Proceedings of IEEE International Conference on Computer Vision (pp. 1617–1624).Google Scholar
  25. Hirschmüller, H. (2008). Stereo processing by semi-global matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 328–341.CrossRefGoogle Scholar
  26. Horn, B. (1970). Shape from shading: A method for obtaining the shape of a smooth opaque object from one view. Ph.D. thesis, Massachusetts Institute of Technology.Google Scholar
  27. Hougen, D. R., & Ahuja, N. (1994). Adaptive polynomial modelling of the reflectance map for shape estimation from stereo and shading. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 991–994).Google Scholar
  28. Immel, D. S., Cohen, M. F., & Greenberg, D. P. (1986). A radiosity method for non-diffuse environments. In Proceedings of SIGGRAPH (pp. 133–142).Google Scholar
  29. Jin, H., Cremers, D., Wang, D., Prados, E., Yezzi, A., & Soatto, S. (2008). 3-D reconstruction of shaded objects from multiple images under unknown illumination. International Journal of Computer Vision, 76(3), 245–256.CrossRefGoogle Scholar
  30. Ju, Y. C., Maurer, D., Breuß, M., & Bruhn, A. (2016). Direct variational perspective shape from shading with Cartesian depth parametrisation. In Perspectives in shape analysis, Mathematics and visualization (pp. 43–72). Springer.  https://doi.org/10.1007/978-3-319-24726-7_3
  31. Kajiya, J. T. (1986). The rendering equation. In Proceedings of SIGGRAPH (pp. 143–150).Google Scholar
  32. Langguth, F., Sunkavalli, K., Hadap, S., & Goesele, M. (2016). Shading-aware multi-view stereo. In Proceedings of European Conference on Computer Vision (pp. 469–485).Google Scholar
  33. Leclerc, Y. G., & Bobick, A. (1991). The direct computation of height from shading. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 552–558).Google Scholar
  34. Liu-Yin, Q., Yu, R., Fitzgibbon, A., Agapito, L., & Russell, C. (2016). Better together: Joint reasoning for non-rigid 3D reconstruction with specularities and shading. In Proceedings of British Machine Vision Conference (pp. 42:1–42:12).Google Scholar
  35. Marr, D., & Poggio, T. (1976). Cooperative computation of stereo disparity. Science, 194, 283–287.CrossRefGoogle Scholar
  36. Maurer, D., Ju, Y. C., Breuß, M., & Bruhn, A. (2015). An efficient linearisation approach for variational perspective shape from shading. In Proceedings of German Conference on Pattern Recognition (pp. 249–261).Google Scholar
  37. Maurer, D., Ju, Y. C., Breuß, M., & Bruhn, A. (2016). Combining shape from shading and stereo: A variational approach for the joint estimation of depth, illumination and albedo. In Proceedings of British Machine Vision Conference (pp. 76:1–76:14).Google Scholar
  38. Mecca, R., Quau, Y., Logothetis, F., & Cipolla, R. (2016). A single-lobe photometric stereo approach for heterogeneous material. SIAM Journal on Imaging Science, 9(4), 1858–1888.MathSciNetCrossRefzbMATHGoogle Scholar
  39. Mémin, E., & Pérez, P. (1998). A multigrid approach for hierarchical motion estimation. In Proceedings of IEEE International Conference on Computer Vision (pp. 933–938).Google Scholar
  40. Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2011). DTAM: Dense tracking and mapping in real-time. In Proceedings of IEEE International Conference on Computer Vision (pp. 2320–2327).Google Scholar
  41. Okatani, T., & Deguchi, K. (1997). Shape reconstruction from an endoscope image by shape from shading technique for a point light source at the projection center. Computer Vision and Image Understanding, 66(2), 119–131.CrossRefGoogle Scholar
  42. Or-El, R., Hershkovitz, R., Wetzler, A., Rosman, G., Bruckstein, A. M., & Kimmel, R. (2016). Real-time depth refinement for specular objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 4378–4386).Google Scholar
  43. Perona, P., & Malik, J. (1990). Scale space and edge detection using anisotropic diffusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2, 629–639.CrossRefGoogle Scholar
  44. Phong, B. T. (1975). Illumination for computer generated pictures. Communications of the ACM, 18(6), 311–317.CrossRefGoogle Scholar
  45. Prados, E., & Faugeras, O. (2005). Shape from shading: A well-posed problem? In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 870–877).Google Scholar
  46. Quéau, Y., Lauze, F., & Durou, J.-D. (2015). A \(L^1\)-TV algorithm for robust perspective photometric stereo with spatially-varying lightings. In Proceedings of International Conference on Scale Space and Variational Methods in Computer Vision (pp. 498–510).Google Scholar
  47. Ranftl, R., Gehrig, S., Pock, T., & Bischof, H. (2012). Pushing the limits of stereo using variational stereo estimation. In Proceedings of IEEE Intelligent Vehicles Symposium (pp. 401–407).Google Scholar
  48. Robert, L., & Deriche, R. (1996). Dense depth map reconstruction: A minimization and regularization approach which preserves discontinuities. In Proceedings of European Conference on Computer Vision (pp. 439–451).Google Scholar
  49. Rouy, E., & Tourin, A. (1992). A viscosity solutions approach to shape-from-shading. SIAM Journal on Numerical Analysis, 29, 867–884.MathSciNetCrossRefzbMATHGoogle Scholar
  50. Samaras, D., Metaxas, D., Fua, P., & Leclerc, Y. G. (2000). Variable Albedo surface reconstruction from stereo and shape from shading. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 480–487).Google Scholar
  51. Schroers, C., Hafner, D., & Weickert, J. (2015). Multiview depth parameterisation with second order regularisation. In Proceedings of International Conference on Scale Space and Variational Methods in Computer Vision (pp. 551–562).Google Scholar
  52. Semerjian, B. (2014). A new variational framework for multiview surface reconstruction. In Proceedings of European Conference on Computer Vision (pp. 719–734).Google Scholar
  53. Simoncelli, E. P., Adelson, E. H., & Heeger, D. J. (1991). Probability distributions of optical flow. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 310–315).Google Scholar
  54. Strecha, C., von Hansen, W., Van Gool, L., Fua, P., & Thoennessen, U. (2008) On benchmarking camera calibration and multi-view stereo for high resolution imagery. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–8).Google Scholar
  55. Valgaerts, L., Wu, C., Bruhn, A., Seidel, H.-P., & Theobalt, C. (2012). Lightweight binocular facial performance capture under uncontrolled lighting. ACM Transactions on Graphics, 31(6), 1–11.CrossRefGoogle Scholar
  56. van Diggelen, J. (1951). A photometric investigation of the slopes and the heights of the ranges of hills in the Maria of the Moon. Bulletin of the Astronomical Institutes of the Netherlands, 1, 283–289.Google Scholar
  57. Vogel, O., Bruhn, A., Weickert, J., & Didas, S. (2007). Direct shape-from-shading with adaptive higher order regularisation. In Proceedings of International Conference on Scale Space and Variational Methods in Computer Vision (pp. 871–882).Google Scholar
  58. Volz, S., Bruhn, A., Valgaerts, L., & Zimmer, H. (2011). Modeling temporal coherence for optical flow. In Proceedings of IEEE International Conference on Computer Vision (pp. 1116–1123).Google Scholar
  59. Weickert, J. (1998). Anisotropic diffusion in image processing. Stuttgart: Teubner.zbMATHGoogle Scholar
  60. Weickert, J., Welk, M., & Wickert, M. (2013). \(L^2\)-stable nonstandard finite differences for anisotropic diffusion. In Proceedings of International Conference on Scale Space and Variational Methods in Computer Vision (pp. 380–391).Google Scholar
  61. Wu, C., Narasimhan, S., & Jaramaz, B. (2010). A multi-image shape-from-shading framework for near-lighting perspective endoscopes. International Journal of Computer Vision, 86, 211–228.MathSciNetCrossRefGoogle Scholar
  62. Wu, C., Wilburn, B., Matsushita, Y., & Theobalt, C. (2011). High-quality shape from multi-view stereo and shading under general illumination. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 969–976).Google Scholar
  63. Wu, C., Zollhöfer, M., Nießner, M., Stamminger, M., Izadi, S., & Theobalt, C. (2014). Real-time shading-based refinement for consumer depth cameras. ACM Transactions on Graphics, 33(6), 200:1–200:10.Google Scholar
  64. Xu, D., Duan, Q., Zheng, J., Zhang, J., Cai, J., & Cham, T.-J. (2014). Recovering surface details under general unknown illumination using shading and coarse multi-view stereo. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1526–1533).Google Scholar
  65. Yoon, K.-J., Prados, E., & Sturm, P. (2010). Joint estimation of shape and reflectance using multiple images with known illumination conditions. International Journal of Computer Vision, 86(2–3), 192–210.Google Scholar
  66. Young, D. M. (1971). Iterative solution of large linear systems. New York: Academic Press.Google Scholar
  67. Yu, L. F., Yeung, S. K., Tai, Y. W., & Lin, S. (2013). Shading-based shape refinement of RGB-D images. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 1415–1422).Google Scholar
  68. Yu, T., Xu, N., & Ahuja, N. (2004). Recovering shape and reflectance model of non-Lambertian objects from multiple views. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (pp. 226–233).Google Scholar
  69. Yu, T., Xu, N., & Ahuja, N. (2007). Shape and view independent reflectance map from multiple views. International Journal of Computer Vision, 73(2), 123–138.CrossRefzbMATHGoogle Scholar
  70. Zimmer, H., Bruhn, A., & Weickert, J. (2011). Optic flow in harmony. International Journal of Computer Vision, 93(3), 368–388.MathSciNetCrossRefzbMATHGoogle Scholar
  71. Zollhöfer, M., Dai, A., Innmann, M., Wu, C., Stamminger, M., Theobalt, C., et al. (2015). Shading-based refinement on volumetric signed distance functions. ACM Transactions on Graphics, 34(4), 96:1–96:14.CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Institute for Visualization and Interactive SystemsUniversity of StuttgartStuttgartGermany
  2. 2.Institute for Applied Mathematics and Scientific ComputingBTU Cottbus-SenftenbergCottbusGermany

Personalised recommendations