Advertisement

Journal of Mathematical Imaging and Vision

, Volume 60, Issue 4, pp 576–593 | Cite as

Normal Integration: A Survey

  • Yvain Quéau
  • Jean-Denis Durou
  • Jean-François Aujol
Article

Abstract

The need for efficient normal integration methods is driven by several computer vision tasks such as shape-from-shading, photometric stereo, deflectometry. In the first part of this survey, we select the most important properties that one may expect from a normal integration method, based on a thorough study of two pioneering works by Horn and rooks (Comput Vis Graph Image Process 33(2): 174–208, 1986) and Frankot and Chellappa (IEEE Trans Pattern Anal Mach Intell 10(4): 439-451, 1988). Apart from accuracy, an integration method should at least be fast and robust to a noisy normal field. In addition, it should be able to handle several types of boundary condition, including the case of a free boundary and a reconstruction domain of any shape, i.e., which is not necessarily rectangular. It is also much appreciated that a minimum number of parameters have to be tuned, or even no parameter at all. Finally, it should preserve the depth discontinuities. In the second part of this survey, we review most of the existing methods in view of this analysis and conclude that none of them satisfies all of the required properties. This work is complemented by a companion paper entitled Variational Methods for Normal Integration, in which we focus on the problem of normal integration in the presence of depth discontinuities, a problem which occurs as soon as there are occlusions.

Keywords

3D-reconstruction Integration Normal field Gradient field 

Notes

Acknowledgements

We are grateful to the reviewers for the constructive discussion during the reviewing process.

References

  1. 1.
    Agrawal, A., Chellappa, R., Raskar, R.: An algebraic approach to surface reconstruction from gradient fields. In: Proceedings of the 10th IEEE International Conference on Computer Vision, vol. I, pp. 174–181. Beijing, China (2005)Google Scholar
  2. 2.
    Agrawal, A., Raskar, R., Chellappa, R.: What is the range of surface reconstructions from a gradient field? In: Proceedings of the 9th European Conference on Computer Vision, vol. I. Lecture Notes in Computer Science, vol. 3951, pp. 578–591. Graz, Austria (2006)Google Scholar
  3. 3.
    Aubert, G., Aujol, J.F.: Poisson skeleton revisited: a new mathematical perspective. J. Math. Imaging Vis. 48(1), 149–159 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Badri, H., Yahia, H.: A non-local low-rank approach to enforce integrability. IEEE Trans. Image Process. 25, 3562 (2016)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Badri, H., Yahia, H., Aboutajdine, D.: Robust surface reconstruction via triple sparsity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio, USA (2014)Google Scholar
  6. 6.
    Bähr, M., Breuß, M.: An improved Eikonal equation for surface normal integration. In: Proceedings of the 37th German Conference on Pattern Recognition. Aachen, Germany (2015)Google Scholar
  7. 7.
    Bähr, M., Breuß, M., Quéau, Y., Boroujerdi, A.S., Durou, J.D.: Fast and accurate surface normal integration on non-rectangular domains. Comput. Vis. Media 3, 107–129 (2017)CrossRefGoogle Scholar
  8. 8.
    Balzer, J.: Shape from specular reflection in calibrated environments and the integration of spatial normal fields. In: Proceedings of the 18th IEEE International Conference on Image Processing, pp. 21–24. Brussels, Belgium (2011)Google Scholar
  9. 9.
    Balzer, J.: A Gauss–Newton method for the integration of spatial normal fields in shape space. J. Math. Imaging Vis. 44(1), 65–79 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Balzer, J., Mörwald, T.: Isogeometric finite-elements methods and variational reconstruction tasks in vision—a perfect match. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Providence, Rhode Island, USA (2012)Google Scholar
  11. 11.
    Chang, J.Y., Lee, K.M., Lee, S.U.: Multiview normal field integration using level set methods. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Workshop on Beyond Multiview Geometry: Robust Estimation and Organization of Shapes from Multiple Cues. Minneapolis, Minnesota, USA (2007)Google Scholar
  12. 12.
    Coleman, E.N., Jain, R.: Obtaining 3-dimensional shape of textured and specular surfaces using four-source photometry. Comput. Graph. Image Process. 18(4), 309–328 (1982)CrossRefGoogle Scholar
  13. 13.
    Crouzil, A., Descombes, X., Durou, J.D.: A multiresolution approach for shape from shading coupling deterministic and stochastic optimization. IEEE Trans. Pattern Anal. Mach. Intell. 25(11), 1416–1421 (2003)CrossRefGoogle Scholar
  14. 14.
    Du, Z., Robles-Kelly, A., Lu, F.: Robust surface reconstruction from gradient field using the L1 norm. In: Proceedings of the 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniques and Applications, pp. 203–209. Glenelg, Australia (2007)Google Scholar
  15. 15.
    Durou, J.D., Aujol, J.F., Courteille, F.: Integration of a normal field in the presence of discontinuities. In: Proceedings of the 7th International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 5681, pp. 261–273. Bonn, Germany (2009)Google Scholar
  16. 16.
    Durou, J.D., Courteille, F.: Integration of a normal field without boundary condition. In: Proceedings of the 11th IEEE International Conference on Computer Vision, 1st Workshop on Photometric Analysis for Computer Vision. Rio de Janeiro, Brazil (2007)Google Scholar
  17. 17.
    Ettl, S., Kaminski, J., Knauer, M.C., Häusler, G.: Shape reconstruction from gradient data. Appl. Opt. 47(12), 2091–2097 (2008)CrossRefGoogle Scholar
  18. 18.
    Fraile, R., Hancock, E.R.: Combinatorial surface integration. In: Proceedings of the 18th International Conference on Pattern Recognition, vol. I, pp. 59–62. Hong Kong (2006)Google Scholar
  19. 19.
    Frankot, R.T., Chellappa, R.: A method for enforcing integrability in shape from shading algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 10(4), 439–451 (1988)CrossRefzbMATHGoogle Scholar
  20. 20.
    Galliani, S., Breuß, M., Ju, Y.C.: Fast and robust surface normal integration by a discrete Eikonal equation. In: Proceedings of the 23rd British Machine Vision Conference. Guildford, UK (2012)Google Scholar
  21. 21.
    Goldman, D.B., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and spatially-varying BRDFs from photometric stereo. In: Proceedings of the 10th IEEE International Conference on Computer Vision, vol. I. Beijing, China (2005)Google Scholar
  22. 22.
    Grisvard, P.: Elliptic Problems in Nonsmooth Domains, Monographs and Studies in Mathematics. Pitman, London (1985)zbMATHGoogle Scholar
  23. 23.
    Harker, M., O’Leary, P.: Least squares surface reconstruction from measured gradient fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, Alaska, USA (2008)Google Scholar
  24. 24.
    Harker, M., O’Leary, P.: Least squares surface reconstruction from gradients: direct algebraic methods with spectral, Tikhonov, and constrained regularization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Colorado Springs, Colorado, USA (2011)Google Scholar
  25. 25.
    Harker, M., O’Leary, P.: Regularized reconstruction of a surface from its measured gradient field. J. Math. Imaging Vis. 51(1), 46–70 (2015)CrossRefzbMATHGoogle Scholar
  26. 26.
    Healey, G., Jain, R.: Depth recovery from surface normals. In: Proceedings of the 7th International Conference on Pattern Recognition, pp. 894–896. Montreal, Canada (1984)Google Scholar
  27. 27.
    Ho, J., Lim, J., Yang, M.H.: Integrating surface normal vectors using fast marching method. In: Proceedings of the 9th European Conference on Computer Vision, vol. III. Lecture Notes in Computer Science, vol. 3953, pp. 239–250. Graz, Austria (2006)Google Scholar
  28. 28.
    Horn, B.K.P.: Robot Vision. MIT Press, Cambridge (1986)Google Scholar
  29. 29.
    Horn, B.K.P., Brooks, M.J.: The variational approach to shape from shading. Comput. Vis. Graph. Image Process. 33(2), 174–208 (1986)CrossRefzbMATHGoogle Scholar
  30. 30.
    Horovitz, I., Kiryati, N.: Bias correction in photometric stereo using control points. Technical report, Department of Electrical Engineering Systems, Tel Aviv University, Israel (2000)Google Scholar
  31. 31.
    Horovitz, I., Kiryati, N.: Depth from gradient fields and control points: bias correction in photometric stereo. Image Vis. Comput. 22(9), 681–694 (2004)CrossRefGoogle Scholar
  32. 32.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)CrossRefGoogle Scholar
  33. 33.
    Ikeuchi, K.: Constructing a depth map from images. Technical Memo AIM-744, Artificial Intelligence Laboratory, Massachusetts Institute of Technology. Cambridge, Massachusetts, USA (1983)Google Scholar
  34. 34.
    Karaçali, B., Snyder, W.: Reconstructing discontinuous surfaces from a given gradient field using partial integrability. Comput. Vis. Image Underst. 92(1), 78–111 (2003)CrossRefGoogle Scholar
  35. 35.
    Karaçali, B., Snyder, W.: Noise reduction in surface reconstruction from a given gradient field. Int. J. Comput. Vis. 60(1), 25–44 (2004)CrossRefGoogle Scholar
  36. 36.
    Kimmel, R., Yavneh, I.: An algebraic multigrid approach for image analysis. SIAM J. Sci. Comput. 24(4), 1218–1231 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Klette, R., Schlüns, K.: Height data from gradient fields. In: Proceedings of the Machine Vision Applications, Architectures, and Systems Integration, Proceedings of the International Society for Optical Engineering, vol. 2908, pp. 204–215. Boston, Massachusetts, USA (1996)Google Scholar
  38. 38.
    Kovesi, P.: Shapelets correlated with surface normals produce surfaces. In: Proceedings of the 10th IEEE International Conference on Computer Vision, vol. II, pp. 994–1001. Beijing, China (2005)Google Scholar
  39. 39.
    Lee, D.: A provably convergent algorithm for shape from shading. In: Proceedings of the DARPA Image Understanding Workshop, pp. 489–496. Miami Beach, Florida, USA (1985)Google Scholar
  40. 40.
    Miranda, C.: Partial Differential Equations of Elliptic Type, 2nd edn. Springer, Berlin (1970)CrossRefzbMATHGoogle Scholar
  41. 41.
    Ng, H.S., Wu, T.P., Tang, C.K.: Surface-from-gradients with incomplete data for single view modeling. In: Proceedings of the 11th IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil (2007)Google Scholar
  42. 42.
    Ng, H.S., Wu, T.P., Tang, C.K.: Surface-from-gradients without discrete integrability enforcement: a Gaussian kernel approach. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 2085–2099 (2010)CrossRefGoogle Scholar
  43. 43.
    Noakes, L., Kozera, R.: The 2-D leap-frog: integrability, noise, and digitization. In: Bertrand, G., Imiya, A., Klette, R. (eds.) Digital and Image Geometry. Lecture Notes in Computer Science, vol. 2243, pp. 352–364. Springer (2001)Google Scholar
  44. 44.
    Noakes, L., Kozera, R.: Nonlinearities and noise reduction in 3-source photometric stereo. J. Math. Imaging Vis. 18(2), 119–127 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Noakes, L., Kozera, R., Klette, R.: The lawn-mowing algorithm for noisy gradient vector fields. In: Vision Geometry VIII, Proceedings of the International Society for Optical Engineering, vol. 3811, pp. 305–316. Denver, Colorado, USA (1999)Google Scholar
  46. 46.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)CrossRefGoogle Scholar
  47. 47.
    Petrovic, N., Cohen, I., Frey, B.J., Koetter, R., Huang, T.S.: Enforcing integrability for surface reconstruction algorithms using belief propagation in graphical models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. I, pp. 743–748. Kauai, Hawaii, USA (2001)Google Scholar
  48. 48.
    Quéau, Y., Durou, J.D., Aujol, J.F.: Variational methods for normal integration (2017). https://arxiv.org/abs/1709.05965/
  49. 49.
    Reddy, D., Agrawal, A., Chellappa, R.: Enforcing integrability by error correction using l1-minimization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Miami, Florida, USA (2009)Google Scholar
  50. 50.
    Saracchini, R.F.V., Stolfi, J., Leitão, H.C.G., Atkinson, G.A., Smith, M.L.: Multi-scale depth from slope with weights. In: Proceedings of the 21st British Machine Vision Conference. Aberystwyth, UK (2010)Google Scholar
  51. 51.
    Saracchini, R.F.V., Stolfi, J., Leitão, H.C.G., Atkinson, G.A., Smith, M.L.: A robust multi-scale integration method to obtain the depth from gradient maps. Comput. Vis. Image Underst. 116(8), 882–895 (2012)CrossRefGoogle Scholar
  52. 52.
    Simchony, T., Chellappa, R., Shao, M.: Direct analytical methods for solving Poisson equations in computer vision problems. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 435–446 (1990)CrossRefzbMATHGoogle Scholar
  53. 53.
    Stoer, J., Burlirsch, R.: Introduction to Numerical Analysis. Text in Applied Mathematics. Springer, Berlin (2002)CrossRefGoogle Scholar
  54. 54.
    Tankus, A., Kiryati, N.: Photometric stereo under perspective projection. In: Proceedings of the 10th IEEE International Conference on Computer Vision, vol. I, pp. 611–616. Beijing, China (2005)Google Scholar
  55. 55.
    Wang, Y., Bu, J., Li, N., Song, M., Tan, P.: Detecting discontinuities for surface reconstruction. In: Proceedings of the 21st International Conference on Pattern Recognition, pp. 2108–2111. Tsukuba, Japan (2012)Google Scholar
  56. 56.
    Wei, T., Klette, R.: Depth recovery from noisy gradient vector fields using regularization. In: Proceedings of the 10th International Conference on Computer Analysis of Images and Patterns. Lecture Notes in Computer Science, vol. 2756, pp. 116–123. Groningen, The Netherlands (2003)Google Scholar
  57. 57.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 139–144 (1980)CrossRefGoogle Scholar
  58. 58.
    Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Proceedings of the 10th Asian Conference on Computer Vision, pp. 703–717. Queenstown, New Zealand (2010)Google Scholar
  59. 59.
    Wu, T.P., Tang, C.K.: Visible surface reconstruction from normals with discontinuity consideration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. II, pp. 1793–1800. New York, USA (2006)Google Scholar
  60. 60.
    Wu, Z., Li, L.: A line-integration based method for depth recovery from surface normals. Comput. Vis. Graph. Image Process. 43(1), 53–66 (1988)CrossRefGoogle Scholar
  61. 61.
    Xie, W., Zhang, Y., Wang, C.C.L., Chung, R.C.K.: Surface-from-gradients: an approach based on discrete geometry processing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Columbus, Ohio, USA (2014)Google Scholar
  62. 62.
    Yamaura, Y., Nanya, T., Imoto, H., Maekawa, T.: Shape reconstruction from a normal map in terms of uniform bi-quadratic B-spline surfaces. Comput. Aided Des. 63, 129–140 (2015)CrossRefGoogle Scholar
  63. 63.
    Yuille, A.L., Snow, D.: Shape and albedo from multiple images using integrability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 158–164. San Juan, Puerto Rico (1997)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Technical University MunichGarchingGermany
  2. 2.IRIT, Université de ToulouseToulouseFrance
  3. 3.IMB, Université de BordeauxBordeauxFrance
  4. 4.Institut Universitaire de FranceParisFrance

Personalised recommendations