Abstract
The integration of surface normals for the purpose of computing the shape of a surface in 3D space is a classic problem in computer vision. However, even nowadays it is still a challenging task to devise a method that is flexible enough to work on non-trivial computational domains with high accuracy, robustness, and computational efficiency. By uniting a classic approach for surface normal integration with modern computational techniques, we construct a solver that fulfils these requirements. Building upon the Poisson integration model, we use an iterative Krylov subspace solver as a core step in tackling the task. While such a method can be very efficient, it may only show its full potential when combined with suitable numerical preconditioning and problem-specific initialisation. We perform a thorough numerical study in order to identify an appropriate preconditioner for this purpose. To provide suitable initialisation, we compute this initial state using a recently developed fast marching integrator. Detailed numerical experiments illustrate the benefits of this novel combination. In addition, we show on real-world photometric stereo datasets that the developed numerical framework is flexible enough to tackle modern computer vision applications.
Article PDF
Similar content being viewed by others
References
Pérez, P.; Gangnet, M.; Blake, A. Poisson image editing. ACM Transactions on Graphics Vol. 22, No. 3, 313–318, 2003.
Horn, B. K. P.; Brooks, M. J. The variational approach to shape from shading. Computer Vision, Graphics and Image Processing Vol. 33, No. 2, 174–208, 1986.
Woodham, R. J. Photometric method for determining surface orientation from multiple images. Optical Engineering Vol. 19, No. 1, 191139, 1980.
Zafeiriou, S.; Atkinson, G. A.; Hansen, M. F.; Smith, W. A. P.; Argyriou, V.; Petrou, M.; Smith, M. L.; Smith, L. N. Face recognition and verification using photometric stereo: The photoface database and a comprehensive evaluation. IEEE Transactions on Information Forensics and Security Vol. 8, No. 1, 121–135, 2013.
Smith, M. L.; Stamp, R. J. Automated inspection of textured ceramic tiles. Computers in Industry Vol. 43, No. 1, 73–82, 2000.
Esteban, C. H.; Vogiatzis, G.; Cipolla, R. Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 30, No. 3, 548–554, 2008.
Haque, S. M.; Chatterjee, A.; Govindu, V. M. High quality photometric reconstruction using a depth camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2275–2282, 2014.
Harker, M.; O’Leary, P. Least squares surface reconstruction from measured gradient fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–7, 2008.
Durou, J.-D.; Aujol, J.-F.; Courteille, F. Integrating the normal field of a surface in the presence of discontinuities. In: Energy Minimization Methods in Computer Vision and Pattern Recognition. Cremers, D.; Boykov, Y.; Blake, A.; Schmidt, F. R. Eds. Springer Berlin Heidelberg, 261–273, 2009.
Klette, R.; Schlüns, K. Height data from gradient fields. In: Proceedings of SPIE 2908, Machine Vision Applications, Architectures, and Systems Integration V, 204–215, 1996.
Coleman Jr., E. N.; Jain, R. Obtaining 3-dimensional shape of textured and specular surfaces using foursource photometry. Computer Graphics and Image Processing Vol. 18, No. 4, 309–328, 1982.
Wu, Z.; Li, L. A line-integration based method for depth recovery from surface normals. Computer Vision, Graphics and Image Processing Vol. 43, No. 1, 53–66, 1988.
Robles-Kelly, A.; Hancock, E. R. A graphspectral method for surface height recovery. Pattern Recognition Vol. 38, No. 8, 1167–1186, 2005.
Ho, J.; Lim, J.; Yang, M. H.; Kriegmann, D. Integrating surface normal vectors using fast marching method. In: Computer Vision–ECCV 2006. Leonardis, A.; Bischof, H.; Pinz, A. Eds. Springer Berlin Heidelberg, 239–250, 2006.
Galliani, S.; Breuß, M.; Ju, Y. C. Fast and robust surface normal integration by a discrete eikonal equation. In: Proceedings of the 23rd British Machine Vision Conference, 2012.
Bähr, M.; Breuß, M. An improved eikonal method for surface normal integration. In: Pattern Recognition. Gall, J.; Gehler, P.; Leibe, B. Eds. Springer International Publishing, 274–284, 2015.
Frankot, R. T.; Chellappa, R. A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 10, No. 4, 439–451, 1988.
Simchony, T.; Chellappa, R.; Shao, M. Direct analytical methods for solving Poisson equations in computer vision problems. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 12, No. 5, 435–446, 1990.
Wei, T.; Klette, R. A wavelet-based algorithm for height from gradients. In: Robot Vision. Klette, R.; Peleg, S.; Sommer, G. Eds. Springer Berlin Heidelberg, 84–90, 2001.
Kovesi, P. Shapelets correlated with surface normals produce surfaces. In: Proceedings of the 10th IEEE International Conference on Computer Vision, Vol. 2, 994–1001, 2005.
Wei, T.; Klette, R. Depth recovery from noisy gradient vector fields using regularization. In: Computer Analysis of Images and Patterns. Petkov, N.; Westenberg, M. A. Eds. Springer Berlin Heidelberg, 116–123, 2003.
Karaçali, B.; Snyder, W. Noise reduction in surface reconstruction from a given gradient field. International Journal on Computer Vision Vol. 60, No. 1, 25–44, 2004.
Agrawal, A.; Raskar, R.; Chellappa, R. What is the range of surface reconstructions from a gradient field? In: Computer Vision–ECCV 2006. Leonardis, A.; Bischof, H.; Pinz, A. Eds. Springer Berlin Heidelberg, 578–591, 2006.
Badri, H.; Yahia, H. M.; Aboutajdine, D. Robust surface reconstruction via triple sparsity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2283–2290, 2014.
Du, Z.; Robles-Kelly, A.; Lu, F. Robust surface reconstruction from gradient field using the L1 norm. In: Proceedings of the 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniques and Applications, 203–209, 2007.
Reddy, D.; Agrawal, A. K.; Chellappa, R. Enforcing integrability by error correction using l1-minimization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2350–2357, 2009.
Quéau, Y.; Durou, J.-D. Edge-preserving integration of a normal field: Weighted least squares, TV and L1 approaches. In: Scale Space and Variational Methods in Computer Vision. Aujol, J.-F.; Nikolova, M.; Papadakis, N. Eds. Springer International Publishing 576–588, 2015.
Harker, M.; O’Leary, P. Regularized reconstruction of a surface from its measured gradient field. Journal of Mathematical Imaging and Vision Vol. 51, No. 1, 46–70, 2015.
Breuß, M.; Quéau, Y.; Bähr, M.; Durou, J.-D. Highly efficient surface normal integration. In: Proceedings of the 20th Conference on Scientific Computing, 204–213, 2016.
Meister, A. Comparison of different Krylov subspace methods embedded in an implicit finite volume scheme for the computation of viscous and inviscid flow fields on unstructured grids. Journal of Computational Physics Vol. 140, No. 2, 311–345, 1998.
Saad, Y. Iterative Methods for Sparse Linear Systems. Society for Industrial and Applied Mathematics, 2003.
Durou, J.-D.; Courteille, F. Integration of a normal field without boundary condition. In: Proceedings of the 1st International Workshop on Photometric Analysis for Computer Vision, 2007.
Kimmel, R.; Sethian, J. A. Optimal algorithm for shape from shading and path planning. Journal of Mathematical Imaging and Vision Vol. 14, No. 3, 237–244, 2001.
Tsitsiklis, J. N. Efficient algorithms for globally optimal trajectories. IEEE Transactions on Automatic Control Vol. 40, No. 9, 1528–1538, 1995.
Sethian, J. A. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences of the United States of America Vol. 93, No. 4, 1591–1595, 1996.
Helmsen, J. J.; Puckett, E. G.; Colella, P.; Dorr, M. Two new methods for simulating photolithography development in 3D. In: Proceedings of SPIE 2726, Optical Microlithography IX, 253–261, 1996.
Sethian, J. A. Level Set Methods and Fast Marching Methods: Evolving Interfaces in Computational Geometry, Fluid Mechanics, Computer Vision, and Materials Science. Cambridge University Press, 1999.
Yatziv, L.; Bartesaghi, A.; Sapiro, G. O(N) implementation of the fast marching algorithm. Journal of Computational Physics Vol. 212, No. 2, 393–399, 2006.
Cacace, S.; Cristiani, E.; Falcone, M. Can local single-pass methods solve any stationary Hamilton–Jacobi–Bellman equation? SIAM Journal on Scientific Computing Vol. 36, No. 2, A570–A587, 2014.
Zimmer, H.; Bruhn, A.; Valgaerts, L.; Breuß, M.; Weickert, J.; Rosenhahn, B.; Seidel, H.-P. PDEbased anisotropic disparity-driven stereo vision. In: Proceddings of the 13th International Fall Workshop Vision, Modeling, and Visualization, 263–272, 2008.
Meister, A. Numerik Linearer Gleichungssysteme. Eine Einführung in Moderne Verfahren. Springer Spektrum, 2014.
Hestenes, M. R.; Stiefel, E. Methods of conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards Vol. 6, No. 49, 46–70, 1952.
Meurant, G. Computer Solution of Large Linear Systems. Elsevier Science, 1999.
Meurant, G. The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations. Society for Industrial and Applied Mathematics, 2006.
Golub, G. H.; van Loan, C. F. Matrix Computation, 3rd edn. Johns Hopkins, 1996.
Meijerink, J. A.; van der Vorst, H. A. An iterative solution method for linear systems of which the coefficient matrix is a symmetric M-matrix. Mathematics of Computation Vol. 31, No. 137, 148–162, 1977.
Kershaw, D. S. The incomplete Cholesky-conjugate gradient method for the iterative solution of systems of linear equations. Journal of Computational Physics Vol. 26, No. 1, 43–65, 1978.
Benzi, M. Preconditioning techniques for large linear systems: A survey. Journal of Computational Physics Vol. 182, No. 2, 418–477, 2002.
Kaasschieter, E. F. Preconditioned conjugate gradients for solving singular systems. Journal of Computational and Applied Mathematics Vol. 24, Nos. 1–2, 265–275, 1988.
Tang, J. M.; Vuik, C. Acceleration of preconditioned Krylov solvers for bubbly flow problems. In: Parallel Processing and Applied Mathematics. Wyrzykowski, R.; Dongarra, J.; Karczewski, K.; Wasniewski, J. Eds. Springer Berlin Heidelberg, 1323–1332, 2008.
Manteuffel, T. A. An incomplete factorization technique for positive definite linear systems. Mathematics of Computation Vol. 34, No. 150, 473–497, 1980.
Wang, Z.; Bovik, A. C.; Sheikh, H. R.; Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing Vol. 13, No. 4, 600–612, 2004.
Noakes, L.; Kozera, R. Nonlinearities and noise reduction in 3-source photometric stereo. Journal of Mathematical Imaging and Vision Vol. 18, No. 2, 119–127, 2003.
Chang, J. Y.; Lee, K. M.; Lee, S. U. Multiview normal field integration using level set methods. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2007.
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is published with open access at Springerlink.com
Martin Bähr is a Ph.D. student in mathematics at the Brandenburg Technical University in Germany. He received his master degree in applied mathematics at the same university in 2013. Since 2013, he works in the applied mathematics group with a scientific focus on mathematical image processing. His research interests include partial differential equations and numerical methods for image processing and computer vision.
Michael Breuß received his doctorate degree in mathematics from the University of Hamburg in 2001, and the habilitation in mathematics from the Technical University in Brunswick in 2006. For several years, he had been a member of the mathematical image analysis group in Saarbr¨ucken, Germany. Since 2016, he is professor for applied mathematics at the Brandenburg Technical University in Cottbus, Germany. His research interests are mainly in mathematical image processing and 3D vision, and include in particular numerical methods.
Yvain Quéau is a postdoctoral researcher at Technical University Munich. He received his Ph.D. degree in computer science from INPENSEEIHT, Université de Toulouse, in 2015. His research interests include 3D-reconstruction by photometric techniques (shape-from-shading and photometric stereo), as well as variational methods for solving computer vision and image processing problems.
Ali Sharifi Boroujerdi is a Ph.D. student at the Brandenburg Technical University in Germany. After being a bachelor of software engineering, he received his master degree in software engineering in 2013. His research interests include dynamic programming techniques as well as the field of artificial intelligence in general, especially deep learning, reinforcement learning, and big data analysis.
Jean-Denis Durou received his Ph.D. degree in computer science from the Université Paris Sud-Orsay in 1993, and the “Habilitation à Diriger les Recherches” from the Université Toulouse III-Paul Sabatier in 2007. He is an assistant professor at the Université Toulouse III since 1994, and a member of the VORTEX team at the IRIT Laboratory. His main research interest is 3D-vision. He is more specifically interested in photometric 3D-reconstruction, i.e., shape-from-shading and photometric stereo.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
Rights and permissions
Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Bähr, M., Breuß, M., Quéau, Y. et al. Fast and accurate surface normal integration on non-rectangular domains. Comp. Visual Media 3, 107–129 (2017). https://doi.org/10.1007/s41095-016-0075-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-016-0075-z