A TV Prior for High-Quality Scalable Multi-View Stereo Reconstruction

Abstract

We present a scalable multi-view stereo method able to reconstruct accurate 3D models from hundreds of high-resolution input images. Local fusion of disparity maps obtained with semi-global matching enables the reconstruction of large scenes that do not fit into main memory. Since disparity maps may vary widely in quality and resolution, careful modeling of the 3D errors is crucial. We derive a sound stereo error model based on disparity uncertainty, which can vary spatially from tenths to several pixels. We introduce a feature based on total variation that allows pixel-wise classification of disparities into different error classes. For each class, we learn a disparity error distribution from ground-truth data using expectation maximization. We present a novel method for stochastic fusion of data with varying quality by adapting a multi-resolution volumetric fusion process that uses our error classes as a prior and models surface probabilities via an octree of voxels. Conflicts during surface extraction are resolved using visibility constraints and preference for voxels at higher resolutions. Experimental results on several challenging large-scale datasets demonstrate that our method yields improved performance both qualitatively and quantitatively.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

References

  1. Bailer, C., Finckh, M., & Lensch, H. (2012). Scale robust multi view stereo. In ECCV.

  2. Bao, S., Chandraker, M., Lin, Y., Savarese, S. (2013). Dense object reconstruction with semantic priors. In CVPR.

  3. Bodenmüller, T. (2009). Streaming surface reconstruction from real time 3D measurements. Ph.D. thesis, Technical University Munich.

  4. Curless, B., & Levoy, M. (1996). A volumetric method for building complex models from range images. In SIGGRAPH.

  5. Frahm, J.M., Georgel, P., Gallup, D., Johnson, T., Raguram, R., Wu, C., Jen, Y. H., Dunn, E., Clipp, B., Lazebnik, S., & Pollefeys, M. (2010). Building Rome on a cloudless day. In ECCV.

  6. Fuhrmann, S., & Goesele, M. (2011). Fusion of depth maps with multiple scales. In SIGGRAPH Asia.

  7. Fuhrmann, S., & Goesele, M. (2014). Floating scale surface reconstruction. In SIGGRAPH.

  8. Furukawa, R., Itano, T., Morisaka, A., & Kawasaki, H. (2007). Improved space carving method for merging and interpolating multiple range images using information of light sources of active stereo. In ACCV.

  9. Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multiview stereopsis. PAMI, 32, 1362–1376.

    Article  Google Scholar 

  10. Goesele, M., Curless, B., & Seitz, S. (2006). Multi-view stereo revisited. In CVPR.

  11. Goesele, M., Snavely, N., Curless, B., Hoppe, H., & Seitz, S. (2007). Multi-view stereo for community photo collections. In ICCV.

  12. Häne, C., Zach, C., Cohen, A., Angst, R., & Pollefeys, M. (2013). Joint 3D scene reconstruction and class segmentation. In CVPR.

  13. Hernández, C., Vogiatzis, G., & Cipolla, R. (2007). Probabilistic visibility for multi-view stereo. In CVPR.

  14. Hirschmüller, H. (2008). Stereo processing by semi-global matching and mutual information. PAMI, 30, 328–341.

    Article  Google Scholar 

  15. Hirschmüller, H., & Scharstein, D. (2009). Evaluation of stereo matching costs on images with radiometric differences. PAMI, 31, 1582–1599.

    Article  Google Scholar 

  16. Hu, X., Mordohai, P. (2012). Least commitment, viewpoint-based, multi-view stereo. In 3DIMPVT.

  17. Kazhdan, M., Bolitho, M., Hoppe, H. (2006). Poisson surface reconstruction. In Eurographics.

  18. Kazhdan, M., Klein, A., Dalal, K., Hoppe, H. (2007). Unconstrained isosurface extraction on arbitrary octrees. In Eurographics.

  19. Kolev, K., Klodt, M., Brox, T., & Cremers, D. (2009). Continuous global optimization in multiview 3D reconstruction. IJCV, 84, 80–96.

    Article  Google Scholar 

  20. Kuhn, A. (2014). Scalable 3D surface reconstruction by local stochastic fusion of disparity maps. Ph.D. thesis, University of the Bundeswehr.

  21. Kuhn, A., Hirschmüller, H., & Mayer, H. (2013). Multi-resolution range data fusion for multi-view stereo reconstruction. In GCPR.

  22. Kuhn, A., & Mayer, H. (2015). Incremental division of very large point clouds for scalable 3D surface reconstruction. In ICCV Workshop (ICCVW).

  23. Kuhn, A., Mayer, H., Hirschmüller, H., & Scharstein, D. (2014). A TV prior for high-quality local multi-view stereo reconstruction. In 3DV.

  24. Mayer, H., Bartelsen, J., Hirschmüller, H., & Kuhn, A. (2011). Dense 3D reconstruction from wide baseline image sets. In 15th International Workshop on Theoretical Foundations of Computer Vision.

  25. Merrell, P., Akbarzadeh, A., Wang, L., Mordohai, P., Frahm, J. M., Yang, R., Nistér, D., Pollefeys, M. (2007). Real-time visibility-based fusion of depth maps. In CVPR.

  26. Molton, N., & Brady, M. (2000). Practical structure and motion from stereo when motion is unconstrained. IJCV, 39(1), 5–23.

  27. Mücke, P., Klowsky, R., & Goesele, M. (2011). Surface reconstruction from multi-resolution sample points. In VMV.

  28. Newcombe, R., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A., Kohli, P., Shotton, J., Hodges, S., & Fitzgibbon, A. (2011). KinectFusion: Real-time dense surface mapping and tracking. In ISMAR.

  29. Ochs, P., Dosovitskiy, A., Brox, T., & Pock, T. (2013). An iterated L1 algorithm for non-smooth non-convex optimization in computer vision. In CVPR.

  30. Pathak, K., Birk, A., & Schwertfeger, S. (2007). 3D forward sensor modeling and application to occupancy grid based sensor fusion. In IROS.

  31. Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60(1), 259–268.

    MathSciNet  Article  MATH  Google Scholar 

  32. Sagawa, R., Nishino, K., & Ikeuchi, K. (2005). Adaptively merging large-scale range data with reflectance properties. PAMI, 27(3), 392–405.

    Article  Google Scholar 

  33. Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nesic, N., Wang, X., & Westling, P. (2014). High-resolution stereo datasets with subpixel-accurate ground truth. In GCPR.

  34. Scharstein, D., & Pal, C. (2007). Learning conditional random fields in stereo. In CVPR.

  35. Schroers, C., Zimmer, H., Valgaerts, L., Bruhn, A., Demetz, O., & Weickert, J. (2012). Anisotropic range image integration. In DAGM.

  36. Seitz, S., Curless, B., Diebel, J., Scharstein, D.,&Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR.

  37. Sinha, S., Scharstein, D., & Szeliski, R. (2014). Efficient high-resolution stereo matching using local plane sweeps. In CVPR.

  38. Steinbrücker, F., Kerl, C., Sturm, J., & Cremers, D. (2013). Large-scale multi-resolution surface reconstruction from RGB-D sequences. In ICCV.

  39. Strecha, C., von Hansen, W., Van Gool, L., Fua, P., Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In CVPR.

  40. Thrun, S. (2003). Learning occupancy grid maps with forward sensor models. Autonomous Robots, 15, 111–127.

    Article  Google Scholar 

  41. Vogiatzis, G., & Hernández, C. (2011). Video-based, real-time multi-view stereo. Image and Vision Computing, 29, 434–441.

    Article  Google Scholar 

  42. Vu, H. H., Labatut, P., Pons, J. P., & Keriven, R. (2012). High accuracy and visibility-consistent dense multiview stereo. PAMI, 34, 889–901.

    Article  Google Scholar 

  43. Wei, J., Resch, B., Lensch, H. (2014). Multi-view depth map estimation with cross-view consistency. In BMVC.

  44. Wheeler, M., Sato, Y., Ikeuchi, K. (1998). Consensus surfaces for modeling 3D objects from multiple range images. In ICCV.

  45. Woodford, O., & Vogiatzis, G. (2012). A generative model for online depth fusion. In ECCV.

  46. Wu, C. (2013). Towards linear-time incremental structure from motion. In 3DV.

  47. Wu, C., Agarwal, S., Curless, B., & Seitz, S. (2011). Multicore bundle adjustment. In CVPR.

  48. Xiong, Y., & Matthies, L. (1997). Error analysis of a real-time stereo system. In CVPR.

  49. Zach, C. (2008). Fast and high quality fusion of depth maps. In 3DPVT.

  50. Zach, C., Pock, T., & Bischof, H. (2007). A globally optimal algorithm for robust TV-L1 range image integration. In ICCV.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Andreas Kuhn.

Additional information

Communicated by Lourdes Agapito, Hiroshi Kawasaki, Katsushi Ikeuchi and Martial Hebert.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 11706 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kuhn, A., Hirschmüller, H., Scharstein, D. et al. A TV Prior for High-Quality Scalable Multi-View Stereo Reconstruction. Int J Comput Vis 124, 2–17 (2017). https://doi.org/10.1007/s11263-016-0946-x

Download citation

Keywords

  • Multi-View Stereo
  • 3D Modeling
  • Scalable 3D Surface Reconstruction