Advertisement

Semantically Aware Urban 3D Reconstruction with Plane-Based Regularization

  • Thomas HolzmannEmail author
  • Michael Maurer
  • Friedrich Fraundorfer
  • Horst Bischof
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11218)

Abstract

We propose a method for urban 3D reconstruction, which incorporates semantic information and plane priors within the reconstruction process in order to generate visually appealing 3D models. We introduce a plane detection algorithm using 3D lines, which detects a more complete and less spurious plane set compared to point-based methods in urban environments. Further, the proposed normalized visibility-based energy formulation eases the combination of several energy terms within a tetrahedra occupancy labeling algorithm and, hence, is well suited for combining it with class specific smoothness terms. As a result, we produce visually appealing and detailed building models (i.e., straight edges and planar surfaces) and a smooth reconstruction of the surroundings.

Notes

Acknowledgements

This research was funded by the Austrian Science Fund (FWF) in the project V-MAV (I-1537). We thank Prof. Werner Lienhart and Slaven Kalenjuk from IGMS, TU Graz, Jesus Pestana and Christian Mostegel for providing datasets and Martin R. Oswald for discussion.

Supplementary material

474202_1_En_29_MOESM1_ESM.pdf (24.8 mb)
Supplementary material 1 (pdf 25356 KB)

Supplementary material 2 (mp4 32119 KB)

References

  1. 1.
    Blaha, M., Vogel, C., Richard, A., Wegner, J., Schindler, K., Pock, T.: Large-scale semantic 3D reconstruction: an adaptive multi-resolution model for multi-class volumetric labeling. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2016).  https://doi.org/10.1109/CVPR.2016.346
  2. 2.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Patt. Anal. Mach. Intell. 20(12), 1222–1239 (2001)CrossRefGoogle Scholar
  3. 3.
    CGAL. Computational Geometry Algorithms Library. http://www.cgal.org
  4. 4.
    Dzitsiuk, M., Sturm, J., Maier, R., Ma, L., Cremers, D.: De-noising, stabilizing and completing 3D reconstructions on-the-go using plane priors. In: International Conference on Robotics and Automation, May 2017Google Scholar
  5. 5.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. IEEE Trans. Patt. Anal. Mach. Intell. 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  6. 6.
    Garland, M., Heckbert, P.S.: Surface simplification using quadric error metrics. In: ACM Transactions on Graphics (SIGGRAPH), pp. 209–216. ACM Press/Addison-Wesley Publishing Co., New York (1997)Google Scholar
  7. 7.
    Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2013)Google Scholar
  8. 8.
    Häne, C., Heng, L., Lee, G.H., Sizov, A., Pollefeys, M.: Real-time direct dense matching on fisheye images using plane-sweeping stereo. In: International Conference on 3D Vision (3DV) (2014)Google Scholar
  9. 9.
    Hofer, M., Maurer, M., Bischof, H.: Efficient 3D scene abstraction using line segments. Comput. Vis. Image Underst. 157, 167–178 (2016).  https://doi.org/10.1016/j.cviu.2016.03.017CrossRefGoogle Scholar
  10. 10.
    Holzmann, T., Fraundorfer, F., Bischof, H.: Regularized 3D modeling from noisy building reconstructions. In: Fourth International Conference on 3D Vision, 3DV 2016, Stanford, CA, USA, 25–28 October 2016, pp. 528–536 (2016)Google Scholar
  11. 11.
    Holzmann, T., Oswald, M.R., Pollefeys, M., Fraundorfer, F., Bischof, H.: Plane-based surface regularization for urban 3D reconstruction. In: 28th British Machine Vision Conference, vol. 28 (2017)Google Scholar
  12. 12.
    Hoppe, C., Klopschitz, M., Donoser, M., Bischof, H.: Incremental surface extraction from sparse structure-from-motion point clouds. In: Proceedings British Machine Vision Conference (2013)Google Scholar
  13. 13.
    Jia, Y., et al.: Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014)
  14. 14.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Eurographics Symposium on Geometry Processing (2006)Google Scholar
  15. 15.
    Korč, F., Förstner, W.: eTRIMS Image Database for interpreting images of man-made scenes. Technical report, TR-IGG-P-2009-01, April 2009. http://www.ipb.uni-bonn.de/projects/etrims_db/
  16. 16.
    Labatut, P., Pons, J.P., Keriven, R.: Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In: Proceedings International Conference on Computer Vision (2007)Google Scholar
  17. 17.
    Labatut, P., Pons, J.P., Keriven, R.: Hierarchical shape-based surface reconstruction for dense multi-view stereo. In: International Workshop on 3-D Digital Imaging and Modeling (3DIM), ICCV Workshops, Kyoto, Japan, pp. 1598–1605, October 2009Google Scholar
  18. 18.
    Lafarge, F., Alliez, P.: Surface reconstruction through point set structuring. Comput. Graph. Forum 32(2), 225–234 (2013)CrossRefGoogle Scholar
  19. 19.
    Lafarge, F., Keriven, R., Brédif, M., Vu, H.: A hybrid multiview stereo algorithm for modeling urban scenes. IEEE Trans. Patt. Anal. Mach. Intell. 35(1), 5–17 (2013)CrossRefGoogle Scholar
  20. 20.
    Li, M., Wonka, P., Nan, L.: Manhattan-world urban reconstruction from point clouds. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 54–69. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46493-0_4CrossRefGoogle Scholar
  21. 21.
    Li, Y., Wu, X., Chrysanthou, Y., Sharf, A., Cohen-Or, D., Mitra, N.J.: GlobFit: consistently fitting primitives by discovering global relations. ACM Trans. Graph. 30(4), 52:1–52:12 (2011)Google Scholar
  22. 22.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  23. 23.
    Maurer, M., Hofer, M., Fraundorfer, F., Bischof, H.: Automated inspection of power line corridors to measure vegetation undercut using UAV-based images. In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences (2017)CrossRefGoogle Scholar
  24. 24.
    Monszpart, A., Mellado, N., Brostow, G., Mitra, N.: RAPter: rebuilding man-made scenes with regular arrangements of planes. In: ACM SIGGRAPH 2015 (2015)Google Scholar
  25. 25.
    Nan, L., Wonka, P.: PolyFit: polygonal surface reconstruction from point clouds. In: Proceedings International Conference on Computer Vision (2017)Google Scholar
  26. 26.
    Oesau, S., Lafarge, F., Alliez, P.: Planar shape detection and regularization in tandem. Comput. Graph. Forum 35(1), 203–215 (2016)CrossRefGoogle Scholar
  27. 27.
  28. 28.
    Rothermel, M., Wenzel, K., Fritsch, D., Haala, N.: Sure: photogrammetric surface reconstruction from imagery. In: Proceedings LC3D Workshop, Berlin (2012)Google Scholar
  29. 29.
    Schnabel, R., Wahl, R., Klein, R.: Efficient ransac for point-cloud shape detection. Comput. Graph. Forum 26(2), 214–226 (2007)CrossRefGoogle Scholar
  30. 30.
    Strecha, C., von Hansen, W., Gool, L.V., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition (2008)Google Scholar
  31. 31.
  32. 32.
    Vogel, C., Richard, A., Pock, T., Schindler, K.: Semantic 3D reconstruction with finite element bases. In: 28th British Machine Vision Conference, vol. 28 (2017)Google Scholar
  33. 33.
    Waechter, M., Moehrle, N., Goesele, M.: Let there be color! Large-scale texturing of 3D reconstructions. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 836–850. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_54CrossRefGoogle Scholar
  34. 34.
    Xiao, J., Furukawa, Y.: Reconstructing the world’s museums. Int. J. Comput. Vis. 110(3), 243–258 (2014)CrossRefGoogle Scholar
  35. 35.
    Zebedin, L., Bauer, J., Karner, K., Bischof, H.: Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5305, pp. 873–886. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-88693-8_64CrossRefGoogle Scholar
  36. 36.
    Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: IEEE Computer Vision and Pattern Recognition (CVPR), pp. 1529–1537 (2015).  https://doi.org/10.1109/ICCV.2015.179, arXiv: 1502.03240

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Thomas Holzmann
    • 1
    Email author
  • Michael Maurer
    • 1
  • Friedrich Fraundorfer
    • 1
  • Horst Bischof
    • 1
  1. 1.Institute of Computer Graphics and VisionGraz University of TechnologyGrazAustria

Personalised recommendations