Advertisement

Computational Visual Media

, Volume 4, Issue 4, pp 367–383 | Cite as

3D floor plan recovery from overlapping spherical images

  • Giovanni PintoreEmail author
  • Fabio Ganovelli
  • Ruggero Pintus
  • Roberto Scopigno
  • Enrico Gobbetti
Open Access
Research Article
  • 100 Downloads

Abstract

We present a novel approach to automatically recover, from a small set of partially overlapping spherical images, an indoor structure representation in terms of a 3D floor plan registered with a set of 3D environment maps. We introduce several improvements over previous approaches based on color and spatial reasoning exploiting Manhattan world priors. In particular, we introduce a new method for geometric context extraction based on a 3D facet representation, which combines color distribution analysis of individual images with sparse multi-view clues. We also introduce an efficient method to combine the facets from different viewpoints in a single consistent model, taking into the reliability of the facet information. The resulting capture and reconstruction pipeline automatically generates 3D multi-room environments in cases where most previous approaches fail, e.g., in the presence of hidden corners and large clutter, without the need for additional dense 3D data or tools. We demonstrate the effectiveness and performance of our approach on different real-world indoor scenes. Our test data is available to allow further studies and comparisons.

Keywords

indoor reconstruction spherical panoramic cameras 360 degree photography multi-room environments 

Notes

Acknowledgements

This work was partially supported by projects VIGEC and 3DCLOUDPRO. The authors also acknowledge the contribution of the Sardinian Regional Authorities.

References

  1. [1]
    Kopf, J. 360 video stabilization. ACM Transactions on Graphics Vol. 35, No. 6, Article No. 195, 2016.Google Scholar
  2. [2]
    Matzen, K.; Cohen, M. F.; Evans, B.; Kopf, J.; Szeliski, R. Low-cost 360 stereo photography and video capture. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 148, 2017.Google Scholar
  3. [3]
    Brown, M.; Lowe, D. G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision Vol. 74, No. 1, 59–73, 2007.CrossRefGoogle Scholar
  4. [4]
    Pintore, G.; Garro, V.; Ganovelli, F.; Gobbetti, E.; Agus, M. Omnidirectional image capture on mobile devices for fast automatic generation of 2.5D indoor maps. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1–9, 2016.Google Scholar
  5. [5]
    Pintore, G.; Gobbetti, E. Effective mobile mapping of multi-room indoor structures. The Visual Computer Vol. 30, Nos. 6–8, 707–716, 2014.CrossRefGoogle Scholar
  6. [6]
    Pintore, G.; Ganovelli, F.; Gobbetti, E.; Scopigno, R. Mobile mapping and visualization of indoor structures to simplify scene understanding and location awareness. In: Computer Vision—ECCV 2016 Workshops. Lecture Notes in Computer Science, Vol. 9914. Hua, G.; Jégou, H. Eds. Springer Cham, 130–145, 2016.Google Scholar
  7. [7]
    Yang, H.; Zhang, H. Efficient 3D room shape recovery from a single panorama. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5422–5430, 2016.Google Scholar
  8. [8]
    Cabral, R.; Furukawa, Y. Piecewise planar and compact floorplan reconstruction from images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 628–635, 2014.Google Scholar
  9. [9]
    Pintore, G.; Ganovelli, F.; Pintus, R.; Scopigno, R.; Gobbetti, E. Recovering 3D indoor floor plans by exploiting low-cost spherical photography. In: Proceedings of the Pacific Graphics, 2018. Available at http://publications.crs4.it/pubdocs/2018/PGPSG18/pg2018s-indoorplan.pdf.Google Scholar
  10. [10]
    Xiong, X.; Adan, A.; Akinci, B.; Huber, D. Automatic creation of semantically rich 3D building models from laser scanner data. Automation in Construction Vol. 31, 325–337, 2013.CrossRefGoogle Scholar
  11. [11]
    Mura, C.; Mattausch, O.; Villanueva, A. J.; Gobbetti, E.; Pajarola, R. Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Computers & Graphics Vol. 44, 20–32, 2014.CrossRefGoogle Scholar
  12. [12]
    Mura, C.; Mattausch, O.; Pajarola, R. Piecewise-planar reconstruction of multi-room interiors with arbitrary wall arrangements. Computer Graphics Forum Vol. 35, No. 7, 179–188, 2016.CrossRefGoogle Scholar
  13. [13]
    Guo, R.; Hoiem, D. Support surface prediction in indoor scenes. In: Proceedings of the IEEE International Conference on Computer Vision, 2144–2151, 2013.Google Scholar
  14. [14]
    Jia, Z.; Gallagher, A.; Saxena, A.; Chen, T. 3Dbased reasoning with blocks, support, and stability. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2013.Google Scholar
  15. [15]
    Google. Tango. 2014. Available at www.google.com/atap/projecttango/.Google Scholar
  16. [16]
    Ikehata, S.; Yang, H.; Furukawa, Y. Structured indoor modeling. In: Proceedings of the IEEE International Conference on Computer Vision, 1323–1331, 2015.Google Scholar
  17. [17]
    Kim, Y. M.; Mitra, N. J.; Yan, D.-M.; Guibas, L. Acquiring 3D indoor environments with variability and repetition. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 138, 2012.Google Scholar
  18. [18]
    Nan, L.; Xie, K.; Sharf, A. A search-classify approach for cluttered indoor scene understanding. ACM Transactions on Graphics Vol. 31, No. 6, Article No. 137, 2012.Google Scholar
  19. [19]
    Autodesk. 123D Catch. Available at www.123dapp.com/catch.Google Scholar
  20. [20]
    Microsoft. Photosynth. Available at photosynth.net/.Google Scholar
  21. [21]
    Seitz, S. M.; Curless, B.; Diebel, J.; Scharstein, D.; Szeliski, R. A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 519–528, 2006.Google Scholar
  22. [22]
    Furukawa, Y.; Curless, B.; Seitz, S. M.; Szeliski, R. Reconstructing building interiors from images. In: Proceedings of the IEEE 12th International Conference on Computer Vision, 80–87, 2009.Google Scholar
  23. [23]
    Flint, A.; Murray, D.; Reid, I. Manhattan scene understanding using monocular, stereo, and 3D features. In: Proceedings of the International Conference on Computer Vision, 2228–2235, 2011.Google Scholar
  24. [24]
    Tsai, G.; Xu, C.; Liu, J.; Kuipers, B. Real-time indoor scene understanding using Bayesian filtering with motion cues. In: Proceedings of the International Conference on Computer Vision, 121–128, 2011.Google Scholar
  25. [25]
    Coughlan, J. M.; Yuille, A. L. Manhattan world: Compass direction from a single image by Bayesian inference. In: Proceedings of the 7th IEEE International Conference on Computer Vision, Vol. 2, 941–947, 1999.Google Scholar
  26. [26]
    Bao, S. Y.; Furlan, A.; Fei-Fei, L.; Savarese, S. Understanding the 3D layout of a cluttered room from multiple images. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 690–697, 2014.CrossRefGoogle Scholar
  27. [27]
    H¨ane, C.; Heng, L.; Lee, G. H.; Sizov, A.; Pollefeys, M. Real-time direct dense matching on fisheye images using plane-sweeping stereo. In: Proceedings of the 2nd International Conference on 3D Vision, 57–64, 2014.Google Scholar
  28. [28]
    Chang, P.; Hebert, M. Omni-directional structure from motion. In: Proceedings of the IEEE Workshop on Omnidirectional Vision, 127–133, 2000.CrossRefGoogle Scholar
  29. [29]
    Sch¨onbein, M.; Geiger, A. Omnidirectional 3D reconstruction in augmented Manhattan worlds. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 716–723, 2014.Google Scholar
  30. [30]
    Micusik, B.; Pajdla, T. Structure from motion with wide circular field of view cameras. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 28, No. 7, 1135–1149, 2006.CrossRefGoogle Scholar
  31. [31]
    Micusik, B.; Pajdla, T. Autocalibration & 3D reconstruction with non-central catadioptric cameras. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I-58–I-65, 2004.Google Scholar
  32. [32]
    Bunschoten, R.; Krose, B. Robust scene reconstruction from an omnidirectional vision system. IEEE Transactions on Robotics and Automation Vol. 19, No. 2, 351–357, 2003.CrossRefGoogle Scholar
  33. [33]
    Zingg, S.; Scaramuzza, D.; Weiss, S.; Siegwart, R. MAV navigation through indoor corridors using optical flow. In: Proceedings of the IEEE International Conference on Robotics and Automation, 3361–3368, 2010.Google Scholar
  34. [34]
    Li, S. Binocular spherical stereo. IEEE Transactions on Intelligent Transportation Systems Vol. 9, No. 4, 589–600, 2008.CrossRefGoogle Scholar
  35. [35]
    Geyer, C.; Daniilidis, K. A unifying theory for central panoramic systems and practical implications. In: Computer Vision—ECCV 2000. Lecture Notes in Computer Science, Vol. 1843. Vernon, D. Ed. Springer Berlin Heidelberg, 445–461, 2000.Google Scholar
  36. [36]
    Kim, H.; Hilton, A. 3D scene reconstruction from multiple spherical stereo pairs. International Journal of Computer Vision Vol. 104, No. 1, 94–116, 2013.MathSciNetCrossRefzbMATHGoogle Scholar
  37. [37]
    Im, S.; Ha, H.; Rameau, F.; Jeon, H.-G.; Choe, G.; Kweon, I. S. All-around depth from small motion with a spherical panoramic camera. In: Computer Vision— ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 156–172, 2016.CrossRefGoogle Scholar
  38. [38]
    Caruso, D.; Engel, J.; Cremers, D. Large-scale direct SLAM for omnidirectional cameras. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 141–148, 2015.Google Scholar
  39. [39]
    Pintore, G.; Pintus, R.; Ganovelli, F.; Scopigno, R.; Gobbetti, E. Recovering 3D existing-conditions of indoor structures from spherical images. Computers & Graphics Vol. 77, 16–29, 2018.CrossRefGoogle Scholar
  40. [40]
    Kangni, F.; Laganiere, R. Orientation and pose recovery from spherical panoramas. In: Proceedings of the IEEE 11th International Conference on Computer Vision, 1–8, 2007.Google Scholar
  41. [41]
    Achanta, R.; Shaji, A.; Smith, K.; Lucchi, A.; Fua, P.; S¨usstrunk, S. SLIC superpixels compared to stateof- the-art superpixel methods. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 34, No. 11, 2274–2282, 2012.CrossRefGoogle Scholar
  42. [42]
    Marroquim, R.; Kraus, M.; Cavalcanti, P. R. Efficient image reconstruction for point-based and line-based rendering. Computers & Graphics Vol. 32, No. 2, 189–203, 2008.CrossRefGoogle Scholar
  43. [43]
    Grompone von Gioi, R.; Jakubowicz, J.; Morel, J.- M.; Randall, G. LSD: A line segment detector. Image Processing On Line No. 2, 35–55, 2012.CrossRefGoogle Scholar
  44. [44]
    Douglas, D. H.; Peucker, T. K. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: The International Journal for Geographic Information and Geovisualization Vol. 10, No. 2, 112–122, 1973.CrossRefGoogle Scholar
  45. [45]
    Lee, D. C.; Hebert, M.; Kanade, T. Geometric reasoning for single image structure recovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2136–2143, 2009.Google Scholar
  46. [46]
    Zhang, Y.; Song, S.; Tan, P.; Xiao, J. PanoContext: A whole-room 3D context model for panoramic scene understanding. In: Computer Vision–ECCV 2014. Lecture Notes in Computer Science, Vol. 8694. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 668–686, 2014.Google Scholar
  47. [47]
    Schindler, G.; Dellaert, F. Atlanta world: An expectation maximization framework for simultaneous low-level edge grouping and camera calibration in complex man-made environments. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, I-203–I-209, 2004.Google Scholar
  48. [48]
    Schwing, A. G.; Urtasun, R. Efficient exact inference for 3D indoor scene understanding. In: Computer Vision–ECCV 2012. Lecture Notes in Computer Science, Vol. 7577. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 299–313, 2012.Google Scholar

Copyright information

© The Author(s) 2018

Open Access The articles published in this journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (https://doi.org/creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Other papers from this open access journal are available free of charge from https://doi.org/www.springer.com/journal/41095. To submit a manuscript, please go to https://doi.org/www.editorialmanager.com/cvmj.

Authors and Affiliations

  • Giovanni Pintore
    • 1
    Email author
  • Fabio Ganovelli
    • 1
  • Ruggero Pintus
    • 1
  • Roberto Scopigno
    • 1
  • Enrico Gobbetti
    • 2
  1. 1.CRS4Visual Computing GroupCagliariItaly
  2. 2.CNR-ISTIVisual Computing GroupPisaItaly

Personalised recommendations