LS3D: Single-View Gestalt 3D Surface Reconstruction from Manhattan Line Segments

  • Yiming Qian
  • Srikumar Ramalingam
  • James H. ElderEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)


Recent deep learning algorithms for single-view 3D reconstruction recover rough 3D layout but fail to capture the crisp linear structures that grace our urban landscape. Here we show that for the particular problem of 3D Manhattan building reconstruction, the explicit application of linear perspective and Manhattan constraints within a classical constructive perceptual organization framework allows accurate and meaningful reconstructions to be computed. The proposed Line-Segment-to-3D (LS3D) algorithm computes a hierarchical representation through repeated application of the Gestalt principle of proximity. Edges are first organized into line segments, and the subset that conforms to a Manhattan frame is extracted. Optimal bipartite grouping of orthogonal line segments by proximity minimizes the total gap and generates a set of Manhattan spanning trees, each of which is then lifted to 3D. For each 3D Manhattan tree we identify the complete set of 3D 3-junctions and 3-paths, and show that each defines a unique minimal spanning cuboid. The cuboids generated by each Manhattan tree together define a solid model and the visible surface for that tree. The relative depths of these solid models are determined by an L1 minimization that is again rooted in a principle of proximity in both depth and image dimensions. The method has relatively fewer parameters and requires no training. For quantitative evaluation, we introduce a new 3D Manhattan building dataset (3DBM). We find that the proposed LS3D method generates 3D reconstructions that are both qualitatively and quantitatively superior to reconstructions produced by state-of-the-art deep learning approaches.



This research was supported by the NSERC Discovery program and the NSERC CREATE Training Program in Data Analytics & Visualization, the Ontario Research Fund, and the York University VISTA and Research Chair programs.


  1. 1.
    Coughlan, J.M., Yuille, A.L.: Manhattan world: orientation and outlier detection by Bayesian inference. Neural Comput. 15, 1063–1088 (2003)CrossRefGoogle Scholar
  2. 2.
    Kubovy, M., Wagemans, J.: Grouping by proximity and multistability in dot lattices: a quantitative Gestalt theory. Psychol. Sci. 6, 225–234 (1995)CrossRefGoogle Scholar
  3. 3.
    Kubovy, M., Holcombe, A.O., Wagemans, J.: On the lawfulness of grouping by proximity. Cogn. Psychol. 35, 71–98 (1998)CrossRefGoogle Scholar
  4. 4.
    Elder, J.H., Goldberg, R.M.: Ecological statistics of Gestalt laws for the perceptual organization of contours. J. Vis. 2, 324–353 (2002)CrossRefGoogle Scholar
  5. 5.
    Wagemans, J., et al.: A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization. Psychol. Bull. 138, 1172 (2012)CrossRefGoogle Scholar
  6. 6.
    Gupta, A., Efros, A.A., Hebert, M.: Blocks world revisited: image understanding using qualitative geometry and mechanics. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 482–496. Springer, Heidelberg (2010). Scholar
  7. 7.
    Roberts, L.G.: Machine perception of three-dimensional solids. Ph.D. thesis, Massachusetts Institute of Technology (1963)Google Scholar
  8. 8.
    Guzman, A.: Computer recognition of three-dimensional objects in a visual scene. Ph.D. thesis, MIT (1968)Google Scholar
  9. 9.
    Waltz, D.L.: Generating semantic descriptions from drawings of scenes with shadows. Technical Report AITR-271, MIT (1972)Google Scholar
  10. 10.
    Kanade, T.: A theory of Origami world. Artif. Intell. 13, 279–311 (1980)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Sugihara, K.: Machine Interpretation of Line Drawings, vol. 1. MIT Press, Cambridge (1986)Google Scholar
  12. 12.
    Hoiem, D., Efros, A.A., Hebert, M.: Recovering surface layout from an image. Int. J. Comput. Vis. 75, 151–172 (2007)CrossRefGoogle Scholar
  13. 13.
    Barinova, O., Konushin, V., Yakubenko, A., Lee, K.C., Lim, H., Konushin, A.: Fast automatic single-view 3-D reconstruction of urban scenes. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 100–113. Springer, Heidelberg (2008). Scholar
  14. 14.
    Haines, O., Calway, A.: Recognising planes in a single image. IEEE TPAMI 37, 1849–1861 (2015)CrossRefGoogle Scholar
  15. 15.
    Coughlan, J.M., Yuille, A.L.: Manhattan world: compass direction from a single image by Bayesian inference. In: CVPR, vol. 2, pp. 941–947 (1999)Google Scholar
  16. 16.
    Denis, P., Elder, J.H., Estrada, F.J.: Efficient edge-based methods for estimating Manhattan frames in urban imagery. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 197–210. Springer, Heidelberg (2008). Scholar
  17. 17.
    Tal, R., Elder, J.H.: An accurate method for line detection and Manhattan frame estimation. In: Park, J.-I., Kim, J. (eds.) ACCV 2012. LNCS, vol. 7729, pp. 580–593. Springer, Heidelberg (2013). Scholar
  18. 18.
    Delage, E., Lee, H., Ng, A.Y.: Automatic single-image 3D reconstructions of indoor Manhattan world scenes. In: Thrun, S., Brooks, R., Durrant-Whyte, H. (eds.) Robotics Research. STAR, vol. 28, pp. 305–321. Springer, Heidelberg (2007). Scholar
  19. 19.
    Hedau, V., Hoiem, D., Forsyth, D.: Recovering the spatial layout of cluttered rooms. In: ICCV, pp. 1849–1856 (2009)Google Scholar
  20. 20.
    Gupta, A., Hebert, M., Kanade, T., Blei, D.M.: Estimating spatial layout of rooms using volumetric reasoning about objects and surfaces. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) NIPS. Curran Associates, Inc. (2010)Google Scholar
  21. 21.
    Ramalingam, S., Pillai, J.K., Jain, A., Taguchi, Y.: Manhattan junction catalogue for spatial reasoning of indoor scenes. In: CVPR 2013, pp. 3065–3072 (2013)Google Scholar
  22. 22.
    Mallya, A., Lazebnik, S.: Learning informative edge maps for indoor scene layout prediction. In: ICCV, pp. 936–944 (2015)Google Scholar
  23. 23.
    Pero, L.D., Bowdish, J., Fried, D., Kermgard, B., Hartley, E., Barnard, K.: Bayesian geometric modeling of indoor scenes. In: CVPR, pp. 2719–2726 (2012)Google Scholar
  24. 24.
    Felzenszwalb, P.F., Veksler, O.: Tiered scene labeling with dynamic programming. In: CVPR, pp. 3097–3104 (2010)Google Scholar
  25. 25.
    Schwing, A.G., Urtasun, R.: Efficient exact inference for 3D indoor scene understanding. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 299–313. Springer, Heidelberg (2012). Scholar
  26. 26.
    Yang, H., Zhang, H.: Efficient 3D room shape recovery from a single panorama. In: CVPR, pp. 5422–5430 (2016)Google Scholar
  27. 27.
    Dasgupta, S., Fang, K., Chen, K., Savarese, S.: Delay: robust spatial layout estimation for cluttered indoor scenes. In: CVPR, pp. 616–624 (2016)Google Scholar
  28. 28.
    Ramalingam, S., Brand, M.: Lifting 3D Manhattan lines from a single image. In: ICCV, pp. 497–504 (2013)Google Scholar
  29. 29.
    Kushal, A., Seitz, S.M.: Single view reconstruction of piecewise swept surfaces. In: 3DV, pp. 239–246 (2013)Google Scholar
  30. 30.
    Saxena, A., Sun, M., Ng, A.Y.: Make3D: learning 3D scene structure from a single still image. IEEE TPAMI 31, 824–840 (2009)CrossRefGoogle Scholar
  31. 31.
    Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: CVPR, pp. 2650–2658 (2015)Google Scholar
  32. 32.
    Liu, F., Shen, C., Lin, G., Reid, I.: Learning depth from single monocular images using deep convolutional neural fields. IEEE TPAMI 38, 2024–2039 (2016)CrossRefGoogle Scholar
  33. 33.
    Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: 3DV, pp. 239–248 (2016)Google Scholar
  34. 34.
    Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimation from a single image. In: CVPR, pp. 5162–5170 (2015)Google Scholar
  35. 35.
    Zhuo, W., Salzmann, M., He, X., Liu, M.: 3D box proposals from a single monocular image of an indoor scene. In: AAAI (2018)Google Scholar
  36. 36.
    Fu, H., Gong, M., Wang, C., Batmanghelich, K., Tao, D.: Deep ordinal regression network for monocular depth estimation. In: CVPR (2018)Google Scholar
  37. 37.
    Xu, D., Ouyang, W., Wang, X., Sebe, N.: PAD-Net: multi-tasks guided prediction-and-distillation network for simultaneous depth estimation and scene parsing. In: CVPR (2018)Google Scholar
  38. 38.
    Qi, X., Liao, R., Liu, Z., Urtasun, R., Jia, J.: GeoNet: geometric neural network for joint depth and surface normal estimation. In: CVPR (2018)Google Scholar
  39. 39.
    Li, Z., Snavely, N.: MegaDepth: learning single-view depth prediction from internet photos. In: CVPR (2018)Google Scholar
  40. 40.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. Int. J. Rob. Res. 32, 1231–1237 (2013)CrossRefGoogle Scholar
  41. 41.
    Garg, R., B.G., V.K., Carneiro, G., Reid, I.: Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 740–756. Springer, Cham (2016). Scholar
  42. 42.
    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: CVPR (2017)Google Scholar
  43. 43.
    Izadinia, H., Shan, Q., Seitz, S.M.: IM2CAD. In: CVPR, pp. 2422–2431. IEEE (2017)Google Scholar
  44. 44.
    Almazan, E.J., Tal, R., Qian, Y., Elder, J.H.: MCMLSD: a dynamic programming approach to line segment detection. In: CVPR (2017)Google Scholar
  45. 45.
    Lee, D., Hebert, M., Kanade, T.: Geometric reasoning for single image structure recovery. In: CVPR, pp. 2136–2143. IEEE (2009)Google Scholar
  46. 46.
    Munkres, J.: Algorithms for the assignment and transportation problems. J. Soc. Ind. Appl. Math. 5, 32–38 (1957)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., Ranzuglia, G.: MeshLab: an open-source mesh processing tool. In: Eurographics Italian Chapter Conference (2008)Google Scholar
  48. 48.
    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NIPS, pp. 2366–2374 (2014)Google Scholar
  49. 49.
    Liu, C., Yang, J., Ceylan, D., Yumer, E., Furukawa, Y.: PlaneNet: piece-wise planar reconstruction from a single RGB image. In: CVPR, pp. 2579–2588 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Yiming Qian
    • 1
  • Srikumar Ramalingam
    • 2
  • James H. Elder
    • 1
    Email author
  1. 1.York UniversityTorontoCanada
  2. 2.University of UtahSalt Lake CityUSA

Personalised recommendations