Real-Time Large-Scale Dense 3D Reconstruction with Loop Closure

  • Olaf Kähler
  • Victor A. PrisacariuEmail author
  • David W. Murray
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9912)


In the highly active research field of dense 3D reconstruction and modelling, loop closure is still a largely unsolved problem. While a number of previous works show how to accumulate keyframes, globally optimize their pose on closure, and compute a dense 3D model as a post-processing step, in this paper we propose an online framework which delivers a consistent 3D model to the user in real time. This is achieved by splitting the scene into submaps, and adjusting the poses of the submaps as and when required. We present a novel technique for accumulating relative pose constraints between the submaps at very little computational cost, and demonstrate how to maintain a lightweight, scalable global optimization of submap poses. In contrast to previous works, the number of submaps grows with the observed 3D scene surface, rather than with time. In addition to loop closure, the paper incorporates relocalization and provides a novel way of assessing tracking quality.


Loop Closure Depth Image Relative Constraint Pairwise Constraint Camera Tracking 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work is partially supported by Huawei Technologies Co. Ltd. any by grant EP/J014990 from the UK’s Engineering and Physical Science Research Council.

Supplementary material

Supplementary material 1 (avi 19770 KB)


  1. 1.
    Newcombe, R.A., Davison, A.J.: Live dense reconstruction with a single moving camera. In: Proceedings 23rd IEEE Conference on Computer Vision and Pattern Recognition, pp. 1498–1505 (2010)Google Scholar
  2. 2.
    Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.: KinectFusion: real-time dense surface mapping and tracking. In: International Symposium on Mixed and Augmented Reality, pp. 127–136 (2011)Google Scholar
  3. 3.
    Whelan, T., Johannsson, H., Kaess, M., Leonard, J.J., McDonald, J.: Robust real-time visual odometry for dense RGB-D mapping. In: Proceedings of 2013 IEEE International Conference on Robotics and Automation, pp. 5724–5731 (2013)Google Scholar
  4. 4.
    Bylow, E., Sturm, J., Kerl, C., Kahl, F., Cremers, D.: Real-time camera tracking and 3D reconstruction using signed distance functions. In: Proceedings Robotics: Science and Systems Conference (2013)Google Scholar
  5. 5.
    Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Proceedings of 13th European Conference on Computer Vision, pp. 834–849 (2014)Google Scholar
  6. 6.
    Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamic fusion: reconstruction and tracking of non-rigid scenes in real time. In: Proceedings of 28th IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  7. 7.
    Pollefeys, M., van Gool, L., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Koch, R.: Visual modeling with a hand-held camera. Int. J. Comput. Vis. 59(3), 207–232 (2004)CrossRefGoogle Scholar
  8. 8.
    Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of 19th IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)Google Scholar
  9. 9.
    Davison, A.J., Reid, I.D., Molton, N.D., Stasse, O.: MonoSLAM: real-time single camera SLAM. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 1052–1067 (2007)CrossRefGoogle Scholar
  10. 10.
    Klein, G., Murray, D.W.: Parallel tracking and mapping for small AR workspaces. In: Proceedings of 6th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 225–234 (2007)Google Scholar
  11. 11.
    Newcombe, R.A., Lovegrove, S.J., Davison, A.J.: DTAM: dense tracking and mapping in real-time. In: International Conference on Computer Vision (ICCV), pp. 2320–2327 (2011)Google Scholar
  12. 12.
    Pradeep, V., Rhemann, C., Izadi, S., Zach, C., Bleyer, M., Bathiche, S.: MonoFusion: real-time 3D reconstruction of small scenes with a single web camera. In: International Symposium on Mixed and Augmented Reality, pp. 83–88 (2013)Google Scholar
  13. 13.
    Steinbruecker, F., Sturm, J., Cremers, D.: Volumetric 3D mapping in real-time on a CPU. In: International Conference on Robotics and Automation (ICRA), pp. 2021–2028 (2014)Google Scholar
  14. 14.
    Chen, J., Bautembach, D., Izadi, S.: Scalable real-time volumetric surface reconstruction. ACM Trans. Graph. 32(4), 113:1–113:16 (2013)zbMATHGoogle Scholar
  15. 15.
    Zeng, M., Zhao, F., Zheng, J., Liu, X.: Octree-based fusion for realtime 3D reconstruction. Graph. Models 75(3), 126–136 (2013)CrossRefGoogle Scholar
  16. 16.
    Nießner, M., Zollhöfer, M., Izadi, S., Stamminger, M.: Real-time 3D reconstruction at scale using voxel hashing. ACM Trans. Graph. 32(6), 169:1–169:11 (2013)CrossRefGoogle Scholar
  17. 17.
    Kähler, O., Prisacariu, V.A., Ren, C.Y., Sun, X., Torr, P.H., Murray, D.W.: Very high frame rate volumetric integration of depth images on mobile devices. IEEE Trans. Vis. Comput. Graph. (Proceedings International Symposium on Mixed and Augmented Reality 2015) 21(11), 1241–1250 (2015)Google Scholar
  18. 18.
    Whelan, T., Leutenegger, S., Moreno, R.S., Glocker, B., Davison, A.: Elasticfusion: dense SLAM without a pose graph. In: Proceedings of Robotics: Science and Systems (2015)Google Scholar
  19. 19.
    Endres, F., Hess, J., Sturm, J., Cremers, D., Burgard, W.: 3-D mapping with an RGB-D camera. IEEE Trans. Robot. 30(1), 177–187 (2014)CrossRefGoogle Scholar
  20. 20.
    Kerl, C., Sturm, J., Cremers, D.: Dense visual SLAM for RGB-D cameras. In: Intelligent Robots and Systems (IROS), pp. 2100–2106 (2013)Google Scholar
  21. 21.
    Meilland, M., Comport, A.I.: On unifying key-frame and voxel-based dense visual SLAM at large scales. In: Intelligent Robots and Systems (IROS), pp. 3677–3683 (2013)Google Scholar
  22. 22.
    Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part II. LNCS, vol. 8690, pp. 834–849. Springer, Heidelberg (2014)Google Scholar
  23. 23.
    Weise, T., Wismer, T., Leibe, B., Gool, L.V.: Online loop closure for real-time interactive 3D scanning. Comput. Vis. Image Underst. 115(5), 635–648 (2011). Special issue on 3D Imaging and ModellingCrossRefGoogle Scholar
  24. 24.
    Stuckler, J., Behnke, S.: Multi-resolution surfel maps for efficient dense 3D modeling and tracking. J. Vis. Commun. Image Representation 25(1), 137–147 (2014). Visual Understanding and Applications with RGB-D CamerasCrossRefGoogle Scholar
  25. 25.
    Fioraio, N., Taylor, J., Fitzgibbon, A., Stefano, L.D., Izadi, S.: Large-scale and drift-free surface reconstruction using online subvolume registration. In: Computer Vision and Pattern Recognition (CVPR), pp. 4475–4483 (2015)Google Scholar
  26. 26.
    Choi, S., Zhou, Q.Y., Koltun, V.: Robust reconstruction of indoor scenes. In: Computer Vision and Pattern Recognition (CVPR), pp. 5556–5565 (2015)Google Scholar
  27. 27.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 303–312 (1996)Google Scholar
  28. 28.
    Klingensmith, M., Dryanovski, I., Srinivasa, S., Xiao, J.: Chisel: Real time large scale 3D reconstruction onboard a mobile device using spatially hashed signed distance fields. In: Proceedings of Robotics: Science and Systems (2015)Google Scholar
  29. 29.
    Schops, T., Sattler, T., Hane, C., Pollefeys, M.: 3D modeling on the go: interactive 3D reconstruction of large-scale scenes on mobile devices. In: International Conference on 3D Vision (3DV), pp. 291–299 (2015)Google Scholar
  30. 30.
    Kümmerle, R., Grisetti, G., Strasdat, H., Konolige, K., Burgard, W.: g2o: a general framework for graph optimization. In: International Conference on Robotics and Automation (ICRA), pp. 3607–3613 (2011)Google Scholar
  31. 31.
    Whelan, T., Kaess, M., Johannsson, H., Fallon, M., Leonard, J.J., McDonald, J.: Real-time large-scale dense RGB-D SLAM with volumetric fusion. Int. J. Robot. Res. 34(4–5), 598626 (2015)Google Scholar
  32. 32.
    Eade, E., Drummond, T.: Unified loop closing and recovery for real time monocular SLAM. In: Proceedings of the British Machine Vision Conference, pp. 6.1–6.10 (2008)Google Scholar
  33. 33.
    Williams, B., Cummins, M., Neira, J., Newman, P., Reid, I., Tardos, J.: An image-to-map loop closing method for monocular SLAM. In: Proceedings of International Conference on Intelligent Robots and and Systems, pp. 2053–2059 (2008)Google Scholar
  34. 34.
    Estrada, C., Neira, J., Tardos, J.D.: Hierarchical SLAM: real-time accurate mapping of large environments. IEEE Trans. Robot. 21(4), 588–596 (2005)CrossRefGoogle Scholar
  35. 35.
    Vedaldi, A., Zisserman, A.: Efficient additive kernels via explicit feature maps. Pattern Anal. Mach. Intell. 34(3), 480–492 (2011)CrossRefGoogle Scholar
  36. 36.
    Glocker, B., Izadi, S., Shotton, J., Criminisi, A.: Real-time RGB-D camera relocalization. In: International Symposium on Mixed and Augmented Reality (ISMAR), pp. 173–179 (2013)Google Scholar
  37. 37.
    Davis, T.A.: Direct Methods for Sparse Linear Systems. SIAM Series on Fundamentals of Algorithms. PWS Publishing, New York (2006)CrossRefzbMATHGoogle Scholar
  38. 38.
    Zhou, Q.Y., Koltun, V.: Dense scene reconstruction with points of interest. ACM Trans. Graph. 32(4), 112:1–112:8 (2013)zbMATHGoogle Scholar
  39. 39.
    Handa, A., Whelan, T., McDonald, J., Davison, A.: A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In: International Conference on Robotics and Automation, pp. 1524–1531 (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Olaf Kähler
    • 1
  • Victor A. Prisacariu
    • 1
    Email author
  • David W. Murray
    • 1
  1. 1.Department of Engineering ScienceUniversity of OxfordOxfordUK

Personalised recommendations