Semantically Guided Depth Upsampling

  • Nick SchneiderEmail author
  • Lukas Schneider
  • Peter Pinggera
  • Uwe Franke
  • Marc Pollefeys
  • Christoph Stiller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9796)


We present a novel method for accurate and efficient upsampling of sparse depth data, guided by high-resolution imagery. Our approach goes beyond the use of intensity cues only and additionally exploits object boundary cues through structured edge detection and semantic scene labeling for guidance. Both cues are combined within a geodesic distance measure that allows for boundary-preserving depth interpolation while utilizing local context. We model the observed scene structure by locally planar elements and formulate the upsampling task as a global energy minimization problem. Our method determines globally consistent solutions and preserves fine details and sharp depth boundaries. In our experiments on several public datasets at different levels of application, we demonstrate superior performance of our approach over the state-of-the-art, even for very sparse measurements.


Markov Random Field Geodesic Distance Total Generalize Variation Sparse Measurement Depth Edge 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware filter for real-time depth upsampling. In: ECCV Workshops (2008)Google Scholar
  2. 2.
    Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected CRFs. In: ICLR (2015)Google Scholar
  3. 3.
    Chen, W., Hou, J., Zhang, M., Xiong, Z., Gao, H.: Semantic stereo: integrating piecewise planar stereo with segmentation and Classification. In: ICIST (2014)Google Scholar
  4. 4.
    Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)Google Scholar
  5. 5.
    Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS (2005)Google Scholar
  6. 6.
    Dollar, P., Zitnick, C.L.: Fast edge detection using structured forests. TPAMI 37(8), 1558–1570 (2015)CrossRefGoogle Scholar
  7. 7.
    Dolson, J., Baek, J., Plagemann, C., Thrun, S.: Upsampling range data in dynamic environments. In: CVPR (2010)Google Scholar
  8. 8.
    Everingham, M., Eslami, S.M.A., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual object classes challenge: a retrospective. IJCV 111, 98–136 (2015)CrossRefGoogle Scholar
  9. 9.
    Facciolo, G., Caselles, V.: Geodesic neighborhoods for piecewise affine interpolation of sparse data. In: ICIP (2009)Google Scholar
  10. 10.
    Ferstl, D., Reinbacher, C., Ranftl, R., Ruether, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: ICCV (2013)Google Scholar
  11. 11.
    Floros, G., Leibe, B.: Joint 2D-3D temporally consistent semantic segmentation of street scenes. In: CVPR (2012)Google Scholar
  12. 12.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the KITTI dataset. IJRR 32, 1231–1237 (2013)Google Scholar
  13. 13.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the KITTI vision Benchmark Suite. In: CVPR (2012)Google Scholar
  14. 14.
    Gong, X., Ren, J., Lai, B., Yan, C., Qian, H.: Guided depth upsampling via a cosparse analysis model. In: CVPR Workshops (2014)Google Scholar
  15. 15.
    Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: CVPR (2013)Google Scholar
  16. 16.
    Hawe, S., Kleinsteuber, M., Diepold, K.: Dense disparity maps from sparse disparity measurements. In: ICCV (2011)Google Scholar
  17. 17.
    He, K., Sun, J., Tang, X.: Guided image filtering. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part I. LNCS, vol. 6311, pp. 1–14. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  18. 18.
    Kiechle, M., Habigt, T., Hawe, S., Kleinsteuber, M.: A bimodal co-sparse analysis model for image processing. IJCV 114, 233–247 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kopf, J., Cohen, M.F., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Trans. Graph. (TOG) 26(3), 96 (2007). ACMCrossRefGoogle Scholar
  20. 20.
    Kundu, A., Li, Y., Dellaert, F., Li, F., Rehg, J.M.: Joint semantic segmentation and 3D reconstruction from monocular video. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 703–718. Springer, Heidelberg (2014)Google Scholar
  21. 21.
    Ladický, L., Shi, J., Pollefeys, M.: Pulling things out of perspective. In: CVPR (2014)Google Scholar
  22. 22.
    Ladický, L., Sturgess, P., Russell, C., Sengupta, S., Bastanlar, Y., Clocksin, W., Torr, P.H.S.: Joint optimization for object class segmentation and dense stereo reconstruction. IJCV 100, 122–133 (2011)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Li, Y., Xue, T., Sun, L., Liu, J.: Joint example-based depth map super-resolution. In: ICME (2012)Google Scholar
  24. 24.
    Lin, G., Shen, C., Reid, I., van dan Hengel, A.: Efficient piecewise trainingof deep structured models for semantic segmentation. Arxiv (2015)Google Scholar
  25. 25.
    Liu, J., Gong, X.: Guided depth enhancement via anisotropic diffusion. In: Huet, B., Ngo, C.-W., Tang, J., Zhou, Z.-H., Hauptmann, A.G., Yan, S. (eds.) PCM 2013. LNCS, vol. 8294, pp. 408–417. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  26. 26.
    Liu, M.Y., Tuzel, O., Taguchi, Y.: Joint geodesic upsampling of depth images. In: CVPR (2013)Google Scholar
  27. 27.
    Liu, Z., Li, X., Luo, P., Change, C., Tang, L.X.: Semantic image segmentation via deep parsing network. In: ICCV (2015)Google Scholar
  28. 28.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  29. 29.
    Lu, J., Forsyth, D.: Sparse depth super resolution. In: CVPR (2015)Google Scholar
  30. 30.
    Matsuo, K., Aoki, Y.: Depth image enhancement using local tangent plane approximations. In: CVPR (2015)Google Scholar
  31. 31.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)Google Scholar
  32. 32.
    Park, J., Kim, H., Tai, Y.-W., Brown, M.S., Kweon, I.: High quality depth map upsampling for 3D-TOF cameras. In: ICCV (2011)Google Scholar
  33. 33.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: EpicFlow: edge-preserving interpolation of correspondences for optical flow. In: CVPR (2015)Google Scholar
  34. 34.
    Scharstein, D., Hirschmüller, H., Kitajima, Y., Krathwohl, G., Nešić, N., Wang, X., Westling, P.: High-resolution stereo datasets with subpixel-accurate ground truth. In: Jiang, X., Hornegger, J., Koch, R. (eds.) GCPR 2014. LNCS, vol. 8753, pp. 31–42. Springer, Heidelberg (2014)Google Scholar
  35. 35.
    Yamaguchi, K., McAllester, D., Urtasun, R.: Efficient joint segmentation, occlusion labeling, stereo and flow estimation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part V. LNCS, vol. 8693, pp. 756–771. Springer, Heidelberg (2014)Google Scholar
  36. 36.
    Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR (2007)Google Scholar
  37. 37.
    Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., Torr, P.H.S.: Conditional random fields as recurrent neural networks. In: ICCV (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Nick Schneider
    • 1
    • 3
    Email author
  • Lukas Schneider
    • 1
    • 2
  • Peter Pinggera
    • 1
  • Uwe Franke
    • 1
  • Marc Pollefeys
    • 2
  • Christoph Stiller
    • 3
  1. 1.Environment Perception, Daimler R&DSindelfingenGermany
  2. 2.ETH ZurichZurichSwitzerland
  3. 3.Karlsruhe Institute of TechnologyKarlsruheGermany

Personalised recommendations