Multimedia Tools and Applications

, Volume 77, Issue 7, pp 9003–9020 | Cite as

Probability contour guided depth map inpainting and superresolution using non-local total generalized variation

  • Hai-Tao Zhang
  • Jun Yu
  • Zeng-Fu Wang


This paper proposes an image-guided depth super-resolution framework to improve the quality of depth map captured by low-cost depth sensors, like the Microsoft Kinect. First, a contour-guided fast marching method is proposed to preprocess the raw depth map for recovering the missing data. Then, by using the non-local total generalized variation (NL-TGV) regularization, a convex optimization model is constructed to up-sample the preprocessed depth map to a high-resolution one. To preserve the sharpness of depth discontinuities, the color image and its multi-level segmentation information are utilized to assign the weights within the NL-TGV through a novel weight combining scheme. The texture energy from color image and local structure coherence around neighbor pixels in low-resolution depth map are applied to adjust the combination weights for further suppressing texture-transfer. Quantitative and qualitative evaluations of the proposed method on the Middlebury datasets and real-sensor datasets show the promising results in quality.


Depth map inpainting Depth map super-resolution Non-local generalized variation First-order primal dual algorithm 


  1. 1.
    Arbelaez P, Maire M, Fowlkes C, Malik J (2011) Contour detection and hierarchical image segmentation. IEEE Trans Pattern Anal Mach Intell 33(5):898–916CrossRefGoogle Scholar
  2. 2.
    Buades A, Coll B, Morel J-M (2011) Non-local means denoising. Image Processing On Line 1:6zbMATHGoogle Scholar
  3. 3.
    Chambolle A, Pock T (2011) A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vis 40(1):120–145MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Chan D, Buisman H, Theobalt C, Thrun S (2008) A noise-aware filter for real-time depth upsampling Proc. ECCV workshop multi-camera multi-modal sensor fus. Algorithms Appl.Google Scholar
  5. 5.
    Cui Y, Schuon S, Chan D, Thrun S, Theobalt C (2010) 3d shape scanning with a time-of-flight camera Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1173–1180Google Scholar
  6. 6.
    Diebel J, Thrun S (2005) An application of markov random fields to range sensing Advances in neural information processing systems (NIPS), vol 5, pp 291–298Google Scholar
  7. 7.
    Dollár P, Zitnick CL (2015) Fast edge detection using structured forests. IEEE Trans Pattern Anal Mach Intell 37(8):1558–1570CrossRefGoogle Scholar
  8. 8.
    Ferstl D, Reinbacher C, Ranftl R, Rüther M, Bischof H (2013) Image guided depth upsampling using anisotropic total generalized variation Proc Int. Conf. Comput. Vis. (ICCV), pp 993–1000Google Scholar
  9. 9.
    Ferstl D, Ruther M, Bischof H (2015) Variational depth superresolution using example-based edge representations Proc Int. Conf. Comput. Vis. (ICCV), pp 513–521Google Scholar
  10. 10.
    He K, Sun J, Tang X (2013) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409CrossRefGoogle Scholar
  11. 11.
    Hui T-W, Loy CC, Tang X (2016) Depth map super-resolution by deep multi-scale guidance Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 353–369Google Scholar
  12. 12.
    Kiechle M, Hawe S, Kleinsteuber M (2013) A joint intensity and depth co-sparse analysis model for depth map super-resolution Proc Int. Conf. Comput. Vis. (ICCV), pp 1545–1552Google Scholar
  13. 13.
    Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph 26(3):96CrossRefGoogle Scholar
  14. 14.
    Kwon H, Tai Y-W, Lin S (2015) Data-driven depth map refinement via multi-scale sparse representation Proc. Comput. Vis. Pattern Recognit (CVPR), pp 159–167Google Scholar
  15. 15.
    Li Y, Min D, Do MN, Lu J (2016) Fast guided global interpolation for depth and motion Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 717–733Google Scholar
  16. 16.
    Liu M-Y, Tuzel O, Taguchi Y (2013) Joint geodesic upsampling of depth images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 169–176Google Scholar
  17. 17.
    Liu W, Jia S, Li P, Chen X, Yang J, Wu Q (2015) An mrf-based depth upsampling: upsample the depth map with its own property. IEEE Signal Process Lett 22(10):1708–1712CrossRefGoogle Scholar
  18. 18.
    Lu J, Shi K, Min D, Lin L, Do MN (2012) Cross-based local multipoint filtering Proc. Comput. Vis. Pattern Recognit (CVPR), pp 430–437Google Scholar
  19. 19.
    Min D, Lu J, Do MN (2012) Depth video enhancement based on weighted mode filtering. IEEE Trans Image Process 21(3):1176–1190MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Min D, Choi S, Lu J, Ham B, Sohn K, Do MN (2014) Fast global image smoothing based on weighted least squares. IEEE Trans Image Process 23(12):5638–5653MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Nair R, Lenzen F, Meister S, Schäfer H, Garbe C, Kondermann D (2012) High accuracy tof and stereo sensor fusion at interactive rates Proc Eur. Conf. Comput. Vis. (ECCV), pp 1–11Google Scholar
  22. 22.
    Nair R, Ruhl K, Lenzen F, Meister S, Schäfer H, Garbe CS, Eisemann M, Magnor M, Kondermann D (2013) A survey on time-of-flight stereo fusion Time-of-Flight and depth imaging. Sensors, Algorithms, and Applications. Springer, pp 105–127Google Scholar
  23. 23.
    Parikh N, Boyd SP (2014) Proximal algorithms. Found Trends Optim 1 (3):127–239CrossRefGoogle Scholar
  24. 24.
    Park J, Kim H, Tai Y-W, Brown MS, Kweon I (2011) High quality depth map upsampling for 3d-tof cameras Proc Int. Conf. Comput. Vis. (ICCV). IEEE, pp 1623–1630Google Scholar
  25. 25.
    Park J, Kim H, Tai Y-W, Brown MS, Kweon IS (2014) High-quality depth map upsampling and completion for rgb-d cameras. IEEE Trans Image Process 23(12):5559–5572MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Pock T, Chambolle A (2011) Diagonal preconditioning for first order primal-dual algorithms in convex optimization Proc Int. Conf. Comput. Vis. (ICCV), pp 1762–1769Google Scholar
  27. 27.
    Ranftl R, Bredies K, Pock T (2014) Non-local total generalized variation for optical flow estimation Proc Eur. Conf. Comput. Vis. (ECCV), pp 439–454Google Scholar
  28. 28.
    Riegler G, Rüther M, Bischof H (2016) Atgv-net: accurate depth super-resolution Proc Eur. Conf. Comput. Vis. (ECCV). Springer, pp 268–284Google Scholar
  29. 29.
    Scharstein D, Pal C (2007) Learning conditional random fields for stereo Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8Google Scholar
  30. 30.
    Schuon S, Theobalt C, Davis J, Thrun S (2008) High-quality scanning using time-of-flight depth superresolution Proc. Comput. Vis. Pattern Recognit Workshop. (CVPRW), pp 1–7Google Scholar
  31. 31.
    Schuon S, Theobalt C, Davis J, Thrun S (2009) Lidarboost: Depth superresolution for tof 3d shape scanning Proc. Comput. Vis. Pattern Recognit. (CVPR), pp 343–350Google Scholar
  32. 32.
    Tan X, Sun C, Pham TD (2014) Multipoint filtering with local polynomial approximation and range guidance Proc. Comput. Vis. Pattern Recognit (CVPR), pp 2941–2948Google Scholar
  33. 33.
    Telea A (2004) An image inpainting technique based on the fast marching method. J Graphics Tools 9(1):23–34CrossRefGoogle Scholar
  34. 34.
    Wang L, Jin H, Yang R, Gong M (2008) Stereoscopic inpainting: joint color and depth completion from stereo images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8Google Scholar
  35. 35.
    Xiaofeng R, Bo L (2012) Discriminatively trained sparse code gradients for contour detection Advances in neural information processing systems (NIPS), pp 584–592Google Scholar
  36. 36.
    Yang Q, Yang R, Davis J, Nistér D (2007) Spatial-depth super resolution for range images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1–8Google Scholar
  37. 37.
    Yang Q, Ahuja N, Yang R, Tan K-H, Davis J, Culbertson B, Apostolopoulos J, Wang G (2013) Fusion of median and bilateral filtering for range image upsampling. IEEE Trans Image Process 22(12):4841–4852MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Yang J, Ye X, Li K, Hou C, Wang Y (2014) Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans Image Process 23 (8):3443–3458MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Yu L-F, Yeung S-K, Tai Y-W, Lin S (2013) Shading-based shape refinement of rgb-d images Proc. Comput. Vis. Pattern Recognit (CVPR), pp 1415–1422Google Scholar
  40. 40.
    Zhang S, Wang C, Chan S-C (2015) A new high resolution depth map estimation system using stereo vision and kinect depth sensing. J Signal Process Sys 79(1):19–31CrossRefGoogle Scholar
  41. 41.
    Zhu J, Wang L, Yang R, Davis JE, Pan Z (2011) Reliability fusion of time-of-flight depth and stereo geometry for high quality depth maps. IEEE Trans Pattern Anal Mach Intell 33(7):1400–1414CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.School of Information Science and TechnologyUniversity of Science and Technology of ChinaHefeiPeople’s Republic of China
  2. 2.School of Information and EngineeringSouthwest University of Science and TechnologyMianyangPeople’s Republic of China
  3. 3.Institute of Intelligent MachinesChinese Academy of SciencesHefeiPeople’s Republic of China

Personalised recommendations