Multimedia Tools and Applications

, Volume 74, Issue 2, pp 467–477 | Cite as

Depth map Super-Resolution based on joint dictionary learning

  • Li-Wei Liu
  • Liang-Hao WangEmail author
  • Ming Zhang


Although Time-of-Flight (ToF) camera can provide real-time depth information from a real scene, the resolution of depth map captured by ToF camera is rather limited compared to HD color cameras, and thus it cannot be directly used in 3D reconstruction. In order to handle this problem, this paper proposes a novel compressive sensing (CS) and dictionary learning based depth map super-resolution (SR) method, which transforms a low resolution depth map to a high resolution depth map. Different from previous depth map SR methods, this algorithm uses a joint dictionary learning method with both low and high resolution depth maps, and this method also builds a sparse vector classification method which is used in depth map SR. Experimental results show that the proposed method outperforms state-of-the-art methods for depth map super-resolution.


Depth map Super-resolution Joint dictionary learning Sparse expression 



This work was supported in part by the National Natural Science Foundation of China (Grant No. 61271338), the National High Technology Research and Development Program (863) of China (Grant No. 2012AA011505), the Zhejiang Provincial Natural Science Foundation of China (Grant No. Q14F010020), and the Open Projects Program of National Laboratory of Pattern Recognition of China (Grant No. 201306308).


  1. 1.
    3dv systems, z-cam,
  2. 2.
    Baker S, Kanade T (2002) Limits on super-resolution and how to break them. IEEE Trans Pattern Anal Mach Intell 24(9):1167–1183CrossRefGoogle Scholar
  3. 3.
    Baraniuk RG (2007) Compressive sensing. IEEE Signal Proc Mag 24(4):118–120CrossRefGoogle Scholar
  4. 4.
    Candès EJ (2006) Compressive sampling. Proc Int Congr Math Madrid Spain 3:1433–1452Google Scholar
  5. 5.
    Canesta Inc (2006) Canestavision electronic perception development kit,
  6. 6.
    Chang H, Yeung D, Xiong Y (2004) Super-resolution through neighbor embedding. IEEE Conference on Computer Vision and Pattern Recognition, 1:1–8Google Scholar
  7. 7.
    Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Freeman W, Jones T, Pasztor E (2002) Example-based superresolution. IEEE Comput Graph Appl 22(2):56–65CrossRefGoogle Scholar
  9. 9.
    Freeman W, Pasztor E, Carmichael O (2000) Learning lowlevel vision. Int J Comput Vis 40(1):25–47CrossRefzbMATHGoogle Scholar
  10. 10.
    Gao X, Zhang K, Li X, Tao D (2012) Joint learning for single-image super-resolution via a coupled constraint. IEEE Trans Image Process 21(2):469–480CrossRefMathSciNetGoogle Scholar
  11. 11.
    Geman S, Geman D (1984) Stochastic relaxation, gibbs distribution, and the bayesian restoration of images. IEEE Trans Pattern Anal Mach Intell 6(4):721–741CrossRefzbMATHGoogle Scholar
  12. 12.
    Glasner D, Bagon S, Irani M (2009) “Super-resolution from a single image” ICCV pp. 349–356Google Scholar
  13. 13.
    Han Y, Wu F, Tian Q, Zhuang Y (2012) Image annotation by input-output structural grouping sparsity. IEEE Trans Image Process (IEEE T-IP) 21(6):3066–3079CrossRefMathSciNetGoogle Scholar
  14. 14.
    Han Y, Yang Y, Ma Z, Shen H, Sebe N, Zhou X (2014) Image attribute adaptation. IEEE Trans Multimed (IEEE T-MM). doi: 10.1109/TMM.2014.2306092 Google Scholar
  15. 15.
    Kopf J, Cohen MF, Lischinski D, Uyttendaele M (2007) Joint bilateral upsampling. ACM Trans Graph 26(3):96CrossRefGoogle Scholar
  16. 16.
    PMD camcube (2009)
  17. 17.
    Protter M, Elad M, Takeda H, Milanfar P (2009) Generaliizing the nonlocal-means to super-resolution resconstruction. IEEE Trans Image Process 18(1):36–51CrossRefMathSciNetGoogle Scholar
  18. 18.
    Roweis S, Saul L (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326CrossRefGoogle Scholar
  19. 19.
    Scharstein D and Szeliski R (2002) Middlebury stereo evaluation-version 2,
  20. 20.
    Tang Y, Yuan Y, Yan P, Li X (2011) Single-image superresolution via local learning. Int J Mach Learn Cybern 6(9):15–23CrossRefGoogle Scholar
  21. 21.
    Tseng P, Yun S (2009) A coordinate gradient descent method for nonsmooth separable minimization. Math Program Ser B 117:387–423CrossRefzbMATHMathSciNetGoogle Scholar
  22. 22.
    Wang J, Zhu S, Gong Y (2009) Resolution enhancement based on learning the sparse association of image patches. Pattern Recogn Lett 31(1):1–10CrossRefGoogle Scholar
  23. 23.
    Wu F, Lu X, Zhang Y, Zhang Z, Yan S, Zhuang Y (2013) Cross-media semantic representation via Bi-directional learning to rank. Proc 2013 ACM Int Conf Multimed (ACM Multimedia, Full Paper) 877–886Google Scholar
  24. 24.
    Wu F, Zhou Y, Yang Y, Siliang T, Zhang Y, Zhuang Y (2014) Sparse multi-modal hashing. IEEE Trans Multimed 16(2):427–439CrossRefGoogle Scholar
  25. 25.
    Xu Z, Schwarte R, Heinol H, Buxbaum B, Ringbeck T (1998) Smart pixel C photonic mixer device (pmd), M2VIP 1998 - Int Conf Mechatron Mach Vision Pract 259–264Google Scholar
  26. 26.
    Yang J, Wright J, Huang T, Ma Y (2010) Image superresolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873CrossRefMathSciNetGoogle Scholar
  27. 27.
    Zhang L, Wu X (2006) An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans Image Process 15(8):2226–2238CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Institute of Information and Communication EngineeringZhejiang UniversityHangzhouChina

Personalised recommendations