Skip to main content
SpringerLink
Log in

Depth image upsampling based on guided filter with low gradient minimization

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we present a novel upsampling framework to enhance the spatial resolution of the depth image. In our framework, the upscaling of a low-resolution depth image is guided by a corresponding intensity images; we formulate it as a cost aggregation problem with the guided filter. However, the guided filter does not make full use of the information of the depth image. Since depth images have quite sparse gradients, it inspires us to regularize the gradients for improving depth upscaling results. Statistics show a special property of depth images, that is, there is a non-ignorable part of pixels whose horizontal or vertical derivatives are equal to \(\pm 1\). Based on this special property, we propose a low gradient regularization method which reduces the penalty for horizontal or vertical derivative \(\pm 1\), and well describes the statistics of the depth image gradients. Then, we present a solution to the low gradient minimization problem based on threshold shrinkage. Finally, the proposed low gradient regularization is integrated with the guided filter into the depth image upsampling method. Experimental results demonstrate the effectiveness of our proposed approach both qualitatively and quantitatively compared with the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Jalal, A., Kim, Y.: Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (2014)

  2. Gupta, S., Girshick, R., Arbelaez, P., Malik, J.: Learning rich features from RGB-D images for object detection and segmentation. In: European Conference on Computer Vision (ECCV) (2014)

  3. Jalal, A., Kamal, S., Kim, D.: A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7), 11735–11759 (2014)

    Article  Google Scholar 

  4. Jalal, A., Kamal, Y.H., Kim, Y.J.: Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recognit. 61, 295–308 (2017)

    Article  Google Scholar 

  5. Jalal, A., Uddin, M., Kim, T.S.: Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE Trans. Consum. Electron. 58(3), 863–871 (2012)

    Article  Google Scholar 

  6. Jalal, A., Uddin, M., Kim, J.T.: Recognition of human home activities via depth silhouettes and r transformation for smart homes. Indoor Built Environ. 21, 184–190 (2012)

    Article  Google Scholar 

  7. Riegler, G., Rüther, M., Bischof, H.: ATGV-net: accurate depth super-resolution. In: IEEE European Conference on Computer Vision (ECCV), pp. 268–284 (2016)

  8. Lasang, P., Kumwilaisak, W., Liu, Y.: Optimal depth recovery using image guided tgv with depth confidence for high-quality view synthesis. J. Visual Commun. Image Represent. 39, 24–39 (2016)

    Article  Google Scholar 

  9. Hui, T.W., Chen, C.L., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: IEEE European Conference on Computer Vision (ECCV), pp. 353–369 (2016)

  10. Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: IEEE International Conference on Computer Vision, pp. 993–1000 (2013)

  11. Kwon, H., Tai, Y.W., Lin, S.: Data-driven depth map refinement via multi-scale spare representations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  12. Yang, H., Zhang, Z., Guan, Y.: An adaptive parameter estimation for guided filter based image deconvolution. Signal Process. 138(1), 16–26 (2017)

    Article  Google Scholar 

  13. He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)

    Article  Google Scholar 

  14. Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)

    Article  Google Scholar 

  15. Ding, K., Wu, X., Chen, W.: Optimum inpainting for depth map based on l0 total variation. Visual Comput. 30(12), 1311–1320 (2014)

    Article  Google Scholar 

  16. Xue, H., Zhang, S., Cai, D.: Depth image inpainting: improving low rank matrix completion with low gradient regularization. IEEE Trans. Image Process. 26(9), 4311–4320 (2017)

    Article  MathSciNet  Google Scholar 

  17. Xu, L., Lu, C., Xu, Y.: Image smoothing via l0, gradient minimization. In: SIGGRAPH Asia Conference, ACM, p. 174. (2011)

  18. Nguyen, R.M.H., Brown, M.S.: Fast and effective l0 gradient minimization by region fusion. In: IEEE International Conference on Computer Vision (ICCV), pp. 208–216 (2015)

  19. Yang, J., Wright, J., Huang, T.S.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  Google Scholar 

  20. Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International Conference on Computer Vision, pp. 1920–1927 (2013)

  21. Li, Y., Xue, T., Sun, L.: Joint example-based depth map super-resolution. In: IEEE International Conference on Multimedia and Expo, pp. 152–157 (2012)

  22. Ferstl, D., Rüther, M., Bischof, H.: Variational depth superresolution using example-based edge representations. In: IEEE International Conference on Computer Vision, pp. 513–521 (2015)

  23. Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: IEEE Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)

  24. Mahmoudi, M., Sapiro, G.: Sparse representations for range data restoration. IEEE Trans. Image Process. 21(5), 2909–2915 (2012)

    Article  MathSciNet  Google Scholar 

  25. Kopf, J., Cohen, M.F., Lischinski, D.: Joint bilateral upsampling. ACM Trans. Graph. 26(3), 96 (2011)

    Article  Google Scholar 

  26. Yang, Q., Yang, R., Davis, J.: Spatial-depth super resolution for range images. In: IEEE Computer Vision and Pattern Recognition, pp. 1–8 (2011)

  27. Liu, M.Y., Tuzel, O., Taguchi, Y.: Joint geodesic upsampling of depth images. In: IEEE Computer Vision and Pattern Recognition, pp. 169–176 (2013)

  28. Lu, J., Forsyth, D.: Sparse depth super resolution. In: IEEE Computer Vision and Pattern Recognition, pp. 2245–2253 (2015)

  29. Li, Y., Min, D., Do, M.N.: Fast guided global interpolation for depth and motion. In: European Conference on Computer Vision, pp. 717–733 (2016)

  30. Jung, C., Yu, S., Kim, J.: Intensity-guided edge-preserving depth upsampling through weighted l0 gradient minimization. J. Visual Commun. Image Represent. 42, 132–144 (2017)

    Article  Google Scholar 

  31. Diebel, J., Thrun, S.: An application of markov random fields to range sensing. Adv. Neural Inf. Process. Syst. 2006, 291–298 (2006)

    Google Scholar 

  32. Park, J., Kim, H., Tai, Y.W.: High quality depth map upsampling for 3d-tof cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 1623–1630 (2011)

  33. Aodha, M., Campbell, N.D.F., Nair, A.: Patch based synthesis for single depth image super-resolution. In: European Conference on Computer Vision (ECCV), pp. 71–84 (2012)

  34. Yang, J., Ye, X., Li, K.: Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans. Image Process. 23(8), 3443–3458 (2014)

    Article  MathSciNet  Google Scholar 

  35. Dong, C., Chen, C.L., He, K.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (ECCV), pp. 184–199 (2014)

  36. Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)

  37. Dong, C., Chen, C.L., He, K.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2014)

    Article  Google Scholar 

  38. Xie, J., Feris, R.S., Sun, M.T.: Edge-guided single depth image super resolution. IEEE Trans. Image Process. 25(1), 428–438 (2016)

    Article  MathSciNet  Google Scholar 

  39. Handa, A., Patraucean, V., Badrinarayanan, V.: Understanding real world indoor scenes with synthetic data. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4077–4085 (2016)

  40. Hui, T.W., Loy, C.C., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: European Conference on Computer Vision (ECCV), pp. 353–369 (2014)

  41. Storath, M., Weinmann, A., Demaret, L.: Jump-sparse and sparse recovery using potts functionals. IEEE Trans. Signal Process. 62(14), 3654–3666 (2014)

    Article  MathSciNet  Google Scholar 

  42. Wang, Y., Yang, Y., Yin, W.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1(3), 248–272 (2008)

    Article  MathSciNet  Google Scholar 

  43. Lu, J., Shi, K., Min, D., Lin, L., Do, M.N.: Cross-based local multipoint filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 430–437 (2012)

Download references

Author information

Authors and Affiliations

  1. Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Science, Changchun, 130033, China

    Hang Yang

  2. Departments of Mathematics, Jilin University, Changchun, 130012, China

    Zhongbo Zhang

Authors
  1. Hang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhongbo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hang Yang.

Ethics declarations

Conflict of interest

We declare that we have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 209 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, H., Zhang, Z. Depth image upsampling based on guided filter with low gradient minimization. Vis Comput 36, 1411–1422 (2020). https://doi.org/10.1007/s00371-019-01748-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01748-w

Keywords

Search

Navigation