Advertisement

Depth image upsampling based on guided filter with low gradient minimization

  • Hang YangEmail author
  • Zhongbo Zhang
Original Article
  • 1 Downloads

Abstract

In this paper, we present a novel upsampling framework to enhance the spatial resolution of the depth image. In our framework, the upscaling of a low-resolution depth image is guided by a corresponding intensity images; we formulate it as a cost aggregation problem with the guided filter. However, the guided filter does not make full use of the information of the depth image. Since depth images have quite sparse gradients, it inspires us to regularize the gradients for improving depth upscaling results. Statistics show a special property of depth images, that is, there is a non-ignorable part of pixels whose horizontal or vertical derivatives are equal to \(\pm 1\). Based on this special property, we propose a low gradient regularization method which reduces the penalty for horizontal or vertical derivative \(\pm 1\), and well describes the statistics of the depth image gradients. Then, we present a solution to the low gradient minimization problem based on threshold shrinkage. Finally, the proposed low gradient regularization is integrated with the guided filter into the depth image upsampling method. Experimental results demonstrate the effectiveness of our proposed approach both qualitatively and quantitatively compared with the state-of-the-art methods.

Keywords

Depth image Upsampling Low gradient minimization Guided filter Regularization method 

Notes

Compliance with ethical standards

Conflict of interest

We declare that we have no conflict of interest.

Supplementary material

371_2019_1748_MOESM1_ESM.pdf (209 kb)
Supplementary material 1 (pdf 209 KB)

References

  1. 1.
    Jalal, A., Kim, Y.: Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (2014)Google Scholar
  2. 2.
    Gupta, S., Girshick, R., Arbelaez, P., Malik, J.: Learning rich features from RGB-D images for object detection and segmentation. In: European Conference on Computer Vision (ECCV) (2014)Google Scholar
  3. 3.
    Jalal, A., Kamal, S., Kim, D.: A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors 14(7), 11735–11759 (2014)CrossRefGoogle Scholar
  4. 4.
    Jalal, A., Kamal, Y.H., Kim, Y.J.: Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern Recognit. 61, 295–308 (2017)CrossRefGoogle Scholar
  5. 5.
    Jalal, A., Uddin, M., Kim, T.S.: Depth video-based human activity recognition system using translation and scaling invariant features for life logging at smart home. IEEE Trans. Consum. Electron. 58(3), 863–871 (2012)CrossRefGoogle Scholar
  6. 6.
    Jalal, A., Uddin, M., Kim, J.T.: Recognition of human home activities via depth silhouettes and r transformation for smart homes. Indoor Built Environ. 21, 184–190 (2012)CrossRefGoogle Scholar
  7. 7.
    Riegler, G., Rüther, M., Bischof, H.: ATGV-net: accurate depth super-resolution. In: IEEE European Conference on Computer Vision (ECCV), pp. 268–284 (2016)Google Scholar
  8. 8.
    Lasang, P., Kumwilaisak, W., Liu, Y.: Optimal depth recovery using image guided tgv with depth confidence for high-quality view synthesis. J. Visual Commun. Image Represent. 39, 24–39 (2016)CrossRefGoogle Scholar
  9. 9.
    Hui, T.W., Chen, C.L., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: IEEE European Conference on Computer Vision (ECCV), pp. 353–369 (2016)Google Scholar
  10. 10.
    Ferstl, D., Reinbacher, C., Ranftl, R., Rüther, M., Bischof, H.: Image guided depth upsampling using anisotropic total generalized variation. In: IEEE International Conference on Computer Vision, pp. 993–1000 (2013)Google Scholar
  11. 11.
    Kwon, H., Tai, Y.W., Lin, S.: Data-driven depth map refinement via multi-scale spare representations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  12. 12.
    Yang, H., Zhang, Z., Guan, Y.: An adaptive parameter estimation for guided filter based image deconvolution. Signal Process. 138(1), 16–26 (2017)CrossRefGoogle Scholar
  13. 13.
    He, K., Sun, J., Tang, X.: Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1397–1409 (2013)CrossRefGoogle Scholar
  14. 14.
    Li, S., Kang, X., Hu, J.: Image fusion with guided filtering. IEEE Trans. Image Process. 22(7), 2864–2875 (2013)CrossRefGoogle Scholar
  15. 15.
    Ding, K., Wu, X., Chen, W.: Optimum inpainting for depth map based on l0 total variation. Visual Comput. 30(12), 1311–1320 (2014)CrossRefGoogle Scholar
  16. 16.
    Xue, H., Zhang, S., Cai, D.: Depth image inpainting: improving low rank matrix completion with low gradient regularization. IEEE Trans. Image Process. 26(9), 4311–4320 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Xu, L., Lu, C., Xu, Y.: Image smoothing via l0, gradient minimization. In: SIGGRAPH Asia Conference, ACM, p. 174. (2011)Google Scholar
  18. 18.
    Nguyen, R.M.H., Brown, M.S.: Fast and effective l0 gradient minimization by region fusion. In: IEEE International Conference on Computer Vision (ICCV), pp. 208–216 (2015)Google Scholar
  19. 19.
    Yang, J., Wright, J., Huang, T.S.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Timofte, R., De, V., Gool, L.V.: Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International Conference on Computer Vision, pp. 1920–1927 (2013)Google Scholar
  21. 21.
    Li, Y., Xue, T., Sun, L.: Joint example-based depth map super-resolution. In: IEEE International Conference on Multimedia and Expo, pp. 152–157 (2012)Google Scholar
  22. 22.
    Ferstl, D., Rüther, M., Bischof, H.: Variational depth superresolution using example-based edge representations. In: IEEE International Conference on Computer Vision, pp. 513–521 (2015)Google Scholar
  23. 23.
    Schulter, S., Leistner, C., Bischof, H.: Fast and accurate image upscaling with super-resolution forests. In: IEEE Computer Vision and Pattern Recognition, pp. 3791–3799 (2015)Google Scholar
  24. 24.
    Mahmoudi, M., Sapiro, G.: Sparse representations for range data restoration. IEEE Trans. Image Process. 21(5), 2909–2915 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Kopf, J., Cohen, M.F., Lischinski, D.: Joint bilateral upsampling. ACM Trans. Graph. 26(3), 96 (2011)CrossRefGoogle Scholar
  26. 26.
    Yang, Q., Yang, R., Davis, J.: Spatial-depth super resolution for range images. In: IEEE Computer Vision and Pattern Recognition, pp. 1–8 (2011)Google Scholar
  27. 27.
    Liu, M.Y., Tuzel, O., Taguchi, Y.: Joint geodesic upsampling of depth images. In: IEEE Computer Vision and Pattern Recognition, pp. 169–176 (2013)Google Scholar
  28. 28.
    Lu, J., Forsyth, D.: Sparse depth super resolution. In: IEEE Computer Vision and Pattern Recognition, pp. 2245–2253 (2015)Google Scholar
  29. 29.
    Li, Y., Min, D., Do, M.N.: Fast guided global interpolation for depth and motion. In: European Conference on Computer Vision, pp. 717–733 (2016)Google Scholar
  30. 30.
    Jung, C., Yu, S., Kim, J.: Intensity-guided edge-preserving depth upsampling through weighted l0 gradient minimization. J. Visual Commun. Image Represent. 42, 132–144 (2017)CrossRefGoogle Scholar
  31. 31.
    Diebel, J., Thrun, S.: An application of markov random fields to range sensing. Adv. Neural Inf. Process. Syst. 2006, 291–298 (2006)Google Scholar
  32. 32.
    Park, J., Kim, H., Tai, Y.W.: High quality depth map upsampling for 3d-tof cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 1623–1630 (2011)Google Scholar
  33. 33.
    Aodha, M., Campbell, N.D.F., Nair, A.: Patch based synthesis for single depth image super-resolution. In: European Conference on Computer Vision (ECCV), pp. 71–84 (2012)Google Scholar
  34. 34.
    Yang, J., Ye, X., Li, K.: Color-guided depth recovery from rgb-d data using an adaptive autoregressive model. IEEE Trans. Image Process. 23(8), 3443–3458 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Dong, C., Chen, C.L., He, K.: Learning a deep convolutional network for image super-resolution. In: European Conference on Computer Vision (ECCV), pp. 184–199 (2014)Google Scholar
  36. 36.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  37. 37.
    Dong, C., Chen, C.L., He, K.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2014)CrossRefGoogle Scholar
  38. 38.
    Xie, J., Feris, R.S., Sun, M.T.: Edge-guided single depth image super resolution. IEEE Trans. Image Process. 25(1), 428–438 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Handa, A., Patraucean, V., Badrinarayanan, V.: Understanding real world indoor scenes with synthetic data. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4077–4085 (2016)Google Scholar
  40. 40.
    Hui, T.W., Loy, C.C., Tang, X.: Depth map super-resolution by deep multi-scale guidance. In: European Conference on Computer Vision (ECCV), pp. 353–369 (2014)Google Scholar
  41. 41.
    Storath, M., Weinmann, A., Demaret, L.: Jump-sparse and sparse recovery using potts functionals. IEEE Trans. Signal Process. 62(14), 3654–3666 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Wang, Y., Yang, Y., Yin, W.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1(3), 248–272 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Lu, J., Shi, K., Min, D., Lin, L., Do, M.N.: Cross-based local multipoint filtering. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 430–437 (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Changchun Institute of Optics, Fine Mechanics and PhysicsChinese Academy of ScienceChangchunChina
  2. 2.Departments of MathematicsJilin UniversityChangchunChina

Personalised recommendations