Real-Time Spatial and Depth Upsampling for Range Data

  • Xueqin Xiang
  • Guangxia Li
  • Jing Tong
  • Mingmin Zhang
  • Zhigeng Pan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6670)


Current active 3D range sensors, such as time-of-flight cameras, enable acquiring of range maps at video frame rate. Unfortunately, the resolution of the range maps is quite limited and the captured data are typically contaminated by noise. We therefore present a simple pipeline to enhance the quality as well as improve the spatial and depth resolution of range data in real time. To improve the spatial resolution of range data, we first upsample the depth information with the data from high resolution video camera. And then, a new strategy is utilized to increase the sub-pixel accuracy. We show that these techniques can greatly improve the reconstruction quality, boost the resolution of the range data to that of video sensor while achieving high computational efficiency for a real-time application.


Time-of-Flight Camera Super Resolution Fast Multi-Lateral Filter Sub-Pixel Estimation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ogger, T., Griesbach, K., et al.: 3D-Imaging in real-time with miniaturized optical range camera. In: Proc. OPTO, pp. 89–94 (2004)Google Scholar
  2. 2.
    SwissRangerTM SR-4000, MESA Imaging inc.,
  3. 3.
    Yang, Q., Wang, L., Yang, R., Stewnius, H., Nistr, D.: Stereo matching with color-weighted correlation, hierarchical belief propagation, and Occlusion Handling. IEEE Trans. PAMI 31(3), 492–504 (2009)CrossRefGoogle Scholar
  4. 4.
    Liang, C.-K., Cheng, C.-C., Lai, Y.-C., Chen, L.-G., Chen, H.: Hardware efficient belief propagation. In: Proc. CVPR, pp. 80–87 (2009)Google Scholar
  5. 5.
    Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: Proc. NIPS (2005)Google Scholar
  6. 6.
    Yang, Q., Yang, R., Davis, J.: Spatial-depth super resolution for range images. In: Proc. CVPR, pp. 1–8 (2007)Google Scholar
  7. 7.
    Tomasi, C., Manduchi, R.: Bilateral ltering for gray and color images. In: Proc. ICCV, pp. 839–846 (1998)Google Scholar
  8. 8.
    Kopf, J., Cohen, M., Lischinski, D., Uyttendaele, M.: Joint bilateral upsampling. ACM Transactions on Graphics (TOG) 26(3), 96(1–5) (2007)Google Scholar
  9. 9.
    Nvidia Corporation. CUDA: compute unified device architecture programming guide. Technical report (2008)Google Scholar
  10. 10.
    Schuon, S., Theobalt, C., Davis, J.: Thrun. S.: High-quality scanning using time-of-flight depth superresolution. In: Proc. CVPRW 2008, pp. 1–7 (2008)Google Scholar
  11. 11.
    Riemens, A.K., Gangwal, O.P., Barenbrug, B., Berretty, R.-P.M.: Multistep joint bilateral depth upsampling. In: SPIE 7257: Proc. VCIP (2009)Google Scholar
  12. 12.
    Porikli, F.: Constant time O(1) bilateral ltering. In: Proc. CVPR, pp. 1–8 (2008)Google Scholar
  13. 13.
    Chen, J., Paris, S., Durand, F.: Real-time edge-aware image processing with the bilateral grid. ACM Transactions on Graphics (TOG) 26(3), 103:1–10 (2007)Google Scholar
  14. 14.
    Yang, Q., Tan, H.-H., Ahuja, N.: Real-time O(1) bilateral ltering. In: Proc. CVPR, pp. 557–564 (2009)Google Scholar
  15. 15.
    Bhme, M., Haker, M., Martinetz, T., Barth, E.: Shading constraint improves accuracy of time-of-flight measurements. In: Proc. CVPR 2008 (2008)Google Scholar
  16. 16.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A noise-aware lter for real-time depth upsampling. In: M2SFA 208: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications (2008)Google Scholar
  17. 17.
    Weiss, B.: Fast median and bilateral ltering. In: Siggraph., vol. 25, pp. 519–526 (2006)Google Scholar
  18. 18.
    Gehrig, S.K., Franke, U.: Improving stereo sub-pixel accuracy for long range stereo. In: Proc. ICCV, pp. 1–7 (2007)Google Scholar
  19. 19.
    Shimizu, M., Okutomi, M.: Precise sub-pixel estimation on area-based matching. In: Proc. ICCV, pp. 90–97 (2001)Google Scholar
  20. 20.
    Szeliski, R., Scharstein, D.: Sampling the disparity space image. IEEE Trans. PAMI 26(3), 419–425 (2004)CrossRefGoogle Scholar
  21. 21.
    Zhu, J., Wang, L., Yang, R., Davis, J.: Fusion of time-of-flight depth and stereo for high accuracy depth maps. In: Proc. CVPR (2008)Google Scholar
  22. 22.
    Guizar-Sicairos, M., Thurman, S.T., Fienup, J.R.: Efficient subpixel image registration algorithms. Opt. Lett. 33, 156–158 (2008)CrossRefGoogle Scholar
  23. 23.
    Lindner, M., Lambers, M., Kolb, A.: Data fusion and edge-enhanced distance fefinement for 2D RGB and 3D range images. IJISTA, Special Issue on Dynamic 3D Imaging 5(1), 344–354 (2008)Google Scholar
  24. 24.
    Kutulakos, K.N., Steger, E.: A theory of refractive and specular 3D shape by light-path triangulation. In: Proc. ICCV, pp. 1448–1455 (2005)Google Scholar
  25. 25.
    Durand, F., Dorsey, J.: Fast bilateral filtering for the display of high-dynamic-range images. In: Siggraph., vol. 21, pp. 1–7 (2002)Google Scholar
  26. 26.
    Barash, D., Camiciu, D.: A common framework for nonlinear diffusion, adaptive smoothing, bilateral filtering and mean shift. Image and Vision Computing 22(1), 73–81 (2004)CrossRefGoogle Scholar
  27. 27.
    Alvarez, L., Deriche, R., Sanchez, J., Weickert, J.: Dense disparity map estimation respecting image discontinuities: A pde and scale-space based approach. Technical report. Research Report 3874, INRIA Sophia Antipolis, France (January 2000), 2,3Google Scholar
  28. 28.
    Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. IJCV 40(1), 25–47 (2000)CrossRefzbMATHGoogle Scholar
  29. 29.
  30. 30.
    Tseng, Y.C., Chang, N., Chang, T.S.: Low memory cost block-based belief propagation for stereo correspondence. In: Proc. ICME, pp. 1415–1418 (2007)Google Scholar
  31. 31.
    Tappen, M.F., Freeman, W.T.: Comparison of graph cuts with belief propagation for stereo, using identical MRF parameters. In: Proc. ICCV, pp. 900–906 (2003)Google Scholar
  32. 32.
    Byrd, R., Lu, P., Nocedal, J., Zhu. C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comp. 16(5), 1190–1208 (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Xueqin Xiang
    • 1
  • Guangxia Li
    • 1
  • Jing Tong
    • 1
  • Mingmin Zhang
    • 1
  • Zhigeng Pan
    • 1
  1. 1.State key Lab of Computer Aided Design and Computer GraphicsZhejiang UniversityHangzhouChina

Personalised recommendations