Patch Based Synthesis for Single Depth Image Super-Resolution

  • Oisin Mac Aodha
  • Neill D. F. Campbell
  • Arun Nair
  • Gabriel J. Brostow
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7574)


We present an algorithm to synthetically increase the resolution of a solitary depth image using only a generic database of local patches. Modern range sensors measure depths with non-Gaussian noise and at lower starting resolutions than typical visible-light cameras. While patch based approaches for upsampling intensity images continue to improve, this is the first exploration of patching for depth images.

We match against the height field of each low resolution input depth patch, and search our database for a list of appropriate high resolution candidate patches. Selecting the right candidate at each location in the depth image is then posed as a Markov random field labeling problem. Our experiments also show how important further depth-specific processing, such as noise removal and correct patch normalization, dramatically improves our results. Perhaps surprisingly, even better results are achieved on a variety of real test scenes by providing our algorithm with only synthetic training depth data.


Depth Image Depth Discontinuity Pairwise Term Input Patch High Resolution Patch 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR (2007)Google Scholar
  2. 2.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: Lidarboost: Depth superresolution for tof 3D shape scanning. In: CVPR (2009)Google Scholar
  3. 3.
    Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. IJCV (2000)Google Scholar
  4. 4.
    Kuster, C., Popa, T., Zach, C., Gotsman, C., Gross, M.: Freecam: A hybrid camera system for interactive free-viewpoint video. In: Proceedings of Vision, Modeling, and Visualization, VMV (2011)Google Scholar
  5. 5.
    Holz, D., Schnabel, R., Droeschel, D., Stückler, J., Behnke, S.: Towards semantic scene analysis with time-of-flight cameras. In: RoboCup International Symposium (2010)Google Scholar
  6. 6.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-time human pose recognition in parts from single depth images. In: CVPR (2011)Google Scholar
  7. 7.
    Park, J., Kim, H., Tai, Y.W., Brown, M., Kweon, I.: High quality depth map upsampling for 3D-ToF cameras. In: ICCV (2011)Google Scholar
  8. 8.
    Fattal, R.: Upsampling via imposed edges statistics. SIGGRAPH (2007)Google Scholar
  9. 9.
    Yang, L., Sander, P.V., Lawrence, J., Hoppe, H.: Antialiasing recovery. ACM Transactions on Graphics (2011)Google Scholar
  10. 10.
    Irani, M., Peleg, S.: Improving resolution by image registration. In: CVGIP: Graph. Models Image Process. (1991)Google Scholar
  11. 11.
    Hahne, U., Alexa, M.: Exposure Fusion for Time-Of-Flight Imaging. In: Pacific Graphics (2011)Google Scholar
  12. 12.
    Rajagopalan, A.N., Bhavsar, A., Wallhoff, F., Rigoll, G.: Resolution Enhancement of PMD Range Maps. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 304–313. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  13. 13.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R.A., Kohli, P., Shotton, J., Hodges, S., Freeman, D., Davison, A.J., Fitzgibbon, A.: Kinectfusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: UIST (2011)Google Scholar
  14. 14.
    Cui, Y., Schuon, S., Derek, C., Thrun, S., Theobalt, C.: 3D shape scanning with a time-of-flight camera. In: CVPR (2010)Google Scholar
  15. 15.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. Computer Graphics and Applications (2002)Google Scholar
  16. 16.
    Freeman, W.T., Liu, C.: Markov random fields for super-resolution and texture synthesis. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011)Google Scholar
  17. 17.
    Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010)Google Scholar
  18. 18.
    Zontak, M., Irani, M.: Internal statistics of a single natural image. In: CVPR (2011)Google Scholar
  19. 19.
    Glasner, D., Bagon, S., Irani, M.: Super-resolution from a single image. In: ICCV (2009)Google Scholar
  20. 20.
    Sun, J., Zhu, J., Tappen, M.: Context-constrained hallucination for image super-resolution. In: CVPR (2010)Google Scholar
  21. 21.
    HaCohen, Y., Fattal, R., Lischinski, D.: Image upsampling via texture hallucination. In: ICCP (2010)Google Scholar
  22. 22.
    Gal, R., Shamir, A., Hassner, T., Pauly, M., Cohen-Or, D.: Surface reconstruction using local shape priors. In: Symposium on Geometry Processing (2007)Google Scholar
  23. 23.
    Golovinskiy, A., Matusik, W., Pfister, H., Rusinkiewicz, S., Funkhouser, T.: A statistical model for synthesis of detailed facial geometry. SIGGRAPH (2006)Google Scholar
  24. 24.
    Huang, J., Lee, A., Mumford, D.: Statistics of range images. In: CVPR (2000)Google Scholar
  25. 25.
    Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS (2005)Google Scholar
  26. 26.
    Chan, D., Buisman, H., Theobalt, C., Thrun, S.: A Noise-Aware Filter for Real-Time Depth Upsampling. In: Workshop on Multi-camera and Multi-modal Sensor Fusion Algorithms and Applications (2008)Google Scholar
  27. 27.
    Schuon, S., Theobalt, C., Davis, J., Thrun, S.: High-quality scanning using time-of-flight depth superresolution. In: CVPR Workshops (2008)Google Scholar
  28. 28.
    Reynolds, M., Doboš, J., Peel, L., Weyrich, T., Brostow, G.J.: Capturing time-of-flight data with confidence. In: CVPR (2011)Google Scholar
  29. 29.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. PAMI (2006)Google Scholar
  30. 30.
    Wainwright, M., Jaakkola, T., Willsky, A.: Map estimation via agreement on (hyper)trees: Message-passing and linear programming approaches. IEEE Transactions on Information Theory (2002)Google Scholar
  31. 31.
    Frank, M., Plaue, M., Rapp, H., Köthe, U., Jähnea, B., Hamprecht, F.A.: Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras. Optical Engineering (2009)Google Scholar
  32. 32.
    Tomasi, C., Manduchi, R.: Bilateral filtering for gray and color images. In: ICCV (1998)Google Scholar
  33. 33.
  34. 34.
    Saxena, A., Sun, M., Ng, A.: Make3d: Learning 3d scene structure from a single still image. PAMI (2009)Google Scholar
  35. 35.
    Saxena, A., Driemeyer, J., Kearns, J., Ng, A.: Robotic grasping of novel objects. In: NIPS (2006)Google Scholar
  36. 36.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. IJCV (2002)Google Scholar
  37. 37.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Oisin Mac Aodha
    • 1
  • Neill D. F. Campbell
    • 1
  • Arun Nair
    • 1
  • Gabriel J. Brostow
    • 1
  1. 1.University College LondonUK

Personalised recommendations