Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset

  • Qi Guo
  • Iuri FrosioEmail author
  • Orazio Gallo
  • Todd Zickler
  • Jan Kautz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11205)


Scene motion, multiple reflections, and sensor noise introduce artifacts in the depth reconstruction performed by time-of-flight cameras. We propose a two-stage, deep-learning approach to address all of these sources of artifacts simultaneously. We also introduce FLAT, a synthetic dataset of 2000 ToF measurements that capture all of these nonidealities, and allows to simulate different camera hardware. Using the Kinect 2 camera as a baseline, we show improved reconstruction errors over state-of-the-art methods, on both simulated and real data.


Time-of-Flight MPI artifacts Motion artifacts 

Supplementary material

474172_1_En_23_MOESM1_ESM.pdf (7.1 mb)
Supplementary material 1 (pdf 7231 KB)


  1. 1.
    Xiang, L., et al.: libfreenect2: Release 0.2, April 2016Google Scholar
  2. 2.
    Bhandari, A., Raskar, R.: Signal processing for time-of-flight imaging sensors: an introduction to inverse problems in computational 3-D imaging. IEEE Signal Process. Mag. 33, 45–58 (2016)CrossRefGoogle Scholar
  3. 3.
    Feigin, M., Bhandari, A., Izadi, S., Rhemann, C., Schmidt, M., Raskar, R.: Resolving multipath interference in kinect: an inverse problem approach. Sensors 16, 3419–3427 (2016)CrossRefGoogle Scholar
  4. 4.
    Marco, J., et al.: DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH ASIA) 36, 219 (2017)Google Scholar
  5. 5.
    Mutny, M., Nair, R., Gottfried, J.: Learning the correction for multi-path deviations in time-of-flight cameras. arXiv preprint arXiv:1512.04077 (2015)
  6. 6.
    Son, K., Liu, M., Taguchi, Y.: Automatic learning to remove multipath distortions in time-of-flight range images for a robotic arm setup. arXiv preprint arXiv:1601.01750 (2016)
  7. 7.
    Dosovitskiy, A. et al.: Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2015)Google Scholar
  8. 8.
    Bako, S., et al.: Kernel-predicting convolutional networks for denoising monte carlo renderings. ACM Trans. Graph. (SIGGRAPH) 36, 97 (2017)CrossRefGoogle Scholar
  9. 9.
    Jung, J., Lee, J.-Y., Kweon, I.S.: Noise aware depth denoising for a time-of-flight camera. In: Korea-Japan Joint Workshop on Frontiers of Computer Vision (2014)Google Scholar
  10. 10.
    Jarabo, A., Masia, B., Marco, J., Gutierrez, D.: Recent advances in transient imaging: A computer graphics and vision perspective. Vis. Inform. 1, 65–79 (2017)CrossRefGoogle Scholar
  11. 11.
    Ferstl, D., Reinbacher, C., Riegler, G., Rüther, M., Bischof, H.: Learning depth calibration of time-of-flight cameras. In: Proceedings of the British Machine Vision Conference (BMVC) (2015)Google Scholar
  12. 12.
    Lenzen, F., et al.: Denoising strategies for time-of-flight data. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. LNCS, vol. 8200, pp. 25–45. Springer, Heidelberg (2013). Scholar
  13. 13.
    Gottfried, J.M., Nair, R., Meister, S., Garbe, C.S., Kondermann, D.: Time of flight motion compensation revisited. In: IEEE International Conference on Image Processing (ICIP), (2014)Google Scholar
  14. 14.
    Lee, S.: Time-of-flight depth camera motion blur detection and deblurring. IEEE Signal Process. Lett. 21, 663–666 (2014)CrossRefGoogle Scholar
  15. 15.
    Hansard, M., Lee, S., Choi, O., Horaud, R.: Time-of-Flight Cameras: Principles Methods and Applications. Springer Publishing Company, London (2012). Scholar
  16. 16.
    Heide, F., Heidrich, W., Hullin, M., Wetzstein, G.: Doppler time-of-flight imaging. ACM Trans. Graph. (SIGGRAPH) 34, 36 (2015)CrossRefGoogle Scholar
  17. 17.
    Freedman, D., Krupka, E., Smolin, Y., Leichter, I., Schmidt, M.: SRA: fast removal of general multipath for tof sensors. arXiv preprint arXiv:1403.5919 (2014)
  18. 18.
    Gupta, M., Nayar, S.K., Hullin, M.B., Martin, J.: Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. 34, 156 (2015)CrossRefGoogle Scholar
  19. 19.
    Nair, R., et al.: Ground truth for evaluating time of flight imaging. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. LNCS, vol. 8200, pp. 52–74. Springer, Heidelberg (2013). Scholar
  20. 20.
    Jarabo, A., Marco, J., Muñoz, A., Buisan, R., Jarosz, W., Gutierrez, D.: A framework for transient rendering. ACM Trans. Graph. (SIGGRAPH ASIA) 33, 177 (2014)zbMATHGoogle Scholar
  21. 21.
    Gushov, V., Solodkin, Y.N.: Automatic processing of fringe patterns in integer interferometers. Opt. Lasers Eng. 14, 311–324 (1991)CrossRefGoogle Scholar
  22. 22.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 303–312. ACM (1996)Google Scholar
  23. 23.
    Burkardt, J.: Obj files: A 3D object format (2016).
  24. 24.
    Dana, K.J., Van Ginneken, B., Nayar, S.K., Koenderink, J.J.: Reflectance and texture of real-world surfaces. ACM Trans. Graph. (TOG) 18, 1–34 (1999)CrossRefGoogle Scholar
  25. 25.
    Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  26. 26.
    Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Qi Guo
    • 1
    • 2
  • Iuri Frosio
    • 1
    Email author
  • Orazio Gallo
    • 1
  • Todd Zickler
    • 2
  • Jan Kautz
    • 1
  1. 1.NVIDIASanta ClaraUSA
  2. 2.Harvard SEASCambridgeUSA

Personalised recommendations