Combination of Spatially-Modulated ToF and Structured Light for MPI-Free Depth Estimation

  • Gianluca AgrestiEmail author
  • Pietro Zanuttigh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11129)


Multi-path Interference (MPI) is one of the major sources of error in Time-of-Flight (ToF) camera depth measurements. A possible solution for its removal is based on the separation of direct and global light through the projection of multiple sinusoidal patterns. In this work we extend this approach by applying a Structured Light (SL) technique on the same projected patterns. This allows to compute two depth maps with a single ToF acquisition, one with the Time-of-Flight principle and the other with the Structured Light principle. The two depth fields are finally combined using a Maximum-Likelihood approach in order to obtain an accurate depth estimation free from MPI error artifacts. Experimental results demonstrate that the proposed method has very good MPI correction properties with state-of-the-art performances.


ToF sensors Multi-path Structured Light Depth acquisition Data fusion 



We would like to thank the Computational Imaging Group at the Sony European Technology Center (EuTEC) for allowing us to use their ToF Explorer simulator and Muhammad Atif, Oliver Erdler, Markus Kamm and Henrik Schaefer for their precious comments and insights.

Supplementary material

478770_1_En_21_MOESM1_ESM.pdf (226 kb)
Supplementary material 1 (pdf 225 KB)


  1. 1.
    Schwarz, L.A., Mkhitaryan, A., Mateus, D., Navab, N.: Human skeleton tracking from depth data using geodesic distances and optical flow. Image Vis. Comput. 30(3), 217–226 (2012)CrossRefGoogle Scholar
  2. 2.
    Van den Bergh, M., Van Gool, L.: Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In: 2011 IEEE Workshop on Applications of Computer Vision (WACV), pp. 66–72. IEEE (2011)Google Scholar
  3. 3.
    Memo, A., Zanuttigh, P.: Head-mounted gesture controlled interface for human-computer interaction. Multimedia Tools Appl. 77(1), 27–53 (2018)CrossRefGoogle Scholar
  4. 4.
    Hussmann, S., Liepert, T.: Robot vision system based on a 3D-ToF camera. In: 2007 IEEE Instrumentation and Measurement Technology Conference Proceedings, IMTC 2007, pp. 1–5. IEEE (2007)Google Scholar
  5. 5.
    Schmidt, M.: Analysis, modeling and dynamic optimization of 3D time-of-flight imaging systems. Ph.D. thesis (2011)Google Scholar
  6. 6.
    Zanuttigh, P., Marin, G., Dal Mutto, C., Dominio, F., Minto, L., Cortelazzo, G.M.: Time-of-Flight and Structured Light Depth Cameras. Springer, Cham (2016). Scholar
  7. 7.
    Gupta, M., Nayar, S.K., Hullin, M.B., Martin, J.: Phasor imaging: a generalization of correlation-based time-of-flight imaging. ACM Trans. Graph. (TOG) 34(5), 156 (2015)CrossRefGoogle Scholar
  8. 8.
    Whyte, R., Streeter, L., Cree, M.J., Dorrington, A.A.: Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods. Opt. Eng. 54(11), 113109 (2015)CrossRefGoogle Scholar
  9. 9.
    Agresti, G., Minto, L., Marin, G., Zanuttigh, P.: Deep learning for confidence information in stereo and ToF data fusion. In: Geometry Meets Deep Learning ICCV Workshop, pp. 697–705 (2017)Google Scholar
  10. 10.
    The Blender Foundation: Blender website. Accessed 7 July 2018
  11. 11.
    Meister, S., Nair, R., Kondermann, D.: Simulation of time-of-flight sensors using global illumination. In: Bronstein, M., Favre, J., Hormann, K. (eds.) Vision, Modeling and Visualization. The Eurographics Association (2013)Google Scholar
  12. 12.
    Bhandari, A., et al.: Resolving multipath interference in time-of-flight imaging via modulation frequency diversity and sparse regularization. Opt. Lett. 39(6), 1705–1708 (2014)CrossRefGoogle Scholar
  13. 13.
    Kadambi, A., et al.: Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (TOG) 32(6), 167 (2013)CrossRefGoogle Scholar
  14. 14.
    Freedman, D., Smolin, Y., Krupka, E., Leichter, I., Schmidt, M.: SRA: fast removal of general multipath for ToF sensors. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 234–249. Springer, Cham (2014). Scholar
  15. 15.
    Whyte, R., Streeter, L., Cree, M.J., Dorrington, A.A.: Review of methods for resolving multi-path interference in time-of-flight range cameras. In: IEEE Sensors, pp. 629–632. IEEE (2014)Google Scholar
  16. 16.
    O’Toole, M., Heide, F., Xiao, L., Hullin, M.B., Heidrich, W., Kutulakos, K.N.: Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. (TOG) 33(4), 87 (2014)zbMATHGoogle Scholar
  17. 17.
    Fuchs, S.: Multipath interference compensation in time-of-flight camera images. In: Proceedings of IEEE International Conference on Pattern Recognition (ICPR), pp. 3583–3586. IEEE (2010)Google Scholar
  18. 18.
    Fuchs, S., Suppa, M., Hellwich, O.: Compensation for multipath in ToF camera measurements supported by photometric calibration and environment integration. In: Chen, M., Leibe, B., Neumann, B. (eds.) ICVS 2013. LNCS, vol. 7963, pp. 31–41. Springer, Heidelberg (2013). Scholar
  19. 19.
    Jiménez, D., Pizarro, D., Mazo, M., Palazuelos, S.: Modeling and correction of multipath interference in time of flight cameras. Image Vis. Comput. 32(1), 1–13 (2014)CrossRefGoogle Scholar
  20. 20.
    Son, K., Liu, M.Y., Taguchi, Y.: Learning to remove multipath distortions in time-of-flight range images for a robotic arm setup. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 3390–3397 (2016)Google Scholar
  21. 21.
    Marco, J., et al.: DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (TOG) 36(6), 219 (2017)CrossRefGoogle Scholar
  22. 22.
    Agresti, G., Zanuttigh, P.: Deep learning for multi-path error removal in ToF sensors. In: Geometry Meets Deep Learning ECCV Workshop (2018)Google Scholar
  23. 23.
    Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6383–6392 (2018)Google Scholar
  24. 24.
    Guo, Q., Frosio, I., Gallo, O., Zickler, T., Kautz, J.: Tackling 3D ToF artifacts through learning and the FLAT dataset. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 381–396. Springer, Cham (2018). Scholar
  25. 25.
    Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., Bing Kang, S.: A light transport model for mitigating multipath interference in time-of-flight sensors. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 73–81 (2015)Google Scholar
  26. 26.
    Nayar, S.K., Krishnan, G., Grossberg, M.D., Raskar, R.: Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. (TOG) 25(3), 935–944 (2006)CrossRefGoogle Scholar
  27. 27.
    Dorrington, A.A., Whyte, R.Z.: Time of flight camera system which resolves direct and multi-path radiation components. US Patent 9,874,638, 23 January 2018Google Scholar
  28. 28.
    Lange, R., Seitz, P., Biber, A., Lauxtermann, S.C.: Demodulation pixels in CCD and CMOS technologies for time-of-flight ranging. In: Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications, vol. 3965. International Society for Optics and Photonics, pp. 177–189 (2000)Google Scholar
  29. 29.
    Mufti, F., Mahony, R.: Statistical analysis of measurement processes for time-of-flight cameras. In: Videometrics, Range Imaging, and Applications X, vol. 7447, p. 74470I. International Society for Optics and Photonics (2009)Google Scholar
  30. 30.
    Spirig, T., Seitz, P., Vietze, O., Heitger, F.: The lock-in CCD-two-dimensional synchronous detection of light. IEEE J. Quantum Electron. 31(9), 1705–1708 (1995)CrossRefGoogle Scholar
  31. 31.
    Xu, Y., Ekstrand, L., Dai, J., Zhang, S.: Phase error compensation for three-dimensional shape measurement with projector defocusing. Appl. Opt. 50(17), 2572–2581 (2011)CrossRefGoogle Scholar
  32. 32.
    Dal Mutto, C., Zanuttigh, P., Cortelazzo, G.: A probabilistic approach to ToF and stereo data fusion. In: 3DPVT, Paris, France, May 2010Google Scholar
  33. 33.
    Mutto, C.D., Zanuttigh, P., Cortelazzo, G.M.: Probabilistic ToF and stereo data fusion based on mixed pixels measurement models. IEEE Trans. Pattern Anal. Mach. Intell. 37(11), 2260–2272 (2015)CrossRefGoogle Scholar
  34. 34.
    Zhu, J., Wang, L., Gao, J., Yang, R.: Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs. IEEE Trans. Pattern Anal. Mach. Intell. 32(5), 899–909 (2010)CrossRefGoogle Scholar
  35. 35.
    Gupta, M., Nayar, S.K.: Micro phase shifting. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 813–820. IEEE (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversity of PadovaPadovaItaly

Personalised recommendations