Human-Computer Interaction through Time-of-Flight and RGB Cameras

  • Piercarlo Dondi
  • Luca Lombardi
  • Marco Porta
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6979)


The number of systems exploiting Time-of-Flight (ToF) cameras for gesture recognition has greatly increased in the last years, confirming a very positive trend of this technology within the field of Human-Computer Interaction. In this work we present a new kind of application for the interaction with a virtual keyboard which is based on the use of an ordinary RGB webcam and a ToF camera. Our approach can be subdivided into two steps: firstly a segmentation of the entire body of the user is achieved exploiting only the ToF data; then the extraction of hands and head is obtained applying color information on the retrieved clusters. The final tracking step, based on the Kalman filter, is able to recognize the chosen hand also in presence of a second hand or the head. Tests, carried out with users of different ages, showed interesting results and a quick learning curve.


Time-of-Flight camera human-computer interaction hand recognition 


  1. 1.
    Kolb, A., Barth, E., Koch, R., Larsen, R.: Time-of-Flight Cameras in Computer Graphics. Computer Graphics Forum 29, 141–159 (2010)CrossRefGoogle Scholar
  2. 2.
    Bartczak, B., Schiller, I., Beder, C., Koch, R.: Integration of a Time-of-Flight camera into a mixed reality system for handling dynamic scenes, moving viewpoints and occlusions in real-time. In: Fourth International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT 2008) (2008)Google Scholar
  3. 3.
    Santrac, N., Friedland, G., Rojas, R.: High resolution segmentation with a time-of-flight 3d-camera using the example of a lecture scene. Technical report (2006),
  4. 4.
    Bleiweiss, A., Werman, M.: Real-time foreground segmentation via range and color imaging. In: Kolb, A., Koch, R. (eds.) Dyn3D 2009. LNCS, vol. 5742, pp. 58–69. Springer, Heidelberg (2009)Google Scholar
  5. 5.
    Dondi, P., Lombardi, L.: Fast Real-Time Segmentation and Tracking of Multiple Subjects by Time-of-Flight Camera. In: 6th International Conference on Computer Vision Theory and Applications (VISAPP 2011), pp. 582–587 (2011)Google Scholar
  6. 6.
    Oprisescu, S., Falie, D., Ciuc, M., Buzuloiu, V.: Measurements with ToF Cameras and Their Necessary Corrections. In: International Symposium on Signals, Circuits and Systems, ISSCS 2007 (2007)Google Scholar
  7. 7.
    Oggier, T., Büttgen, B., Lustenberger, F., Becker, G., Rüegg, B., Hodac, A.: Swissranger SR3000 and First Experiences based on Miniaturized 3D-TOF Cameras. In: Proceedings, 1st Range Imaging Research Day, September 8-9, pp. 97–108. ETH Zurich Switzerland (2005)Google Scholar
  8. 8.
    Ghobadi, S., Loepprich, O., Hartmann, K., Loffeld, O.: Hand Segmentation Using 2D/3D Images. In: Image and Vision Computing, New Zealand, pp. 64–69 (2007)Google Scholar
  9. 9.
    Breuer, P., Eckes, C., Müller, S.: Hand Gesture Recognition with a Novel IR Time-of-Flight Range Camera–A Pilot Study. In: Gagalowicz, A., Philips, W. (eds.) MIRAGE 2007. LNCS, vol. 4418, pp. 247–260. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  10. 10.
    Kollorz, E., Penne, J., Hornegger, J., Barke, A.: Gesture recognition with a Time-Of-Flight camera. Int. J. Intell. Syst. Technol. Appl. 5(3/4), 334–343 (2008)Google Scholar
  11. 11.
    Van den Bergh, M., Van Gool, L.: Combining RGB and ToF cameras for real-time 3D hand gesture interaction. In: IEEE Workshop on Applications of Computer Vision (WACV 2011), pp. 66–72 (2011)Google Scholar
  12. 12.
    Haker, M., Böhme, M., Martinetz, T., Barth, E.: Deictic Gestures with a Time-of-Flight Camera. In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS, vol. 5934, pp. 110–121. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Soutschek, S., Penne, J., Hornegger, J., Kornhuber, J.: 3-D Gesture-Based Scene Navigation in Medical Imaging Applications Using Time-Of-Flight Cameras. In: IEEE Computer Vision and Pattern Recognition Workshops, pp. 1–6 (2008)Google Scholar
  14. 14.
    Reulke, R.: Combination of distance data with high resolution images. In: Image Engeeniring and Vision Metrology, IEVM 2006 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Piercarlo Dondi
    • 1
  • Luca Lombardi
    • 1
  • Marco Porta
    • 1
  1. 1.Department of Computer Engineering and Systems ScienceUniversity of PaviaPaviaItaly

Personalised recommendations