Neural Network Based Data Fusion for Hand Pose Recognition with Multiple ToF Sensors

  • Thomas Kopinski
  • Alexander Gepperth
  • Stefan Geisler
  • Uwe Handmann
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)


We present a study on 3D based hand pose recognition using a new generation of low-cost time-of-flight(ToF) sensors intended for outdoor use in automotive human-machine interaction. As signal quality is impaired compared to Kinect-type sensors, we study several ways to improve performance when a large number of gesture classes is involved. We investigate the performance of different 3D descriptors, as well as the fusion of two ToF sensor streams. By basing a data fusion strategy on the fact that multilayer perceptrons can produce normalized confidences individually for each class, and similarly by designing information-theoretic online measures for assessing confidences of decisions, we show that appropriately chosen fusion strategies can improve overall performance to a very satisfactory level. Real-time capability is retained as the used 3D descriptors, the fusion strategy as well as the online confidence measures are computationally efficient.


Point Cloud Hand Gesture Fusion Strategy Depth Sensor Late Fusion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Oprisescu, S., Rasche, C., Su, B.: Automatic static hand gesture recognition using tof cameras. In: 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), pp. 2748–2751. IEEE (2012)Google Scholar
  2. 2.
    Kollorz, E., Penne, J., Hornegger, J., Barke, A.: Gesture recognition with a time-of-flight camera. International Journal of Intelligent Systems Technologies and Applications 5(3), 334–343 (2008)CrossRefGoogle Scholar
  3. 3.
    Soutschek, S., Penne, J., Hornegger, J., Kornhuber, J.: 3-d gesture-based scene navigation in medical imaging applications using time-of-flight cameras. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2008, pp. 1–6. IEEE (2008)Google Scholar
  4. 4.
    Wen, Y., Hu, C., Yu, G., Wang, C.: A robust method of detecting hand gestures using depth sensors. In: 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE), pp. 72–77. IEEE (2012)Google Scholar
  5. 5.
    Kapuscinski, T., Oszust, M., Wysocki, M.: Hand gesture recognition using time-of-flight camera and viewpoint feature histogram. In: Korbicz, J., Kowal, M. (eds.) Intelligent Systems in Technical and Medical Diagnostics. AISC, vol. 230, pp. 403–414. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  6. 6.
    Tang, M.: Recognizing hand gestures with Microsoft’s kinect (2011),
  7. 7.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Efficient model-based 3d tracking of hand articulations using kinect. In: BMVC, pp. 1–11 (2011)Google Scholar
  8. 8.
    Bailly, G., Walter, R., Müller, J., Ning, T., Lecolinet, E.: Comparing free hand menu techniques for distant displays using linear, marking and finger-count menus. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part II. LNCS, vol. 6947, pp. 248–262. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  9. 9.
    Wilson, A.D., Benko, H.: Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology, pp. 273–282. ACM (2010)Google Scholar
  10. 10.
    Pickering, C.A., Burnham, K.J., Richardson, M.J.: A research study of hand gesture recognition technologies and applications for human vehicle interaction. In: 3rd Conf. on Automotive Electronics, Citeseer (2007)Google Scholar
  11. 11.
    Wohlkinger, W., Vincze, M.: Ensemble of shape functions for 3d object classification. In: 2011 IEEE International Conference on Robotics and Biomimetics (ROBIO), pp. 2987–2992. IEEE (2011)Google Scholar
  12. 12.
    Rusu, R.B., Bradski, G., Thibaux, R., Hsu, J.: Fast 3d recognition and pose using the viewpoint feature histogram. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2155–2162. IEEE (2010)Google Scholar
  13. 13.
    Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (fpfh) for 3d registration. In: IEEE International Conference on Robotics and Automation, ICRA 2009, pp. 3212–3217. IEEE (2009)Google Scholar
  14. 14.
    Haykin, S.: Neural networks: a comprehensive foundation. Prentice Hall (1999)Google Scholar
  15. 15.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer vision with the OpenCV library. O’Reilly Media, Incorporated (2008)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Thomas Kopinski
    • 1
  • Alexander Gepperth
    • 2
  • Stefan Geisler
    • 1
  • Uwe Handmann
    • 1
  1. 1.Computer Science InstituteUniversity of Applied Sciences BottropMühlheimGermany
  2. 2.ENSTA ParisTech- UIIS Lab, 828 Blvd des MaréchauxPalaiseauFrance

Personalised recommendations