Online Robot Teleoperation Using Human Hand Gestures: A Case Study for Assembly Operation

  • Nuno MendesEmail author
  • Pedro Neto
  • Mohammad Safeea
  • António Paulo Moreira
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 418)


A solution for intuitive robot command and fast robot programming is presented to assemble pins in car doors. Static and dynamic gestures are used to instruct an industrial robot in the execution of the assembly task. An artificial neural network (ANN) was used in the recognition of twelve static gestures and a hidden Markov model (HMM) architecture was used in the recognition of ten dynamic gestures. Results of these two architectures are compared with results displayed by a third architecture based on support vector machine (SVM). Results show recognition rates of 96 % and 94 % for static and dynamic gestures when the ANN and HMM architectures are used, respectively. The SVM architecture presents better results achieving recognition rates of 97 % and 96 % for static and dynamic gestures, respectively.


Gesture spotting Robot programming Robotic assembly Industrial robot 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ren, Z., Yuan, J., Meng, J., Zhang, Z.: Robust Part-Based Hand Gesture Recognition Using Kinect Sensor. IEEE Trans. Multimed. 15, 1110–1120 (2013)CrossRefGoogle Scholar
  2. 2.
    Huang, P.-C., Jeng, S.-K.: Human body pose recognition from a single-view depth camera. In: 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2144–2149. IEEE (2012)Google Scholar
  3. 3.
    Seal, A., Bhattacharjee, D., Nasipuri, M., Basu, D.K.: Thermal human face recognition based on GappyPCA. In: 2013 IEEE Second International Conference on Image Information Processing (ICIIP-2013), pp. 597–600. IEEE (2013)Google Scholar
  4. 4.
    Kirishima, T., Sato, K., Chihara, K.: Real-time gesture recognition by learning and selective control of visual interest points. IEEE Trans. Pattern Anal. Mach. Intell. 27, 351–364 (2005)CrossRefGoogle Scholar
  5. 5.
    Lambrecht, J., Kruger, J.: Spatial programming for industrial robots based on gestures and Augmented Reality. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 466–472. IEEE (2012)Google Scholar
  6. 6.
    Oz, C., Leu, M.C.: Linguistic properties based on American Sign Language isolated word recognition with artificial neural networks using a sensory glove and motion tracker. Neurocomputing 70, 2891–2901 (2007)CrossRefGoogle Scholar
  7. 7.
    Neto, P., Pires, J.N., Moreira, A.P.: High-level programming and control for industrial robotics: using a hand-held accelerometer-based input device for gesture and posture recognition. Ind. Robot. An. Int. J. 37, 137–147 (2010)CrossRefGoogle Scholar
  8. 8.
    Neto, P., Pires, J.N., Moreira, A.P.: Accelerometer-based control of an industrial robotic arm. In: RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication, pp. 1192–1197. IEEE (2009)Google Scholar
  9. 9.
    Mitra, S., Acharya, T.: Gesture Recognition: A Survey. IEEE Trans. Syst. Man Cybern. Part C Applications Rev. 37, 311–324 (2007)CrossRefGoogle Scholar
  10. 10.
    Yang, J., Bang, W., Choi, E., Cho, S., Oh, J., Cho, J., Kim, S., Ki, E., Kim, D.: A 3D hand-drawn gesture input device using fuzzy ARTMAP-based recognizer. J. Syst. Cybern. Informatics 4, 1–7 (2006)Google Scholar
  11. 11.
    Yamashita, Y., Tani, J.: Emergence of functional hierarchy in a multiple timescale neural network model: a humanoid robot experiment. PLoS Comput. Biol. 4 (2008)Google Scholar
  12. 12.
    Peng, B., Qian, G.: Online gesture spotting from visual hull data. IEEE Trans. Pattern Anal. Mach. Intell. 33, 1175–1188 (2011)CrossRefGoogle Scholar
  13. 13.
    Badi, H.S., Hussein, S.: Hand posture and gesture recognition technology. Neural Comput. Appl. 25, 871–878 (2014)CrossRefGoogle Scholar
  14. 14.
    Wang, X., Xia, M., Cai, H., Gao, Y., Cattani, C.: Hidden-Markov-Models-Based Dynamic Hand Gesture Recognition. Math. Probl. Eng. (2012)Google Scholar
  15. 15.
    Bertsch, F.A., Hafner, V. V.: Real-time dynamic visual gesture recognition in human-robot interaction. In: 9th IEEE-RAS International Conference on Humanoid Robots, pp. 447–453. IEEE (2009)Google Scholar
  16. 16.
    Kurakin, A., Zhang, Z., Liu, Z.: A real time system for dynamic hand gesture recognition with a depth sensor. In: 20th European Signal Processing Conference (EUSIPCO 2012), pp. 1975–1979 (2012)Google Scholar
  17. 17.
    Zhang, Y., Zhang, L., Hossain, M.A.: Adaptive 3D facial action intensity estimation and emotion recognition. Expert Syst. Appl. 42, 1446–1464 (2015)CrossRefGoogle Scholar
  18. 18.
    El-Baz, A.H., Tolba, A.S.: An efficient algorithm for 3D hand gesture recognition using combined neural classifiers. Neural Comput. Appl. 22, 1477–1484 (2012)CrossRefGoogle Scholar
  19. 19.
    Badi, H., Hussein, S.H., Kareem, S.A.: Feature extraction and ML techniques for static gesture recognition. Neural Comput. Appl. 25, 733–741 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Nuno Mendes
    • 1
    Email author
  • Pedro Neto
    • 2
  • Mohammad Safeea
    • 2
  • António Paulo Moreira
    • 1
  1. 1.Centre for Robotics and Intelligent SystemsInstitute for Systems and Computer Engineering Technology and SciencePortoPortugal
  2. 2.Centre for Mechanical Engineering of the University of CoimbraUniversity of CoimbraCoimbraPortugal

Personalised recommendations