Journal of Intelligent & Robotic Systems

, Volume 80, Issue 3–4, pp 573–594 | Cite as

Generic System for Human-Computer Gesture Interaction: Applications on Sign Language Recognition and Robotic Soccer Refereeing

  • Paulo Trigueiros
  • Fernando Ribeiro
  • Luis Paulo Reis


Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for real-time human-machine interaction. Its novelty is the integration of different tools for gesture spotting and the proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of vision-based interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained with a centroid distance dataset composed of 2170 records, able to achieve a final accuracy of 99.4 %. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each one of the defined gestures that the system should recognize with a final average accuracy of 93.7 %. The datasets were built from four different users with a total of 25 gestures per user, totalling 1100 records for model construction. The proposed solution has the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real-time.


Human-machine interaction Gesture recognition Computer vision Machine learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mitra, S., Acharya, T.: Gesture recognition: A Survey. In: IEEE Transactions on Systems, Man and Cybernetics, pp. 311–324. IEEE (2007)Google Scholar
  2. 2.
    Bourennane, S., Fossati, C.: Comparison of shape descriptors for hand posture recognition in video. Sig. Image Video Process. 6(1), 147–157 (2010)CrossRefGoogle Scholar
  3. 3.
    Yoon, J.-H., Park, J.-S., Sung, M.Y.: Vision-based barehand gesture interface for interactive augmented reality applications. In: 5th International Conference on Entertainment Computing. Springer-Verlag, Cambridge (2006)Google Scholar
  4. 4.
    Buchmann, V., et al.: FingARtips: Gesture based direct manipulation in Augmented Reality. In: 2nd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia, pp. 212–221. ACM, Singapore (2004)Google Scholar
  5. 5.
    Trigueiros, P., Ribeiro, F., Lopes, G.: Vision-based hand segmentation techniques for human-robot interaction for real-time applications. In: III ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Taylor and Francis, Publication, Olhão, Algarve, Portugal (2011)Google Scholar
  6. 6.
    Vatavu, R.-D., Anthony, L., Wobbrock, J.O.: Gestures as point clouds: a $P recognizer for user interface prototypes. In: 14th ACM International Conference on Multimodal Interaction, pp. 273–280. ACM, California (2012)Google Scholar
  7. 7.
    Wobbrock, J.O., Wilson, A.D., Li, Y.: Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology, pp. 159–168. ACM, Newport, Rhode Island (2007)Google Scholar
  8. 8.
    Li, Y.: Protractor: A fast and accurate gesture recognizer. In: Conference on Human Factors in Computing Systems, pp. 2169–2172. ACM, Atlanta, Georgia (2010)Google Scholar
  9. 9.
    Kratz, S., Rohs, M.: Protractor3D: A closed-form solution to rotation-invariant 3D gestures. In: 16th International Conference on Intelligent User Interfaces, pp. 371–374. ACM, Palo Alto (2011)Google Scholar
  10. 10.
    Kim, T.: In-Depth: Eye To Eye - The History Of EyeToy. 2008 November 6, 2008 [cited 2013 29-03-2013]; Available from:
  11. 11.
    Chowdhury, J.R.: Kinect Sensor for Xbox Gaming. IIT Kharagpur (2012)Google Scholar
  12. 12.
    Zafrulla, Z. et al.: American sign language recognition with the kinect. In: 13th International Conference on Multimodal Interfaces, pp. 279–286. ACM, Alicante, Spain (2011)Google Scholar
  13. 13.
    Ong, S.C., Ranganath, S.: Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Trans. Pattern Anal. Mach. Intell. 27(6), 873–91 (2005)Google Scholar
  14. 14.
    Holt, G.A.t. et al.: Influence of handshape information on automatic sign language recognition. In: 8th International Conference on Gesture in Embodied Communication and Human-Computer Interaction. Springer-Verlag, Bielefeld, Germany (2010)Google Scholar
  15. 15.
    Tara, R.Y., Santosa, P.I., Adji, T.B.: Sign language recognition in robot teleoperation using centroid distance fourier descriptors. Int. J. Comput. Appl. 48(2) (2012)Google Scholar
  16. 16.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: A Comparative Study of different image features for hand gesture machine learning. In: 5th International Conference on Agents and Artificial Intelligence. Barcelona, Spain (2013)Google Scholar
  17. 17.
    Murthy, G.R.S., Jadon, R.S.: A review of vision based hand gestures recognition. Int. J. Inf. Technol. Knowl. Manag. 2(2), 405–410 (2009)Google Scholar
  18. 18.
    Maung, T.H.H.: Real-time hand tracking and gesture recognition system using neural networks. Proc. World Acad. Sci. Eng. Technol. 50, 466–470 (2009)Google Scholar
  19. 19.
    Snyder, W.E., Qi, H.: Machine Vision. Cambridge University Press (2004)Google Scholar
  20. 20.
    Vicen-Bueno, R., et al.: Complexity Reduction. In: Neural Networks Appplied to Traffic Sign Recognition Tasks (2004)Google Scholar
  21. 21.
    Bailador, G., et al.: Real time gesture recognition using continuous time recurrent neural networks. In: 2nd International Conference on Body Area Networks. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), Florence, Italy (2007)Google Scholar
  22. 22.
    Ben-Hur, A., Weston, J.: A User’s Guide to Support Vector Machines. In: Data Mining Techniques for the Life Sciences, pp. 223–239. Humana Press (2008)Google Scholar
  23. 23.
    Faria, B.M., Lau, N., Reis, L.P.: Classification of Facial Expressions Using Data Mining and machine Learning Algorithms. In: 4a Conferência Ibérica de Sistemas e Tecnologias de Informação. Póvoa de Varim, Portugal (2009)Google Scholar
  24. 24.
    Faria, B.M., et al.: Machine Learning Algorithms applied to the Classification of Robotic Soccer Formations and Opponent Teams. In: IEEE Conference on Cybernetics and Intelligent Systems (CIS), pp. 344–349. Singapore (2010)Google Scholar
  25. 25.
    Ke, W., et al.: Real-Time Hand Gesture Recognition for Service Robot, pp. 976–979 (2010)Google Scholar
  26. 26.
    Maldonado-Báscon, S., et al.: Road-Sign detection and Recognition Based on Support Vector Machines. In: IEEE Transactions on Intelligent Transportation Systems, pp. 264–278 (2007)Google Scholar
  27. 27.
    Oshita, M., Matsunaga, T.: Automatic learning of gesture recognition model using SOM and SVM. In: International Conference on Advances in Visual Computing. Springer-Verlag, Las Vegas (2010)Google Scholar
  28. 28.
    Almeida, R., Reis, L.P., Jorge, A.M.: Analysis and Forecast of Team Formation in the Simulated Robotic Soccer Domain. In: Proceedings of the 14th Portuguese Conference on Artificial Intelligence: Progress in Artificial Intelligence, pp. 239–250. Springer-Verlag, Aveiro, Portugal (2009)Google Scholar
  29. 29.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: A comparison of machine learning algorithms applied to hand gesture recognition. In: 7th Iberian Conference on Information Systems and Technologies. Madrid, Spain (2012)Google Scholar
  30. 30.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision-based Sign Language Recognition System. In: World Conference on Information Systems and Technologies. Madeira, Portugal (2014)Google Scholar
  31. 31.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision-based Gesture Recognition System for Human-Computer Interaction. In: IV ECCOMAS Thematic Conference on Computational Vision and Medical Image Processing. Taylor and Francis, Publication, Funchal, Madeira (2013)Google Scholar
  32. 32.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proc. 77(2), 257–286 (1989)CrossRefGoogle Scholar
  33. 33.
    Oka, K., Sato, Y., Koike, H.: Real-time fingertip tracking and gesture recognition. IEEE Comput. Graph. Appl. 22(6), 64–71 (2002)CrossRefGoogle Scholar
  34. 34.
    Perrin, S., Cassinelli, A., Ishikawa, M.: Gesture recognition using laser-based tracking system. In: Sixth IEEE International Conference on Automatic Face and Gesture Recognition. Seoul, South Korea (2004)Google Scholar
  35. 35.
    Binh, N.D., Shuichi, E., Ejima, T.: Real-Time Hand Tracking and Gesture Recognition System. In: Proceedings of International Conference on Graphics, Vision and Image. Cairo - Egypt (2005)Google Scholar
  36. 36.
    Chen, F.-S., Fu, C.-M., Huang, C.-L.: Hand gesture recognition using a real-time tracking method and hidden Markov models. Image Vis. Comput. 21(8), 745–758 (2003)CrossRefGoogle Scholar
  37. 37.
    Kelly, D., McDonald, J., Markham, C. In: Wang, L., et al. (eds.): Recognition of Spatiotemporal Gestures in Sign Language Using Gesture Threshold HMMs. In: Machine Learning for Vision-Based Motion Analysis, pp. 307–348. Springer, London (2011)Google Scholar
  38. 38.
    Trigueiros, P., Ribeiro, F., Reis, L.P.: Vision Based Referee Sign Language Recognition System for the RoboCup MSL League. In: 17th annual RoboCup International Symposium. Eindhoven, Holland (2013)Google Scholar
  39. 39.
    Helen, C., Richard, B.: Large lexicon detection of sign language. In: 11th International Conference on Human-Computer Interaction. Springer-Verlag, Rio de Janeiro, Brazil (2007)Google Scholar
  40. 40.
    Milosevic, B., Farella, E., Benini, L.: Continuous Gesture Recognition for Resource Constrained Smart Objects. In: The Fourth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies. Florence, Italy (2010)Google Scholar
  41. 41.
    Elmezain, M., et al.: A Hidden Markov Model-based Continuous Gesture Recognition System for Hand Motion Trajectory. In: 19th International Conference on Pattern Recognition. Tampa, Florida (2008)Google Scholar
  42. 42.
    Wachs, J.P., Stern, H., Edan, Y.: Cluster labeling and parameter estimation for the automated setup of a hand-gesture recognition system. IEEE Trans. Syst. Man Cybern. - Part A Syst. Humans 35(6), 932–944 (2005)Google Scholar
  43. 43.
    Jain, A.K., Duin, R.P.W., Mao, J.: Statisitical pattern recognition: A review. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 4–36 (2000)Google Scholar
  44. 44.
    Zhang, D., Lu, G.: A comparative Study of Fourier Descriptors for Shape Representation and Retrieval. In: Proceedings of 5th Asian Conference on Computer Vision (ACCV). Springer, Melbourne, Australia (2002)Google Scholar
  45. 45.
    Kauppinen, H., Seppanen, T., Pietikainen, M.: An experimental comparison of autoregressive and Fourier-based descriptors in 2D shape classification. IEEE Trans. Pattern Anal. Mach. Intell. 17(2), 201–207 (1995)CrossRefGoogle Scholar
  46. 46.
    Zhang, D., Lu, G.: A comparative study on shape retrieval using fourier descriptors with different shape signatures. J. Vis. Commun. Image Represent. 14(1), 41–60 (2003)CrossRefGoogle Scholar
  47. 47.
    Alpaydin, E.: Introduction to Machine Learning. MIT Press (2004)Google Scholar
  48. 48.
    Montgomery, D.C., Runger, G.C.: Applied Statistics and Probability for Engineers. Wiley (1994)Google Scholar
  49. 49.
    Witten, I.H., Frank, E., Hall, M.A. Data Mining - Pratical Machine Learning Tools and Techniques, 3rd edn. Elsevier (2011)Google Scholar
  50. 50.
    Theodoridis, S., Koutroumbas, K.: Pattern Recognition, 4th edn. Elsevier (2009)Google Scholar
  51. 51.
    Sayad, D.S.: Support Vector Machine - Classification (SVM). 2010 8 Nov 2012]; Available from:
  52. 52.
    Theodoridis, S., Koutroumbas, K.: An Introduction to Pattern Recognition: A Matlab Approach. Academic Press (2010)Google Scholar
  53. 53.
    Rabiner, L.R., Juang, B.H.: An introduction to hidden Markov models. IEEE ASSp Magazine (1986)Google Scholar
  54. 54.
    Wu, Y., Huang, T.S.: Vision-Based Gesture Recognition: A Review. In: Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction, pp. 103–115. Springer-Verlag (1999)Google Scholar
  55. 55.
    Fink, G.A.: Markov Models for Pattern recognition - From Theory to Applications, p 248. Springer (2008)Google Scholar
  56. 56.
    Camastra, F., Vinciarelli, A.: Machine Learning for Audio, Image and Video Analysis. Springer (2008)Google Scholar
  57. 57.
    Reis, L.P., Lau, N.: COACH UNILANG - A Standard Language for Coaching a (Robo) Soccer Team. In: Birk, V.A., Coradeschi, S., Tadokoro, S. (eds.) RoboCup 2001: Robot Soccer World Cup, pp. 183–192. Springer Berlin Heidelberg (2002)Google Scholar
  58. 58.
    Backus, J.W., et al.: Revised Report on the Algorithmic Language ALGOL 60. In: Communications of the ACM, pp. 1–17. ACM (1960)Google Scholar
  59. 59.
    Buckland, M.: Programming Game AI by Example, p 521. Wordware Publishing, Inc. (2005)Google Scholar
  60. 60.
    Millington, I., Funge, J.: In: Kaufmann, M. (ed.) Artificial Intelligence for Games, 2nd edn, p. 870. Elsevier (2009)Google Scholar
  61. 61.
    Lieberman, Z., Watson, T., Castro, A.: openFrameworks. 2004 10 October 2013 [cited 2011; openFrameworks is an open source C ++ toolkit designed to assist the creative process by providing a simple and intuitive framework for experimentation]. Available from:
  62. 62.
    Bradski, G., Kaehler, A.: Learning OpenCV: Computer Vision with the OpenCV Library, 1st edn. OReilly Media (2008)Google Scholar
  63. 63.
    OpenNI. The standard framework for 3D sensing. 2013; Available from:
  64. 64.
    King, D.E.: Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  65. 65.
    Murphy, K.: Hidden Markov Model (HMM) Toolbox for Matlab. 1998 8 June 2005 [cited 2012; This toolbox supports inference and learning for HMMs with discrete outputs (dhmm’s), Gaussian outputs (ghmm’s), or mixtures of Gaussians output (mhmm’s). The Gaussians can be full, diagonal, or spherical (isotropic). It also supports discrete inputs, as in a POMDP. The inference routines support filtering, smoothing, and fixed-lag smoothing.]. Available from:
  66. 66.
    Miner, R.: RapidMiner : Report the Future. December 2011]; Available from:
  67. 67.
    Ching-Tang, H., et al.: A real time hand gesture recognition system based on DFT and SVM. In: 8th International Conference on Information Science and Digital Content Technology. Jeju, Korea (2012)Google Scholar
  68. 68.
    Yen-Ting, C., Kuo-Tsung, T.: Multiple-angle Hand Gesture Recognition by Fusing SVM Classifiers. In: IEEE International Conference on Automation Science and Engineering. Washington, DC (2007)Google Scholar
  69. 69.
    Gao, J., Cao, Q.: Adaptive HOG-LBP Based Learning for Palm Tracking. In: 2nd International Conference on Computer and Information Applications. Taiyuan, Shanxi, China (2012)Google Scholar
  70. 70.
    Liu, Y., Zhang, P.: An Automatic Hand Gesture Recognition System Based on Viola-Jones Method and SVMs. In: Second International Workshop on Computer Science and Engineering, WCSE ’09 (2009)Google Scholar
  71. 71.
    Dardas, N.H., Georganas, N.D.: Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Trans. Instrum. Meas. 60(11), 3592–3607 (2011)CrossRefGoogle Scholar
  72. 72.
    Yoon, H.-S., et al.: Hand gesture recognition using combined features of location, angle and velocity. Pattern Recogn. 34(7), 1491–1501 (2001)MATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2015

Authors and Affiliations

  • Paulo Trigueiros
    • 1
    • 2
    • 4
    • 5
  • Fernando Ribeiro
    • 1
    • 4
  • Luis Paulo Reis
    • 3
    • 4
    • 5
  1. 1.Escola de Engenharia da Universidade do Minho, Department Eletrónica IndustrialGuimarãesPortugal
  2. 2.Departamento de InformáticaInstituto Politécnico do PortoGuimarãesPortugal
  3. 3.Department Sistemas de InformaçãoEscola de Engenharia da Universidade do MinhoGuimãraesPortugal
  4. 4.Centro ALGORITMIUniversidade do MinhoGuimarãesPortugal
  5. 5.Laboratório de Inteligência Artificial e Ciência de ComputadoresPortoPortugal

Personalised recommendations