Kinect-Sign: Teaching Sign Language to “Listeners” through a Game

  • João Gameiro
  • Tiago Cardoso
  • Yves Rybarczyk
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 425)


Sign language is the hearing impaired form of communicating with other people, including listeners. Most cases, impaired people have learned sign language form childhood. The problem arises when a listener comes in contact with an impaired person. For instances, if a couple has a child which is impaired, the parents find a challenge to learn the sign language. In this article, a new playful approach to assist the listeners to learn sign language is proposed. This proposal is a serious game composed of two modes: School-mode and Competition-mode. The first offers a virtual school where the user learns to sign letters and the second offers an environment towards applying the learned letters. Behind the scenes, the proposal contains a sign language recognition system, based on three modules: 1 – the standardization of the Kinect depth camera data; 2 – a gesture library relying on the standardized data; and 3 – the real-time recognition of gestures. A prototype was developed – Kinect-Sign – and tested in a Portuguese Sign-Language school and on eNTERFACE’13 resulting in a joyful acceptance of the approach.


Kinect Sensor Sign Language Serious Game Gesture Recognition 


  1. 1.
    World Health Organization, Deafness and hearing loss. World Health Organization (February 2013), (accessed October 2, 2013)
  2. 2.
    Cooper, H., Holt, B., Bowden, R.: Sign Language Recognition. In: Visual Analysis of Humans, pp. 539–562. Springer (2011)Google Scholar
  3. 3.
    Song, Y., Yin, Y.: Sign Language Recognition (2013),
  4. 4.
    “Kinect,” Wikipedia (August 13, 2013), (accessed August 22, 2013)
  5. 5.
    Chai, X., Li, G., Lin, Y., Xu, Z., Tang, Y., Chen, X., Zhou, M.: Sign Language Recognition and Translation with Kinect, Beijing, China (2013)Google Scholar
  6. 6.
    Gameiro, J., Cardoso, T., Rybarczyk, Y.: Kinect-Sign - Teaching sign language to ‘listeners’ through a game. In: Conference on Electronics, Telecommunications and Computers, Lisboa, Portugal (2013)Google Scholar
  7. 7.
    “Sign Language,” Wikipedia (August 16, 2013), (accessed August 23, 2013)
  8. 8.
    Winnie, T., Drennan, A.: Sign Language Bingo (2008), (accessed August 23, 2013)
  9. 9.
  10. 10.
    “Sign the Alphabet,” Funbrain, (accessed August 23, 2013)
  11. 11.
    Breuer, J.S., Bente, G.: Why so serious? On the relation of serious games and learning. Eludamos. Journal for Computer Game Culture 4, nº Serious Games (2010)Google Scholar
  12. 12.
    “Kinect Could Reinvent Educational Gaming,” Winextra (2013), (accessed August 23, 2013)
  13. 13.
    Starner, T.E.: Visual Recognition of American Sign Language Using Hidden Markov Models (1995), (accessed August 24, 2013)
  14. 14.
    “Hidden Markov model,” Wikipedia (August 4, 2013), (accessed August 24, 2013)
  15. 15.
    Correia, M.M.: Reconhecimento de Elementos da Língua Gestual Portuguesa com Kinect (2013), (accessed August 19, 2013)
  16. 16.
    “Top 10 Kinect Games,” Squidoo (2013), (accessed August 22, 2013)
  17. 17.
    Zoo Games, Minute To Win It (October 2011), (accessed August 22, 2013)
  18. 18.
    Camarinha-Matos, L., Menzel, K., Cardoso, T.: ICT support infrastructures and interoperability for VOs. Virtual Organisations Cluster–VOSTER WP4 D (2003)Google Scholar
  19. 19.
    Klen, E., Cardoso, T., Camarinha-Matos, L.: Teaching Initiatives on Collaborative Networked Organizations. In: 8th CIRP - International Seminar on Manufacturing, Florianópolis-SC, Brazil (2005)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2014

Authors and Affiliations

  • João Gameiro
    • 1
  • Tiago Cardoso
    • 1
  • Yves Rybarczyk
    • 1
  1. 1.Faculdade de Ciências e TecnologiaUniversidade Nova de LisboaMonte da CaparicaPortugal

Personalised recommendations