Head Tracking and Hand Segmentation during Hand over Face Occlusion in Sign Language

  • Matilde Gonzalez
  • Christophe Collet
  • Rémi Dubot
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6553)


This paper presents a method to accurately segment the hand over the face. The similarity of colours and the important variability of the hand shape make it challenging. We propose a method based on the combination of two features: pixel colour and edges orientation. First, a specific skin model is used to find, before occlusion, the face position and the face template. Then, during occlusion the face template is registered using local gradient orientations to track the face position. Colour information is extracted from changes on pixel colours and edges are classified as belonging to the hand or to the face by mapping edges orientation to the face template. Finally by merging both features and by using an hysteresis threshold, which considers connectivity, a robust hand segmentation is reached. Experiments were performed using the Dicta-Sign corpus and showed the versatility of the proposed approach.


Hand segmentation sign language head registration 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Imagawa, I., Matsuo, H., Taniguchi, R., Arita, D., Lu, S., Igi, S.: Recognition of local features for camera-based sign language recognition system. In: Proc. 15th International Conference on Pattern Recognition, vol. 4, pp. 849–853 (2000)Google Scholar
  2. 2.
    Liang, R., Ouhyoung, M.: A real-time continuous gesture recognition system for sign language. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 558–567 (1998)Google Scholar
  3. 3.
    Habili, N., Lim, C., Moini, A.: Segmentation of the face and hands in sign language video sequences using color and motion cues. IEEE Transactions on Circuits and Systems for Video Technology 14, 1086–1097 (2004)CrossRefGoogle Scholar
  4. 4.
    Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., Sloetjes, H.: Elan: a professional framework for multimodality research. In: Proc. of the 5th International Conference on Language Resources and Evaluation, LREC 2006, pp. 1556–1559 (2006)Google Scholar
  5. 5.
    Kipp, M.: Anvil - a generic annotation tool for multimodal dialogue. In: Proc. of 7th European Conference on Speech Communication and Technology, Eurospeech, pp. 1367–1370 (2001)Google Scholar
  6. 6.
    Hanke, T.: ilex - a tool for sign language lexicography and corpus analysis. In: Proc. of 3rd International Conference on Language Resources and Evaluation, LREC 2002, Las Palmas de Gran Canaria, Spain, pp. 923–926 (2002)Google Scholar
  7. 7.
    Hanke, T., Storz, J.: ilex - a database tool for integrating sign language corpus linguistics and sign language lexicography. In: Proc. of 6th International Conference on Language Resources and Evaluation, LREC 2008, Marrakesh, pp. W25-64–W25-67 (2008)Google Scholar
  8. 8.
    Braffort, A., Choisier, A., Collet, C., Dalle, P., Gianni, F., Lenseigne, B., Segouat, J.: Toward an annotation software for video of sign language, including image processing tools and signing space modelling. In: Proc. of 4th International Conference on Language Resources and Evaluation, LREC 2004, Lisbon, Portugal, vol. 1, pp. 201–203 (2004)Google Scholar
  9. 9.
    Collet, C., Gonzalez, M., Milachon, F.: Distributed system architecture for assisted annotation of video corpora. In: International Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, LREC, Valletta, Malte, pp. 49–52 (2010)Google Scholar
  10. 10.
    Gianni, F., Collet, C., Dalle, P.: Robust Tracking for Processing of Videos of Communication’s Gestures. In: Sales Dias, M., Gibet, S., Wanderley, M.M., Bastos, R. (eds.) GW 2007. LNCS (LNAI), vol. 5085, pp. 93–101. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  11. 11.
    Hamada, Y., Shimada, N., Shirai, Y.: Hand shape estimation using sequence of multi-ocular images based on transition network. In: Proceedings of the International Conference on Vision Interface (2002)Google Scholar
  12. 12.
    Ramamoorthy, A., Vaswani, N., Chaudhury, S., Banerjee, S.: Recognition of dynamic hand gestures. Pattern Recognition 36, 2069–2081 (2003)zbMATHCrossRefGoogle Scholar
  13. 13.
    Ahmad, T., Taylor, C., Lanitis, A., Cootes, T.: Tracking and recognising hand gestures, using statistical shape models. Image and Vision Computing 15, 345–352 (1997)CrossRefGoogle Scholar
  14. 14.
    Holden, E., Lee, G., Owens, R.: Australian sign language recognition. Machine Vision and Applications 16, 312–320 (2005)CrossRefGoogle Scholar
  15. 15.
    Tanibata, N., Shimada, N., Shirai, Y.: Extraction of hand features for recognition of sign language words. In: International Conference on Vision Interface, pp. 391–398 (2002)Google Scholar
  16. 16.
    Smith, P., da Vitoria Lobo, N., Shah, M.: Resolving hand over face occlusion. Image and Vision Computing 25, 1432–1448 (2007)CrossRefGoogle Scholar
  17. 17.
    Matthes, S., Hanke, T., Storz, J., Efthimiou, E., Dimiou, N., Karioris, P., Braffort, A., Choisier, A., Pelhate, J., Safar, E.: Elicitation tasks and materials designed for dicta-sign’s multi-lingual corpus. In: International Workshop on the Representation and Processing of Sign Languages: Corpora and Sign Language Technologies, LREC, Valletta, Malte, pp. 158–163 (2010)Google Scholar
  18. 18.
    Vassili, V.V., Sazonov, V., Andreeva, A.: A survey on pixel-based skin color detection techniques. In: Proc. Graphicon 2003, pp. 85–92 (2003)Google Scholar
  19. 19.
    Kovac, J., Peer, P., Solina, F.: Human skin color clustering for face detection. In: EUROCON International Conference on Computer as a Tool, vol. 2, pp. 144–148 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Matilde Gonzalez
    • 1
  • Christophe Collet
    • 1
  • Rémi Dubot
    • 1
  1. 1.IRIT (UPS - CNRS UMR 5505) Université Paul SabatierToulouse Cedex 9France

Personalised recommendations