Fingerspelling Recognition through Classification of Letter-to-Letter Transitions

  • Susanna Ricco
  • Carlo Tomasi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5996)


We propose a new principle for recognizing fingerspelling sequences from American Sign Language (ASL). Instead of training a system to recognize the static posture for each letter from an isolated frame, we recognize the dynamic gestures corresponding to transitions between letters. This eliminates the need for an explicit temporal segmentation step, which we show is error-prone at speeds used by native signers. We present results from our system recognizing 82 different words signed by a single signer, using more than an hour of training and test video. We demonstrate that recognizing letter-to-letter transitions without temporal segmentation is feasible and results in improved performance.


Observation Model American Sign Language Letter Pair Dynamic Gesture Transitional Motion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tsechpenakis, G., Metaxas, D., Neidle, C.: Learning-based dynamic coupling of discrete and continuous trackers. Computer Vision and Image Understanding 104(2-3), 140–156 (2006)CrossRefGoogle Scholar
  2. 2.
    Starner, T., Pentland, A., Weaver, J.: Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(12), 1371–1375 (1998)CrossRefGoogle Scholar
  3. 3.
    Vogler, C., Metaxas, D.: A Framework for Recognizing the Simultaneous Aspects of American Sign Language. Computer Vision and Image Understanding 81(3), 358–384 (2001)zbMATHCrossRefGoogle Scholar
  4. 4.
    Fang, G., Gao, W., Chen, X., Wang, C., Ma, J.: Signer-independent Continuous Sign Language Recognition Based on SRN/HMM. In: IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time, pp. 90–95 (2001)Google Scholar
  5. 5.
    Allen, J., Asselin, P., Foulds, R.: American Sign Language Finger Spelling Recognition System. In: IEEE 29th Annual Northeast Bioengineering Conference, pp. 285–286 (2003)Google Scholar
  6. 6.
    Oz, C., Leu, M.: Recognition of Finger Spelling of American Sign Language with Artificial Neural Network Using Position/Orientation Sensors and Data Glove. In: 2nd International Symposium on Neural Networks, pp. 157–164 (2005)Google Scholar
  7. 7.
    Hernandez-Rebollar, J., Lindeman, R., Kyriakopoulos, N.: A Multi-Class Pattern Recognition System for Practical Finger Spelling Translation. In: 4th International Conference on Multimodal Interfaces, pp. 185–190 (2002)Google Scholar
  8. 8.
    Dreuw, P., Keysers, D., Deselaers, T., Ney, H.: Gesture Recognition Using Image Comparison Methods. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 124–128. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  9. 9.
    Feris, R., Turk, M., Raskar, R., Tan, K., Ohashi, G.: Exploiting Depth Discontinuities for Vision-Based Fingerspelling Recognition. In: IEEE Workshop on Real-time Vision for Human-Computer Interaction (2004)Google Scholar
  10. 10.
    Lamar, M., Bhuiyan, M., Iwata, A.: Hand Alphabet Recognition using Morphological PCA and Neural Networks. In: International Joint Conference on Neural Networks, pp. 2839–2844 (1999)Google Scholar
  11. 11.
    Tomasi, C., Petrov, S., Sastry, A.: 3D Tracking = Classification + Interpolation. In: International Conference on Computer Vision, pp. 1441–1448 (2003)Google Scholar
  12. 12.
    Birk, H., Moeslund, T., Madsen, C.: Real-Time Recognition of Hand Alphabet Gestures Using Principal Component Analysis. In: 10th Scandinavian Conference on Image Analysis, pp. 261–268 (1997)Google Scholar
  13. 13.
    Athitsos, V., Sclaroff, S.: Estimating 3D Hand Pose from a Cluttered Image. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 432–439 (2003)Google Scholar
  14. 14.
    Goh, P., Holden, E.: Dynamic Fingerspelling Recognition using Geometric and Motion Features. In: International Conference on Image Processing, pp. 2741–2744 (2006)Google Scholar
  15. 15.
    Videos used in experiments include clips from the John A. Logan College Interpreter Training Program ( and the DVDs Fast Expressive Fingerspelling Practice, Fingerspelled Stories from 10 to 45 words per minute (both available from, and Fingerspelling: Expressive and Receptive Fluency (
  16. 16.
    Padden, C.: Learning to fingerspell twice: Young signing children’s acquisition of fingerspelling. In: Marschark, M., Schick, B., Spencer, P. (eds.) Advances in Sign Language Development by Deaf Children, pp. 189–201. Oxford University Press, New York (2006)Google Scholar
  17. 17.
    Vogler, C., Metaxas, D.: Handshapes and Movements: Multiple-Channel American Sign Language Recognition. In: 5th International Workshop on Gesture and Sign Language Based Human-Computer Interaction, pp. 247–258 (2003)Google Scholar
  18. 18.
    Jones, M., Rehg, J.: Statistical Color Models with Application to Skin Detection. International Journal of Computer Vision 46(1), 81–96 (2002)zbMATHCrossRefGoogle Scholar
  19. 19.
    Rabiner, L.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  20. 20.
    Bakis, R.: Continuous speech recognition via centisecond acoustic states. Journal fo the Acoustical Society of America 59(S1), S97 (1976)CrossRefGoogle Scholar
  21. 21.
    Jelinek, F., Mercer, R., Bahl, L., Baker, J.: Perplexity–a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America 62(S1), S63 (1977)CrossRefGoogle Scholar
  22. 22.
    Jerde, T., Soechting, J., Flanders, M.: Coarticulation in Fluent Fingerspelling. Journal of Neuroscience 23(6), 2383 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Susanna Ricco
    • 1
  • Carlo Tomasi
    • 1
  1. 1.Department of Computer ScienceDuke UniversityDurham

Personalised recommendations