Abstract
We propose a new principle for recognizing fingerspelling sequences from American Sign Language (ASL). Instead of training a system to recognize the static posture for each letter from an isolated frame, we recognize the dynamic gestures corresponding to transitions between letters. This eliminates the need for an explicit temporal segmentation step, which we show is error-prone at speeds used by native signers. We present results from our system recognizing 82 different words signed by a single signer, using more than an hour of training and test video. We demonstrate that recognizing letter-to-letter transitions without temporal segmentation is feasible and results in improved performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Tsechpenakis, G., Metaxas, D., Neidle, C.: Learning-based dynamic coupling of discrete and continuous trackers. Computer Vision and Image Understanding 104(2-3), 140–156 (2006)
Starner, T., Pentland, A., Weaver, J.: Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(12), 1371–1375 (1998)
Vogler, C., Metaxas, D.: A Framework for Recognizing the Simultaneous Aspects of American Sign Language. Computer Vision and Image Understanding 81(3), 358–384 (2001)
Fang, G., Gao, W., Chen, X., Wang, C., Ma, J.: Signer-independent Continuous Sign Language Recognition Based on SRN/HMM. In: IEEE ICCV Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time, pp. 90–95 (2001)
Allen, J., Asselin, P., Foulds, R.: American Sign Language Finger Spelling Recognition System. In: IEEE 29th Annual Northeast Bioengineering Conference, pp. 285–286 (2003)
Oz, C., Leu, M.: Recognition of Finger Spelling of American Sign Language with Artificial Neural Network Using Position/Orientation Sensors and Data Glove. In: 2nd International Symposium on Neural Networks, pp. 157–164 (2005)
Hernandez-Rebollar, J., Lindeman, R., Kyriakopoulos, N.: A Multi-Class Pattern Recognition System for Practical Finger Spelling Translation. In: 4th International Conference on Multimodal Interfaces, pp. 185–190 (2002)
Dreuw, P., Keysers, D., Deselaers, T., Ney, H.: Gesture Recognition Using Image Comparison Methods. In: Gibet, S., Courty, N., Kamp, J.-F. (eds.) GW 2005. LNCS (LNAI), vol. 3881, pp. 124–128. Springer, Heidelberg (2006)
Feris, R., Turk, M., Raskar, R., Tan, K., Ohashi, G.: Exploiting Depth Discontinuities for Vision-Based Fingerspelling Recognition. In: IEEE Workshop on Real-time Vision for Human-Computer Interaction (2004)
Lamar, M., Bhuiyan, M., Iwata, A.: Hand Alphabet Recognition using Morphological PCA and Neural Networks. In: International Joint Conference on Neural Networks, pp. 2839–2844 (1999)
Tomasi, C., Petrov, S., Sastry, A.: 3D Tracking = Classification + Interpolation. In: International Conference on Computer Vision, pp. 1441–1448 (2003)
Birk, H., Moeslund, T., Madsen, C.: Real-Time Recognition of Hand Alphabet Gestures Using Principal Component Analysis. In: 10th Scandinavian Conference on Image Analysis, pp. 261–268 (1997)
Athitsos, V., Sclaroff, S.: Estimating 3D Hand Pose from a Cluttered Image. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 432–439 (2003)
Goh, P., Holden, E.: Dynamic Fingerspelling Recognition using Geometric and Motion Features. In: International Conference on Image Processing, pp. 2741–2744 (2006)
Videos used in experiments include clips from the John A. Logan College Interpreter Training Program ( www.jalc.edu/ipp ) and the DVDs Fast Expressive Fingerspelling Practice, Fingerspelled Stories from 10 to 45 words per minute (both available from www.drsign.com), and Fingerspelling: Expressive and Receptive Fluency ( www.dawnsign.com )
Padden, C.: Learning to fingerspell twice: Young signing children’s acquisition of fingerspelling. In: Marschark, M., Schick, B., Spencer, P. (eds.) Advances in Sign Language Development by Deaf Children, pp. 189–201. Oxford University Press, New York (2006)
Vogler, C., Metaxas, D.: Handshapes and Movements: Multiple-Channel American Sign Language Recognition. In: 5th International Workshop on Gesture and Sign Language Based Human-Computer Interaction, pp. 247–258 (2003)
Jones, M., Rehg, J.: Statistical Color Models with Application to Skin Detection. International Journal of Computer Vision 46(1), 81–96 (2002)
Rabiner, L.: A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE 77(2), 257–286 (1989)
Bakis, R.: Continuous speech recognition via centisecond acoustic states. Journal fo the Acoustical Society of America 59(S1), S97 (1976)
Jelinek, F., Mercer, R., Bahl, L., Baker, J.: Perplexity–a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America 62(S1), S63 (1977)
Jerde, T., Soechting, J., Flanders, M.: Coarticulation in Fluent Fingerspelling. Journal of Neuroscience 23(6), 2383 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ricco, S., Tomasi, C. (2010). Fingerspelling Recognition through Classification of Letter-to-Letter Transitions. In: Zha, H., Taniguchi, Ri., Maybank, S. (eds) Computer Vision – ACCV 2009. ACCV 2009. Lecture Notes in Computer Science, vol 5996. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12297-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-12297-2_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12296-5
Online ISBN: 978-3-642-12297-2
eBook Packages: Computer ScienceComputer Science (R0)