Abstract
In this paper we present a framework for recognizing American Sign Language (ASL). The main challenges in developing scalable recognition systems are to devise the basic building blocks from which to build up the signs, and to handle simultaneous events, such as signs where both the hand moves and the handshape changes. The latter challenge is particularly thorny, because a naive approach to handling them can quickly result in a combinatorial explosion.
We loosely follow the Movement-Hold model to devise a breakdown of the signs into their constituent phonemes, which provide the fundamental building blocks. We also show how to integrate the handshape into this breakdown, and discuss what handshape representation works best. To handle simultaneous events, we split up the signs into a number of channels that are independent from one another. We validate our framework in experiments with a 22-sign vocabulary and up to three channels.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Valli, C., Lucas, C.: Linguistics of American Sign Language: An Introduction. Gallaudet University Press, Washington DC (1995)
Liddell, S.K., Johnson, R.E.: American Sign Language: The phonological base. Sign Language Studies 64, 195–277 (1989)
Vogler, C., Metaxas, D.: Toward scalability in ASL recognition: Breaking down signs into phonemes. In: Braffort, A., Gibet, S., Teil, D., Gherbi, R., Richardson, J. (eds.) GW 1999. LNCS (LNAI), vol. 1739, pp. 211–224. Springer, Heidelberg (2000)
Vogler, C., Metaxas, D.: A framework for recognizing the simultaneous aspects of American Sign Language. Computer Vision and Image Understanding, 358–384 (2001)
Pavlovic, V., Sharma, R., Huang, T.S.: Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 677–695 (1997)
Wang, C., Gao, W., Ma, J.: A real-time large vocabulary recognition system for Chinese Sign Language. In: Wachsmuth, I., Sowa, T. (eds.) GW 2001. LNCS (LNAI), vol. 2298, pp. 86–95. Springer, Heidelberg (2002)
Starner, T., Weaver, J., Pentland, A.: Real-time American Sign Language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 1371–1375 (1998)
Fang, G., Gao, W., Chen, X., Wang, C., Ma, J.: Signer-independent continuous sign language recognition based on SRN/HMM. In: Wachsmuth, I., Sowa, T. (eds.) GW 2001. LNCS (LNAI), vol. 2298, pp. 76–85. Springer, Heidelberg (2002)
Hienz, H., Kraiss, K.F., Bauer, B.: Continuous sign language recognition using hidden Markov models. In: Tang, Y. (ed.) ICMI 1999, Hong Kong, pp. IV10–IV15 (1999)
Vogler, C., Metaxas, D.: Adapting hidden Markov models for ASL recognition by using three-dimensional computer vision methods. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, Orlando, FL, pp. 156–161 (1997)
Bauer, B., Kraiss, K.F.: Towards an Automatic Sign Language Recognition System Using Subunits. In: Wachsmuth, I., Sowa, T. (eds.) GW 2001. LNCS (LNAI), vol. 2298, pp. 64–75. Springer, Heidelberg (2002)
Bauer, B., Kraiss, K.F.: Video-Based Sign Recognition using Self-Organizing Subunits. In: International Conference on Pattern Recognition (2002)
Liang, R.H., Ouhyoung, M.: A real-time continuous gesture recognition system for sign language. In: Proceedings of the Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan, pp. 558–565 (1998)
Stokoe, W.C.: Sign Language Structure: An Outline of the Visual Communication System of the American Deaf. Studies in Linguistics: Occasional Papers 8. Linstok Press, Silver Spring, MD (1960) (Revised 1978)
Brentari, D.: Sign language phonology: ASL. In: Goldsmith, J.A., ed.: The Handbook of Phonological Theory. In: Blackwell Handbooks in Linguistics. Blackwell, Oxford, pp. 615–639 (1995)
Perlmutter, D.: Sonority and syllable structure in American Sign Language. Linguistic Inquiry 23, 407–442 (1992)
Battison, R.: In: Lexical borrowing in American Sign Language. Linstok Press, Silver Spring, MD, pp. 19–58 (1978); Reprinted as Analyzing Signs in Lucas, C. and Valli, C. Linguistics of American Sign Language (1995)
Sandler, W.: Representing Handshapes. In: Edmondson, W.H., Wilbur, R. (eds.) International Review of Sign Linguistics, vol. 1, pp. 115–158. Lawrence Erlbaum Associates, Inc., Mahwah (1996)
Rabiner, L.R.: A tutorial on Hidden Markov Models and selected applications in speech recognition. Proceedings of the IEEE 77, 257–286 (1989)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Vogler, C., Metaxas, D. (2004). Handshapes and Movements: Multiple-Channel American Sign Language Recognition. In: Camurri, A., Volpe, G. (eds) Gesture-Based Communication in Human-Computer Interaction. GW 2003. Lecture Notes in Computer Science(), vol 2915. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24598-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-540-24598-8_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21072-6
Online ISBN: 978-3-540-24598-8
eBook Packages: Springer Book Archive