Abstract
Developing multimodal interfaces is not only a matter of technology. Rather, it implies an adequate tailoring of the interface to the user’s communication needs. In command and control applications, the user most often has the initiative, and in that perspective gestures and speech (the user’s communication channels) have to be carefully studied to support a sensible interaction style. In this chapter, we introduce the notion of semantic frame to integrate gestures and speech in multimodal interfaces. We describe the main elements of a model that has been developed to integrate the use of both channels, and illustrate the model by two fully implemented systems. Possible extensions of the model are presented to improve the supported style, as technologies develop.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Amalberti, R., Carbonell, N., and Falzon, P. (1989) User’s representation of the machine in human-computer interaction, INRIA Research Report 1125.
Bisson, P., Kakez, S., Poulet-Mathis, C., Pouteau, X., and Cavazza, C. (1995) Augmented Reality for Telepresence, 4th Conference on Interface to Real and Virtual Worlds, Montpellier (France).
Bolt R. (1980) ‘Put-That-There’: Voice and Gesture at the Graphics Interface, Computer Graphics, 14, 262–270.
Bunt H. (1998) Issues in Multimodal Human-Computer Communication, in H. Bunt, R.J. Beun, and T. Borghuis (eds.) Multimodal Human-Computer Communication. Berlin: Springer, 1–12.
Buxton, W. (1986) There’s More to Interaction than Meets the Eye: Some Issues in Manual Input, in D. Norman and S. Draper (eds.) (1986), User Centered System Design: New Perspectives on Human-Computer Interaction, Hillsdale, New Jersey: Lawrence Erlbaum Associates, 319–337.
Cadoz, C., Lisowski, L., and Florens, J.-L. (1990) A Modular Feedback Keyboard Design, Computer Music Journal, 4:2, 47–51.
Cadoz, C. (1992) Le geste canal de communication homme-machine, 6th Autumn School, Thomson Campus, Jouy-en-Josas (F), September 7–11.
Falzon, P. (1983) Understanding a technical language: a schema-based approach, INRIA Research Report 237.
Feyereisen, P. (1987) Gesture and Speech, interactions and separations: A reply to McNeill (1985), in Psychological Review, 94:4, 493–498.
Feyereisen, P., Van de Wiele, F., and Dubois, F. (1988) The meaning of gestures: What can be understood without speech? in Cahiers de Psychologie Cognitive, 8:1, 3–25.
Gaiffe B., Romary L., and Pierrel J. (1991) References in a Multimodal Dialogue: Towards a Unified Process, in Proc. EUROSPEECH’91, Genova.
Gaiffe, B. (1992) Référence et dialogue homme-machine: vers un modèle adapté au multimodal, Ph.D. thesis, Université Nancy I.
Hauptmann, A. and Mc Avinney, P. (1993) Gesture with speech for graphic manipulation, Journal of Man-Machine Studies, 38, 231–249.
Johnston, M., Cohen, P., McGee, D., Oviatt, S., Pittman, J., and Smith, I. (1997) Unification-based Multimodal Integration, in Proc ACL97, Madrid, 281–288.
Kendon, A. (1986) Current issues in the study of gesture, in The Biological Foundations of Gestures: Motor and Semiotic Aspects, Nespoulos, Perron, Lecours (eds.), Hillsdale, New Jersey: Lawrence Erlbaum Associates.
Mc Kevitt, P. (1997) IntelliMedia TourGuide: understanding reference at the language/vision interface, in Proc European Science Foundation (ESF), Network on Converging Computing Methodologies in Astronomy (CCMA), Sonthofen (Germany).
McNeill, D. (1985) So you think gestures are nonverbal? in Psychological Review, 92:3, 350–371.
McNeill, D. (1987) So you do think gestures are nonverbal? Reply to Feyereisen, in Psychological Review, 94:4, 499–504.
Piwek, P., Beun, R., and Cremers, A. (1995) Dutch and English demonstratives: A comparison. IPO manuscript 1134 (submitted for publication to Computational Linguistics).
Pouteau, X., Romary, L., and Pierrel, J. (1993) A knowledge-based approach towards operative multimodal dialogues: MELODIA experiment, in Proc ESCA/NATO-RsG 10 Workshop on Speech Technology and its applications, Lautrach (Germany), 183–186.
Pouteau, X. (1995) Dialogue homme-machine multimodal: une communication naturelle pour l’opérateur? Ph.D. thesis, Université Henri Poincaré/Nancy I.
Pouteau, X., Krahmer, E., and Landsbergen, J. (1997) Robust Spoken Dialogue Management for Driver Information Systems, in Proc. Euro speech’97, Rhodos, 2207–2210.
Rasmussen, J. (1976) Information processing and human-machine interaction. Amsterdam: North Holland-Elsevier.
Rimé, B. and Schiaratura, L. (1991) Gesture and Speech, in Feldman, R., Rim’e, B. (eds.) Fundamentals of nonverbal behaviour, Cambridge University Press, 239–281.
Sidner, C. (1986) Focusing in the comprehension of definite anaphora, in Readings in Natural Language Processing, Palo Alto: Morgan-Kauffman, 363–394.
Wahlster, W. (1991) User and Discourse Models for Multimodal Communication, in Intelligent User Interfaces, ACM Press Series, Sullivan and Tyler (eds.), Reading, Mass.: Addison-Wesley, 45–67.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Pouteau, X. (2001). Interpretation of Gestures and Speech: A Practical Approach to Multimodal Communication. In: Bunt, H., Beun, R.J. (eds) Cooperative Multimodal Communication. CMC 1998. Lecture Notes in Computer Science(), vol 2155. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45520-5_10
Download citation
DOI: https://doi.org/10.1007/3-540-45520-5_10
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42806-0
Online ISBN: 978-3-540-45520-2
eBook Packages: Springer Book Archive