Abstract
This paper presents the processing of tactile and oral multimodal events that occur in inputs of the GEORAL tactile system. First we shall describe the principal results of observations on the behaviour of users in a multimodal interaction environment. These observations, combined with consideration of the technical possibilities and limitations of speech recognition, lead us to proposing a system architecture. The chapter ends with details of the modeling of communicative acts, based on plan operators, and the principal stages of the merging of tactile and oral acts.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Allen, J. (1987) Natural Language Understanding. Redwood City: Benjamin Cummings.
Bourguet, M. L. (1992) Conception et réalisation d'une interface de dialogue personnemachine multimodale. Thèse de troisième cycle, Institut National Polytechnique de Grenoble.
Cohen, P. R. (1984) The Pragmatics of Referring and the Modality of Communication. Computational Linguistics 10 (2), 1984, 97–146.
Cohen, P. R. (1992) The Role of Natural Language in a Multimodal Interface. Proc. 2nd FRIEND21, International Symposium on Next Generation Human Interface Technology, Tokyo, November 1991, 143–149.
Feiner, S. K. and McKeown, K. R. (1990) Coordinating Text and Graphics in Explanation Generation. Proc. AAAI-90, vol. 1, 442–449.
Gavignet, F., Guyomard, M., Siroux, J. (1991) Implementing an oral and geographic multimodal application: the Géoral project. In Pre-proceedings of Second Venaco Workshop on the Structure of Multimodal Dialogue, Acquafredda di Maratea, Italy, September 1991.
Green, G. M. (1987) Pragmatics and Natural Language Understanding. Hillsdale, New Jersey: Lawrence Erlbaum.
Guyomard, M., Siroux, J. (1989) Suggestive and corrective answers: a simple mechanism. In M. M. Taylor, D. G. Bouwhuis, and F. Néel (eds) The Structure of Multimodal Dialog, Amsterdam: North Holland, 361–374.
Guyomard, M., Le Meur, D., Poignonnec, S., and Siroux, J. (1995) Experimental work for the dual usage of voice and touch screen for a cartographic application. Proc. ESCA Tutorial and Research Workshop on Spoken Dialogue Systems, VigsØ, Denmark, May 30–June 2, 1995, 153–156.
Lee, J. (1991) Graphics and Natural Language in Multi-Modal Dialogues. Pre-proceedings of Second Venaco Workshop on the Structure of Multimodal Dialogue, Acquafredda di Maratea, Italy, September, 1991.
Litman, D.J. and Allen, J.A. (1987) A Plan Recognition Model for Subdialogue in Conversations. Cognitive Science 11, 163–200.
Maybury, M. T. (1991) Planning Multimedia Explanations Using Communicative Acts. Proc. Ninth AAAI National Conference on Artificial Intelligence, Anaheim, CA, July 1991, 61–66.
Maybury, M. T. (1992) Communicative Acts for Explanation Generation. International Journal of Man-Machine Studies, 37 (2), 135–172.
Multon, F. (1994) GEORAL tactile un système multimodal. Rapport de DEA, IFSIC, Université de Rennes 1.
Oviatt, S., Cohen, P. R., Fong, M., and Frank, M. (1992) A rapid semi-automatic simulation technique for investigating interactive speech and handwriting. Proc. 7th International Conference on Spoken Language Processing, Banff, Canada, October 1992, vol. 2, 1351–1354.
Tyler, S.W., Schlossberg, J.L. and Cook, L.K. (1991) CHORIS: An Intelligent Interface Architecture for Multimodal Interaction. Proc. AAAI91 Workshop on Intelligent Multimedia Interfaces, Anaheim, CA, July 1991.
Wachtel, T. (1985) Discourse structure. LOKI Report NLI1-1.1, Esprit Project 107, Research Unit for Information Science and Artificial Intelligence, University of Hamburg.
Wahlster, W., André, E., Graf, W., and Rist, T. (1991) Designing Illustrated Texts: How Language Production is Influenced by Graphics Generation. Proc. EACL 91, Berlin, April 1991, 8–14.
Zancanaro, M., Stock, O., and Strapparava, C. (1993) Dialog Cohesion Sharing and Adjusting in an Enhanced Multimodal Environment. Proc. IJCAI 1993, Chambéry, France, 1230–1235.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1998 Springer-Verlag
About this paper
Cite this paper
Siroux, J., Guyomard, M., Multon, F., Remondeau, C. (1998). Modeling and processing of oral and tactile activities in the GEORAL system. In: Bunt, H., Beun, RJ., Borghuis, T. (eds) Multimodal Human-Computer Communication. CMC 1995. Lecture Notes in Computer Science, vol 1374. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0052315
Download citation
DOI: https://doi.org/10.1007/BFb0052315
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-64380-7
Online ISBN: 978-3-540-69764-0
eBook Packages: Springer Book Archive