Abstract
By integrating conversationally used natural language with graphical interfaces, gestural interaction looses the simplicity of direct manipulation. Input in one channel may change the meaning of input in another channel rather drastically. This introduces the wait problem, i.e. the problem of when and how long the system has to wait for input in other channels, before it triggers an action. Additionally, pointing devices and feedback may cause obtrusive effects on the natural synchronisation of deictic gestures and deictic expressions. Certain uses of gestures are observed that are distinct from natural communication and direct manipulation as well, e.g. focusing gestures. The discussion of these problems is based on our experience with the multimodal prototypes Mofa and Talky that we implemented in different variations, each showing a slightly different style of interaction. The notion of passive and active gesture forms is introduced as well as the notion of active and passive objects. Incremental natural language interpretation and the provision of incremental and preliminary feedback turn out to be important challenges for the upcoming technology of multimodal interfaces.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
eferences
Amtrup, J.W., H. Heine, and U. Jost (1996) Whats in a Word Graph, Evaluation and Enhancement of Word Lattices, Verbmobil Report 186. Universität Hamburg.
Bellalem, N. and L. Romary (1995) Reference interpretation in a multimodal environment combining speech and gestures”. In Proc. First International Workshop on Intelligence and Multimedia Interfaces: Research and Applications. Edinburgh University.
Faconti, G.P. and M. Massink (1997) A Syndetic Approach to Referring Phenomena in Multimodal Interaction. In Proceedings of the Workshop ‘Referring Phenomena in a Multimedia Context and Their Computational Treatment’, ACL/EACL 1997, Madrid.
Huls, C., E. Bos, and W. Claassen (1995) Automatic Referent Resolution of Deictic and Anaphoric Expressions, Computational Linguistics 21(1): 59–79.
Kaemmerer, B. and Ch. Maggioni (1995) Gesture Computer-from Research to Practice. In Proc. Conference on Real and Virtual Worlds, 1995.
Oviatt, S.L., A. DeAngeli, and K. Kuhn (1997) Integration and Synchronization of Input Modes during Multimodal Human-Computer Interaction. In Proc. Conference on Human Factors in Computing Systems: CHI’97. New York: ACM Press.
Oviatt, S.L. A. DeAngeli, and K. Kuhn (1999) Ten Myths of Multimodal Interaction. Communications of the ACM Vol. 42, No. 11.
Pittmann, J.A., I. Smith, P. Cohen, S. Oviatt, and T. Yang (1996) QuickSet: A Mul-timodal Interface for Military Simulation. In Proc. 6th Conference on Computer-Generated Forces and Behavioral Representation, University of Central Florida, 217–224.
Streit, M. and A. Krüger (1996) Eine agentenorientierte Architektur für multimediale Benutzerschnittstellen. In Online 96-Congressband VI, Hamburg.
Streit, M. (1997) Active and Passive Gestures-Problems with the Resolution of Deictic and Elliptic Expressions in a Multimodal System. In Proceedings of the Workshop’ Referring Phenomena in a Multimedia Context and Their Computational Treatment’, ACL/EACL 1997, Madrid.
Streit, M. (1999a) The Interaction of Speech, Deixis, and Graphics in the Multi-modal Office Agent Talky. In Proceedings of the ESCA Workshop on Interactive Dialogue in Multi-Modal Systems, ESCA and Center for Person Kommunikation Aalborg University, Aalborg, Denmark.
Streit, M. (1999b) Interaction of Speech, Deixis, and Graphical Interface, Proceedings of the Workshop on Deixis, Demonstration and Deictic Belief. In Proc. 11th European Summer School in Logic, Language and Information, Utrecht.
Sullivan, J.W. and S.W. Tyler (eds.) (1991) Intelligent User Interfaces. Frontier Series, New York: ACM Press.
Wahlster, W. (1991) User and Discourse Models for Multimodal Communication. In J.W. Sullivan and S.W. Tyler (eds.) Intelligent User Interfaces. New York: ACM Press, 45–67.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Streit, M. (2001). Why Are Multimodal Systems so Difficult to Build? - About the Difference between Deictic Gestures and Direct Manipulation. In: Bunt, H., Beun, R.J. (eds) Cooperative Multimodal Communication. CMC 1998. Lecture Notes in Computer Science(), vol 2155. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45520-5_11
Download citation
DOI: https://doi.org/10.1007/3-540-45520-5_11
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-42806-0
Online ISBN: 978-3-540-45520-2
eBook Packages: Springer Book Archive