Abstract
In this chapter we discuss multimodal interface technology. We present examples of multimodal interfaces and show problems and opportunities. Fusion of modalities is discussed and some roadmap discussions on research in mul-timodality are summarized. This chapter also discusses future developments where, rather than communicating with a single computer, users communicate with their environment using multimodal interactions and where the environmental interface has perceptual competence that includes being able to interpret what is going on in the environment. We contribute roles to virtual humans in order to allow daily users of future computing environments to establish relationships with the environments, or more in particular, these virtual humans.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Argyle, M., and M. Cook [ 1976 ]. Gaze and Mutual Gaze. Cambridge University Press, Cambridge.
Broersen, A., and A. Nijholt [ 2002 ]. Developing a virtual piano playing environment. In Proc. IEEE International Conference on Advanced Learning Technologies (ICALT 2002), V. Petrushin, P. Kommers, Kinshuk, and I. Galeev (eds.), Kazan, Russia, pages 278–282.
Bunt, H., and R.-J. Beun (eds.) [2001]. Cooperative Multimodal Communication. CMC’98 Selected Papers, Springer.
Bunt, H., M. Kipp, M.T. Maybury, and W. Wahlster [ 2003 ]. Fusion and coordination for multimodal interactive information presentation. Chapter in Intelligent Information Presentation, O. Stock & M. Zancanaro (eds. ), Kluwer Academic Publishers.
Cassell, J., J. Sullivan, S. Prevost, and E. Churchill (eds.) [2000]. Embodied Conversational Agents. The MIT Press.
Darken, R.P, and J.L. Silbert [ 1996 ]. Way finding strategies and behaviors in virtual worlds. In Proc. CHI, pages 142–149.
Evers, M., and A. Nijholt [ 2000 ]. Jacob–an animated instruction agent for virtual reality. In Advances in Multimodal Interfaces–ICMI2000, Proc. Third International Conference on Multimodal Interfaces, Beijing, China, Lecture Notes in Computer Science 1948, T. Tan, Y. Shi, and W. Gao (eds.), Springer-Verlag, Berlin, pages 526–533.
Gebhard, P. [ 2001 ]. Enhancing embodied intelligent agents with affective user modelling. UM2001, 8th International Conference, J. Vassileva and P. Gmytrasiewicz, (eds.), Berlin, Springer.
Heylen, D., I. van Es, B. van Dijk, and A. Nijholt [ 2003 ]. Experimenting with the gaze of a conversational agent. Chapter in Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, J. van Kuppevelt, L. Dybkjaer, and N.O. Bernsen (eds. ), Kluwer Academic Publishers.
Hofs, D., R. op den Akker, and A. Nijholt [ 2003 ]. A generic architecture and dialogue model for multimodal interaction. Submitted for publication.
Hook, K., et al. [ 1988 ]. Towards a framework for design and evaluation of navigation in electronic spaces. Personal deliverable for the EC.
Hospers, M., E. Kroezen, A. Nijholt, R. op den Akker, and D. Heylen [2003]. Developing a generic agent-based intelligent tutoring system and applying it to nurse education. In Proc. IEEE International Conference on Advanced Language Technologies (ICALT ‘03), Athens, Greece.
Johnston, M., P. Cohen, D. McGee, S. Oviat, J. Pittman, and I. Smith [ 1997 ]. Unification-based multimodal integration. In Proc. of the 35th Annual ACL Conference, New Jersey, pages 281–288.
Johnston, M., and S. Bangalore [2000]. Finite-state multimodal parsing and understanding. In Proc. ofCOLING-2000, Saarbriicken, Germany.
Kendon, A. [ 1980 ]. Gesticulation and speech: Two aspects of the process of utterance. In The Relation of Verbal and Nonverbal Communication, M.R. Key (ed.), Mouton, The Hague, the Netherlands.
Lester, J.C., et al. [ 1997 ]. The persona effect: Affective impact of animated pedagogical agents. CHI’97 Human Factors in Computing Systems, ACM, pages 359–356.
Luin, J. van, R. op den Akker, and A. Nijholt [ 2001 ]. A dialogue agent for navigation support in virtual reality. Extended abstracts ACM SIGCHI Conference CHI 2001: Anyone. Anywhere. ACM, J. Jacko and A. Sears (eds.), Seattle, pages 117–118.
Maybury, M., and W. Wahlster (eds.) [1988]. Readings in Intelligent User Interfaces. Morgan Kaufmann Press.
McCowan, I., S. Bengio, D. Gatica-Perez, G. Lathoud, F. Monay, D. Moore, P. Wellner, and H. Bourlard [2003]. Modeling human interaction in meetings. In Proc. IEEE ICASSP, Hong Kong.
Mikic, I., K. Huang, and M. Trivedi [2000]. Activity monitoring and summarization for an intelligent meeting room. In Proc. IEEE Workshop on Human Motion, Austin, Texas.
Nigay, L., and J. Coutaz [ 1995 ]. A generic platform for addressing the multimodal challenge. In Proc. ACM CHI, pages 98–105.
Nijholt, A., and J. Hulstijn [ 2000 ]. Multimodal interactions with agents in virtual worlds. Chapter 8 in Future Directions for Intelligent Information Systems and Information Science, N. Kasabov (ed.), Physica-Verlag, Springer, Heidelberg, pages 148–173.
Nijholt, A. [ 2002 ]. Embodied agents: A new impetus to humor research. The April Fools Day Workshop on Computational Humour, O. Stock, C. Strapparava, and A. Nijholt (eds.), In Proc. Twente Workshop on Language Technology 20 (TWLT 20), Trento, Italy, pages 101-111
Nijholt, A., J. Zwiers, and B. van Dijk [ 2003 ]. Maps, agents and dialogue for exploring a virtual world. Chapter in Web Computing. J. Aguilar, N. Callaos, and E.L. Leiss (eds).
Oviat, S., P. Cohen, L. Wu, J. Vergo, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro [ 2000 ]. Designing the User Interface for Multimodal Speech and Gesture Applications: State-ofthe-Art Systems and Research Directions for 2000 and Beyond. Report.
Potter, D. (WP5-Team) [ 2003 ]. Future Workspaces: A Strategic Roadmap for Defining Distributed Engineering Workspaces of the Future. IST-2001–38346 deliverable.
Reeves, B., and C. Nass [ 1996 ]. The Media Equation. Cambridge University Press, Cambridge.
Shechtman, N., and L.M. Horowitz [2003]. Media inequality in conversation: how people behave differently when interacting with computers and people. In Proc. SIGCHI-ACM CHI 2003: New Horizons, ACM, New York, pages 281–288.
Stronks, B., A. Nijholt, P. van der Vet, and D. Heylen [ 2002 ]. Designing for friendship: Becoming friends with your EC A. In Proc. Embodied Conversational Agents–Let’s Specify and Evaluate Them!, A. Marriott, C. Pelachaud, T. Rist, and Zs. Ruttkay (eds.), Bologna, Italy, pages 91–97.
Torres, O., J. Cassell, and S. Prévost [1997]. Modeling gaze behavior as a function of discourse structure. In Proc. First International Workshop on Human Computer Conversations. Bella-gio, Italy.
Zobl, M., F. Wallhoff, and G. Rigoll [2003]. Action recognition in meeting scenarios using global motion features. In Proc. IEEE International Workshop on Performance Evaluation of Tracking and Surveillance.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Nijholt, A. (2004). Multimodality and Ambient Intelligence. In: Verhaegh, W.F.J., Aarts, E., Korst, J. (eds) Algorithms in Ambient Intelligence. Philips Research, vol 2. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-0703-9_2
Download citation
DOI: https://doi.org/10.1007/978-94-017-0703-9_2
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-6490-5
Online ISBN: 978-94-017-0703-9
eBook Packages: Springer Book Archive