Skip to main content
SpringerLink
Log in

User Interface Patterns for Multimodal Interaction

  • Conference paper
Transactions on Pattern Languages of Programming III

Part of the book series: Lecture Notes in Computer Science ((TPLOP,volume 7840))

Abstract

Multimodal interaction aims at more flexible, more robust, more efficient and more natural interaction than can be achieved with traditional unimodal interactive systems. For this, the developer needs some design support in order to select appropriate modalities, to find appropriate modality combinations and to implement promising modality adaptation strategies. This paper presents first patterns for multimodal interaction and focuses on patterns for “fast input”, “robust interaction” and patterns for “flexible interaction”. Before these patterns are outlined in detail, an introduction to the field of multimodal interaction is given and the pattern identification process that was the basis of this work is presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 54.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 72.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Oviatt, S.L., Kuhn, K.: Referential features and linguistic indirection in multimodal language. In: Proceedings of the International Conference on Spoken Language Processing, vol. 6, pp. 2339–2342. ASSTA (1998)

    Google Scholar 

  2. Niedermaier, F.B.: Entwicklung und Bewertung eines Rapid-Prototyping Ansatzes zur multimodalen Mensch-Maschine-Interaktion im Kraftfahrzeug. PhD thesis, Fakultät für Elektrotechnik und Informationstechnik der Technischen Universität München (2003)

    Google Scholar 

  3. Dragičević, P.: Un modèle d’interaction en entrée pour des systèmes interactifs multi-dispositifs hautement configurables. PhD thesis, Université de Nantes, école doctorale sciences et technologies de l’information et des matérieaux (March 2004), http://www.dgp.toronto.edu/~dragice/these/html/memoire_dragicevic.html (checked: June 20, 2008)

  4. Duarte, C., Carriço, L.: A conceptual framework for developing adaptive multimodal applications. In: IUI 2006: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 132–139. ACM, New York (2006)

    Google Scholar 

  5. Bernsen, N.O.: Multimodality in language and speech systems - from theory to design support tool. Lectures at the 7th European Summer School on Language and Speech Communication (ESSLSC) (July 1999), http://www.nis.sdu.dk/~nob/stockholm.zip (checked: June 20, 2008)

  6. Bürgy, C.: An Interaction Constraints Model for Mobile and Wearable Computer-Aided Engineering Systems in Industrial Applications. PhD thesis, Department of Civil and Environmental Engineering, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA (2002)

    Google Scholar 

  7. Obrenović, Z., Abascal, J., Starčević, D.: Universal accessibility as a multimodal design issue. Commun. ACM 50(5), 83–88 (2007)

    Google Scholar 

  8. Ratzka, A., Wolff, C.: A pattern-based methodology for multimodal interaction design. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2006. LNCS (LNAI), vol. 4188, pp. 677–686. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  9. Coram, T., Lee, J.: Experiences – a pattern language for user interface design (1996), http://www.maplefish.com/todd/papers/experiences/Experiences.html (checked: June 20, 2008)

  10. van Welie, M., Trætteberg, H.: Interaction patterns in user interfaces. In: Proceedings of the Seventh Pattern Languages of Programs Conference, Monticello, Illinois, USA (2000), http://www.cs.vu.nl/~martijn/patterns/PLoP2k-Welie.pdf (checked: June 20, 2008)

  11. Tidwell, J.: Common ground: A pattern language for human-computer interface design (1999), http://www.mit.edu/~jtidwell/common_ground.html (checked: June 20, 2008)

  12. Tidwell, J.: Designing Interfaces: Patterns for Effective Interaction Design. O’Reilly (2005)

    Google Scholar 

  13. Benoît, C., Martin, J.C., Pelachaud, C., Schomaker, L., Suhm, B.: Audio-visual and multimodal speech systems. In: Gibbon, D. (ed.) Handbook of Standards and Resources for Spoken Language Systems - Supplement (1998)

    Google Scholar 

  14. Charwat, H.J.: Lexikon der Mensch-Maschine-Kommunikation. Oldenbourg (1992)

    Google Scholar 

  15. Schomaker, L., Nijtmans, J., Camurri, A., Lavagetto, F., Morasso, P., Benoît, C., Guiard-Marigny, T., Goff, B.L., Robert-Ribes, J., Adjoudani, A., Defée, I., Münch, S., Hartung, K., Blauert, J.: A taxonomy of multimodal interaction in the human information processing system. Technical report (February 1995)

    Google Scholar 

  16. Hedicke, V.: Multimodalität in mensch-maschine schnittstellen. In: Timpe, K.P., Kolrep, H. (eds.) Mensch-Maschine-Systemtechnik, Konzepte, Modellirung, Gestaltung, Evaluation, Symposion, pp. 203–232 (2002)

    Google Scholar 

  17. Bernsen, N.O.: A reference model for output information in intelligent multimedia presentation systems. In: Faconti, G.P., Rist, T. (eds.) ECAI 1996 Workshop: Towards a Standard Reference Model for Intelligent Multimedia Systems (1996)

    Google Scholar 

  18. Bernsen, N.O.: A toolbox of output modalities: Representing output information in multimodal interfaces (1997), http://www.nis.sdu.dk/publications/papers/toolbox_paper/index.html (checked: June 20, 2008)

  19. Nigay, L., Coutaz, J.: A design space for multimodal systems: concurrent processing and data fusion. In: Proceedings of INTERCHI 1993 Conference on Human Factors in Computing Systems, pp. 172–178. ACM Press (1993), http://iihm.imag.fr/publs/1993/InterCHI93_DataFusion.pdf (checked: June 20, 2008)

  20. Nigay, L., Coutaz, J.: A generic platform for addressing the multimodal challenge. In: CHI 1995: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 98–105. ACM Press, Addison-Wesley Publishing Co., New York (1995)

    Google Scholar 

  21. Varela, F., Thompson, E., Rosch, E.: The embodied mind: Cognitive science and human experience. MIT Press, Cambridge (1991)

    Google Scholar 

  22. Shneiderman, B., Plaisant, C.: Designing the User Interface. Strategies for Effective Human-Computer Interaction. Person Addison-Wesley, Boston (2005)

    Google Scholar 

  23. Denes, P., Pinson, E.: The Speech Chain: The Physics and Biology of Spoken Language, 2nd edn. W.H. Freeman and Company, New York (1993)

    Google Scholar 

  24. Raskin, J.: The humane interface: new directions for designing interactive systems. ACM Press, Addison-Wesley Publishing Co., New York (2000)

    Google Scholar 

  25. Steinmetz, R.: Multimedia-Technologie: Einführung und Grundlagen. Springer (1993)

    Google Scholar 

  26. Bouchet, J., Nigay, L., Balzagette, D.: ICARE: approche à composants pour l’interaction multimodale. In: Mobilité & Ubiquité / UbiMob 2004: Proceedings of the 1st French-speaking Conference on Mobility and Ubiquity Computing, pp. 36–43. ACM Press, New York (2004)

    Google Scholar 

  27. Ratzka, A.: Identifying user interface patterns from pertinent multimodal interaction use cases. In: Herczeg, M., Kindsmüller, M.C. (eds.) Mensch & Computer 2008 – 8. fachübergreifende Konferenz für interaktive und kooperative Medien – Viel Mehr Interaktion, Lübeck, Oldenburg Wissenschaftsverlag, pp. 347–356 (September 2008)

    Google Scholar 

  28. Ratzka, A.: Steps in Identifying Interaction Design Patterns for Multimodal Systems. In: Forbrig, P., Paternò, F. (eds.) HCSE/TAMODIA 2008. LNCS, vol. 5247, pp. 58–71. Springer, Heidelberg (2008)

    Google Scholar 

  29. Bernsen, N.O.: Multimodality in language and speech systems – from theory to design support tool. In: Granström, B. (ed.) Multimodality in Language and Speech Systems. Kluwer, Dordrecht (2001), http://www.nis.sdu.dk/demos/multimodality/multimodality.pdf (checked: June 20, 2008)

  30. UDC: Universal designers & consultants, http://www.universaldesign.com/ (checked: November 6, 2010)

  31. Erlandson, R.F.: Universal and Accessible Design for Products, Services, and Processes. CRC Press, Boca Raton (2008)

    Google Scholar 

  32. W3C: Web content accessibility guidelines 1.0. W3c recommendation, W3C (May 1999), http://www.w3.org/TR/WAI-WEBCONTENT/ (checked: June 20, 2008)

  33. Thatcher, J.: Constructing Accessible Web Sites. Glasshaus, Birmingham (2002)

    Google Scholar 

  34. Kannengiesser, I., Prickartz, B.: Web-Ergonomie und Barrierefreiheit im Internet. Ferger, Bergisch Gladbach (2006)

    Google Scholar 

  35. Glinert, E.P.: Ensuring access for people with disabilities to the national information infrastructure and multimedia computing. SIGCAPH Comput. Phys. Handicap. (59), 10–16 (1997)

    Google Scholar 

  36. Bellik, Y., Burger, D.: Multimodal interfaces: new solutions to the problem of computer accessibilty for the blind. In: CHI 1994: Conference Companion on Human Factors in Computing Systems, pp. 267–268. ACM Press, New York (1994)

    Google Scholar 

  37. Uzan, G., Teixeira, A.: Speech-based interaction as seen by blind users: from services evaluation to the evaluation of an interaction model. In: IHM 2003: Proceedings of the 15th French-speaking Conference on Human-Computer Interaction / 15ème Conférence Francophone sur l’Interaction Homme-Machine, pp. 174–181. ACM Press, New York (2003)

    Google Scholar 

  38. Zajicek, M.: Patterns for encapsulating speech interface design solutions for older adults. In: CUU 2003: Proceedings of the 2003 Conference on Universal Usability, pp. 54–60. ACM Press, New York (2003)

    Google Scholar 

  39. Zajicek, M., Morrissey, W.: Multimodality and interactional differences in older adults. Universal Access in the Information Society 2(2), 125–133 (2003)

    Article  Google Scholar 

  40. Emery, V.K., Edwards, P.J., Jacko, J.A., Moloney, K.P., Barnard, L., Kongnakorn, T., Sainfort, F., Scott, I.U.: Toward achieving universal usability for older adults through multimodal feedback. In: CUU 2003: Proceedings of the 2003 Conference on Universal Usability, pp. 46–53. ACM Press, New York (2003)

    Google Scholar 

  41. Vitense, H.S., Jacko, J.A., Emery, V.K.: Multimodal feedback: establishing a performance baseline for improved access by individuals with visual impairments. In: Assets 2002: Proceedings of the Fifth International ACM Conference on Assistive Technologies, pp. 49–56. ACM Press, New York (2002)

    Google Scholar 

  42. Jacko, J.A., Scott, I.U., Sainfort, F., Barnard, L., Edwards, P.J., Emery, V.K., Kongnakorn, T., Moloney, K.P., Zorich, B.S.: Older adults and visual impairment: what do exposure times and accuracy tell us about performance gains associated with multimodal feedback? In: CHI 2003: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 33–40. ACM Press, New York (2003)

    Google Scholar 

  43. Jacko, J.A., Barnard, L., Kongnakorn, T., Moloney, K.P., Edwards, P.J., Emery, V.K., Sainfort, F.: Isolating the effects of visual impairment: exploring the effect of amd on the utility of multimodal feedback. In: CHI 2004: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 311–318. ACM Press, New York (2004)

    Google Scholar 

  44. Edwards, P.J., Barnard, L., Emery, V.K., Yi, J.S., Moloney, K.P., Kongnakorn, T., Jacko, J.A., Sainfort, F., Oliver, P.R., Pizzimenti, J., Bade, A., Fecho, G., Shallo-Hoffmann, J.: Strategic design for users with diabetic retinopathy: factors influencing performance in a menu-selection task. In: Assets 2004: Proceedings of the 6th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 118–125. ACM Press, New York (2004)

    Google Scholar 

  45. Manaris, B., Harkreader, A.: Suitekeys: a speech understanding interface for the motor-control challenged. In: Assets 1998: Proceedings of the Third International ACM Conference on Assistive Technologies, pp. 108–115. ACM Press, New York (1998)

    Google Scholar 

  46. Hwang, F., Keates, S., Langdon, P., Clarkson, P.J., Robinson, P.: Perception and haptics: towards more accessible computers for motion-impaired users. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive user Interfaces, pp. 1–9. ACM Press, New York (2001)

    Google Scholar 

  47. Thevenin, D., Coutaz, J.: Plasticity of user interfaces: Framework and research agenda. In: Sasse, A., Johnson, C. (eds.) Human-Computer Interaction – INTERACT 1999 , IFIP TC, vol. 13. IOS Press (1999)

    Google Scholar 

  48. Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)

    Article  Google Scholar 

  49. Calvary, G., Dâassi, O., Balme, L., Demeure, A.: Towards a new generation of widgets for supporting software plasticity: The ”Comet”. In: Feige, U., Roth, J. (eds.) EHCI-DSVIS 2004. LNCS, vol. 3425, pp. 306–324. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  50. Jabarin, B., Graham, T.C.N.: Architectures for widget-level plasticity. In: Jorge, J.A., Jardim Nunes, N., Falcão e Cunha, J. (eds.) DSV-IS 2003. LNCS, vol. 2844, pp. 124–138. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  51. Bass, L., Faneuf, R., Little, R., Mayer, N., Pellegrino, B., Reed, S., Seacord, R., Sheppard, S., Szezur, M.R.: Arch, a metamodel for the runtime architecture of an interactive system. uims tool developers workshop. SIGCHI Bulletin 24(1) (1992)

    Google Scholar 

  52. Coutaz, J.: Software architecture modeling for user interfaces. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, pp. 38–49. Wiley, Chichester (1994)

    Google Scholar 

  53. Frankel, D.S.: Model Driven ArchitectureTM. Applying MDATM to Enterprise Computing. OMG Press, Wiley, Indianapolis, Indiana (2003)

    Google Scholar 

  54. Coutaz, J., Crowley, J.L., Dobson, S., Garlan, D.: Context is key. Commun. ACM 48(3), 49–53 (2005)

    Article  Google Scholar 

  55. Wickens, C.D.: The structure of attentional resources. In: Nickerson, R.S. (ed.) Attention and Performance VIII, pp. 239–257. Lawrence Erlbaum, Hillsdale (1980)

    Google Scholar 

  56. Wickens, C.D.: Engineering Psychology and Human Performance. Harper Collins, New York (1992)

    Google Scholar 

  57. Bengler, K., Geutner, P., Steffens, F.: ,,eyes free – hands free“ oder ,,zeit der stille“. ein demonstrator zur multimodalen bedienung im fahrzeug. In: Gärtner, K.P. (ed.) Multimodale Interaktion im Bereich der Fahrzeug- und Prozessführung. DGLR-Bericht 200-02, München, Deutsche Gesellschaft für Luft- und Raumfahrttechnik d.V (DGLR), pp. 299–307 (2000)

    Google Scholar 

  58. Neuss, R.: Usability Engineering als Ansatz zum Multimodalen Mensch-Maschine-Dialog. PhD thesis, Fakultät für Elektrotechnik und Informationstechnik, Technische Universität München (2001)

    Google Scholar 

  59. Salmen, A., Großmann, P., Hitzenberger, L., Creutzburg, U.: Dialog systems in traffic environment. In: Proceedings of ESCA: Tutorial and Research Workshop on Interactive Dialogue in Multi-Modal Systems, Kloster Irsee (1999)

    Google Scholar 

  60. Salmen, A.: Multimodale Menüausgabe im Fahrzeug. PhD thesis, Lehrstuhl für Informationswissenschaft, Philosophische Fakultät IV, Universität Regensburg (2002)

    Google Scholar 

  61. Becker, T., Blaylock, N., Gerstenberger, C., Korthauer, A., Perera, N., Pitz, M., Poller, P., Schehl, J., Steffens, F., Stegmann, R.: D5.3: In-car showcase based on talk libraries. Deliverable 5.3, Universität des Saarlandes (2007)

    Google Scholar 

  62. Mayer, R.E., Moreno, R.: split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology 90(2), 312–320 (1998)

    Article  Google Scholar 

  63. Cohen, P.R., McGee, D., Clow, J.: The efficiency of multimodal interaction for a map-based task. In: Proceedings of the Sixth Conference on Applied Natural Language Processing, pp. 331–338. Morgan Kaufmann Publishers Inc., San Francisco (2000), http://www.aclweb.org/anthology-new/A/A00/A00-1046.pdf (checked: June 20, 2008)

  64. Saenko, K., Darrell, T., Glass, J.R.: Articulatory features for robust visual speech recognition. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 152–158. ACM Press, New York (2004)

    Google Scholar 

  65. Yang, J., Zhu, X., Gross, R., Kominek, J., Pan, Y., Waibel, A.: Multimodal people id for a multimedia meeting browser. In: MULTIMEDIA 1999: Proceedings of the Seventh ACM International Conference on Multimedia (Part 1), pp. 159–168. ACM Press, New York (1999)

    Google Scholar 

  66. Hazen, T.J., Weinstein, E., Park, A.: Towards robust person recognition on handheld devices using face and speaker identification technologies. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 289–292. ACM Press, New York (2003)

    Google Scholar 

  67. Selcon, S.J., Taylor, R.M.: Integrating multiple information sources: Using redundancy in the design of warnings. Ergonomics 38(11), 2362–2370 (1995)

    Article  Google Scholar 

  68. Sumby, W.H., Pollack, I.: Visual contribution to speech intelligibility in noise. Journal of the Acoustical Society of America 26, 212–215 (1954)

    Article  Google Scholar 

  69. Neely, K.K.: Effect of visual factors on the intelligibility of speech. Journal of the Acoustical Society of America 28, 1275–1277 (1956)

    Article  Google Scholar 

  70. Binnie, C.A., Montgomery, A.A., Jackson, P.L.: Auditory and visual contributions to the perception of consonants. Journal of Speech & Hearing Research 17, 619–630 (1974)

    Google Scholar 

  71. Erber, N.P.: Interaction of audition and vision in the recognition of oral speech stimuli. Journal of Speech & Hearing Research 12, 423–425 (1969)

    Google Scholar 

  72. Erber, N.P.: Auditory-visual perception of speech. Journal of Speech & Hearing Disorders 40, 481–492 (1975)

    Google Scholar 

  73. André, E., Muller, A.J., Rist, T.: The ppp persona: A multipurpose animated presentation agent. In: et al (ed.): Advanced Visual Interfaces, pp. 245–247. ACM Press (1996)

    Google Scholar 

  74. Kopp, S., Tepper, P., Cassell, J.: Towards integrated microplanning of language and iconic gesture for multimodal output. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 97–104. ACM Press, New York (2004)

    Google Scholar 

  75. Foster, M.E., White, M., Setzer, A., Catizone, R.: Multimodal generation in the comic dialogue system. In: ACL 2005: Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions, pp. 45–48. Association for Computational Linguistics, Morristown (2005)

    Google Scholar 

  76. Wahlster, W., Reithinger, N., Blocher, A.: Smartkom: Towards multimodal dialogues with anthropomorphic interface agents. In: Wolf, G., Klein, G. (eds.) Proceedings of International Status Conference: Lead Projects Human-Computer-Interaction, Saarbrücken, Projektträger des BMBF für Informationstechnik: Deutsches Zentrum für Luft- und Raumfahrttechnik (DLR) e.V., pp. 23–32 (2001)

    Google Scholar 

  77. Bernsen, N.O.: What is natural interactivity. In: Dybkjær (ed.) Proceedings of the Workshop From Spoken Dialogue to Full Natural Interactive Dialogue. Theory, Empirical Analysis and Evaluation, pp. 34–37. European Language Resources Association, Athen (2000)

    Google Scholar 

  78. Picard, R.W.: Perceptual user interfaces: affective perception. Commun. ACM 43(3), 50–51 (2000)

    Google Scholar 

  79. Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)

    Google Scholar 

  80. Brave, S., Nass, C.: Emotion in human-computer interaction. In: Jacko, J.A., Sears, A. (eds.) The Human-Computer Interaction Handbook. Fundamentals, Evolving Technologies, and Emerging Applications, pp. 81–96. Lawrence Erlbaum Assoc., Mahwah (2003)

    Google Scholar 

  81. Pantic, M., Sebe, N., Cohn, J.F., Huang, T.: Affective multimodal human-computer interaction. In: MULTIMEDIA 2005: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 669–676. ACM Press, New York (2005)

    Google Scholar 

  82. Ambady, N., Rosenthal, R.: Thin slices of behavior as predictors of interpersonal consequences: A meta-analysis. Psychological Bulletin 2, 256–274 (1992)

    Article  Google Scholar 

  83. Lisetti, C.L., Nasoz, F.: Maui: a multimodal affective user interface. In: MULTIMEDIA 2002: Proceedings of the Tenth ACM International Conference on Multimedia, pp. 161–170. ACM Press, New York (2002)

    Google Scholar 

  84. Essa, I., Darrell, T., Pentland, A.: Tracking facial motion. In: Proceedings of the IEEE Workshop on Nonrigid and Articulate Motion (1994)

    Google Scholar 

  85. Black, M., Yakoob, Y.: Recognizing faces showing expressions. In: Proceedings of the International Workshop on Automatic Face and Gesture Recognition. IEEE Press (1995)

    Google Scholar 

  86. Black, M., Yacoob, Y.: Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image motion. In: Proceedings of the International Conference on Computer Vision, pp. 374–381 (1995)

    Google Scholar 

  87. Terzopoulos, D., Waters, K.: Analysis and synthesis of facial images using physical and anatomical models. In: Proceedings of the International Conference on Computer Vision, pp. 727–732 (1990)

    Google Scholar 

  88. Kearney, G., McKenzie, S.: Machine interpretation of emotion: Design of a memory-based expert system for interpreting facial expressions in terms of signaled emotions. Cognitive Science 17 (1993)

    Google Scholar 

  89. Murray, I., Arnott, J.: Toward the simulation of emotion in synthetic speech: A review of the literature on human vocal emotion. Journal Acostical Society of America 93(2), 1097–1108 (1993)

    Google Scholar 

  90. Kompe, R.: Prosody in Speech Understanding Systems. LNCS, vol. 1307. Springer, Heidelberg (1997)

    Book  Google Scholar 

  91. Batliner, A., Buckow, A., Niemann, H., Nöth, E., Warnke, V.: The prosody module. In: Wahlster, W. (ed.) Verbmobil: Foundations of Speech-to-Speech Translations, pp. 106–121. Springer, Berlin (2000)

    Chapter  Google Scholar 

  92. Picard, R.: Affective Computing. MIT Press, Cambridge (1997)

    Google Scholar 

  93. Healey, J., Picard, R.: Smartcar: Detecting driver stress. In: Proceedings of ICPR 2000, Barcelona, Spanien (2000)

    Google Scholar 

  94. Ark, W., Dryer, D., Lu, D.: The emotion mouse. In: Bullinger, H.J., Ziegler, J. (eds.) Human-Computer Interaction: Ergonomics and User Interfaces, pp. 818–823. Lawrence Erlbaum Assoc. (1999)

    Google Scholar 

  95. Crosby, M.E., Auernheimer, B., Aschwanden, C., Ikehara, C.: Physiological data feedback for application in distance education. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive user Interfaces, pp. 1–5. ACM Press, New York (2001)

    Google Scholar 

  96. Qi, Y., Reynolds, C., Picard, R.W.: The bayes point machine for computer-user frustration detection via pressuremouse. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive User Interfaces, pp. 1–5. ACM Press, New York (2001)

    Google Scholar 

  97. Mentis, H.M.: Using touchpad pressure to detect negative affect. In: ICMI 2002: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p. 406. IEEE Computer Society, Washington, DC (2002)

    Google Scholar 

  98. O’Rorke, P., Ortony, A.: Explaining emotions. Cognitive Science 18(2), 283–323 (1994)

    Google Scholar 

  99. Holzapfel, H., Fuegen, C.: Integrating emotional cues into a framework for dialogue management. In: ICMI 2002: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, p. 141. IEEE Computer Society, Washington, DC (2002)

    Google Scholar 

  100. Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C.M., Kazemzadeh, A., Lee, S., Neumann, U., Narayanan, S.: Analysis of emotion recognition using facial expressions, speech and multimodal information. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 205–211. ACM Press, New York (2004)

    Google Scholar 

  101. Zeng, Z., Tu, J., Liu, M., Zhang, T., Rizzolo, N., Zhang, Z., Huang, T.S., Roth, D., Levinson, S.: Bimodal hci-related affect recognition. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 137–143. ACM Press, New York (2004)

    Google Scholar 

  102. Gunes, H., Piccardi, M., Jan, T.: Face and body gesture recognition for a vision-based multimodal analyzer. In: VIP 2005: Proceedings of the Pan-Sydney Area Workshop on Visual Information Processing, pp. 19–28. Australian Computer Society, Inc., Darlinghurst (2004)

    Google Scholar 

  103. Shi, R.P., Adelhardt, J., Zeissler, V., Batliner, A., Frank, C., Nöth, E., Niemann, H.: Using speech and gesture to explore user states in multimodal dialogue systems. Technical Report 36, Lehrstuhl für Mustererkennung, Institut für Informatik, Friedrich Alexander Universität Erlangen, Martensstraße 3, 91058 Erlangen (June 2003)

    Google Scholar 

  104. Rudmann, D.S., McConkie, G.W., Zheng, X.S.: Eyetracking in cognitive state detection for hci. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 159–163. ACM Press, New York (2003)

    Google Scholar 

  105. Lindner, C. (ed.): Avatare. Digitale Sprecher für Business und Marketing. Springer, Berlin (2003)

    Google Scholar 

  106. Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., Shapiro, S.C.: Natural language with integrated deictic and graphic gestures. In: HLT 1989: Proceedings of the Workshop on Speech and Natural Language, pp. 410–423. Association for Computational Linguistics, Morristown (1989)

    Google Scholar 

  107. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for distributed applications. In: MULTIMEDIA 1997: Proceedings of the Fifth ACM International Conference on Multimedia, pp. 31–40. ACM Press, New York (1997)

    Google Scholar 

  108. Malaka, R., Häußler, J., Aras, H.: Smartkom mobile: intelligent ubiquitous user interaction. In: IUI 2004: Proceedings of the 9th International Conference on Intelligent user Interface, pp. 310–312. ACM Press, New York (2004)

    Google Scholar 

  109. Johnston, M., Bangalore, S., Vasireddy, G., Stent, A., Ehlen, P., Walker, M., Whittaker, S., Maloor, P.: Match: an architecture for multimodal dialogue systems. In: ACL 2002: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 376–383. Association for Computational Linguistics, Morristown (2002)

    Google Scholar 

  110. Hastie, H.W., Johnston, M., Ehlen, P.: Context-sensitive help for multimodal dialogue. In: ICMI 2002: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces. IEEE Computer Society, Washington, DC (2002), http://www.research.att.com/~johnston/papers/hastieh_mmhelp.pdf (checked: June 20, 2008)

  111. Almeida, L., Amdal, I., Beires, N., Boualem, M., Boves, L., den Os, E., Filoche, P., Gomes, R., Knudsen, J.E., Kvale, K., Rugelbak, J., Tallec, C., Warakagoda, N.: Implementing and evaluating a multimodal and multilingual tourist guide. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, Copenhagen, Denmark (2002)

    Google Scholar 

  112. van Welie, M.: Task-based User Interface Design. PhD thesis, Dutch Graduate School for Information and Knowledge Systems, Vrije Universiteit Amsterdam (2001)

    Google Scholar 

  113. Gourdol, A., Nigay, L., Salber, D., Coutaz, J.: Two case studies of software architecture for multimodal interactive systems: Voicepaint and a voice-enabled graphical notebook. In: Larson, J., Unger, C. (eds.) Proceedings of IFIP TC3/WG2.7: Working Conference on Engineering for Human Computer Interaction, pp. 271–284. North Holland Publications (1992)

    Google Scholar 

  114. Nishimoto, T., Shida, N., Kobayashi, T., Shirai, K.: Improving human interface in drawing tool using speech. In: Proceedings of 4th IEEE International Workshop on Robot and Human Communication, ROMAN 1995, Tokyo, Japan, pp. 107–112 (1995)

    Google Scholar 

  115. Sedivy, J., Johnson, H.: Multimodal tool support for creative tasks in the visual arts. Knowledge-Based Systems 13(7-8), 441–450 (2000)

    Google Scholar 

  116. Sezgin, T.M., Stahovich, T., Davis, R.: Sketch based interfaces: early processing for sketch understanding. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive user Interfaces, pp. 1–8. ACM Press, New York (2001)

    Google Scholar 

  117. Forbus, K.D., Ferguson, R.W., Usher, J.M.: Towards a computational model of sketching. In: IUI 2001: Proceedings of the 6th International Conference on Intelligent user Interfaces, pp. 77–83. ACM Press, New York (2001)

    Google Scholar 

  118. Poirier, F., Julia, L., Rossignol, S., Faure, C.: Tapage: édition de tableaux sur ordinateur à stylo vers une désignation naturelle. In: Proc. IHM 1993 (1993)

    Google Scholar 

  119. Milota, A.D.: Modality fusion for graphic design applications. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 167–174. ACM Press, New York (2004)

    Google Scholar 

  120. Microsoft: Mipad: Speech powered prototype to simplify communication between users and handheld devices, http://www.microsoft.com/presspass/features/2000/05-22mipad.asp (checked: June 20, 2008) Microsoft

  121. Microsoft: Your pad or mipad, http://research.microsoft.com/srg/mipad.aspx (checked: June 20, 2008) Microsoft

  122. Huang, X., Acero, A., Chelba, C., Deng, L., Duchene, D., Goodman, J., Hon, H., Jacoby, D., Jiang, L., Loynd, R., Mahajan, M., Mau, P., Meredith, S., Mughal, S., Neto, S., Plumpe, M., Wang, K., Wang, Y.: Mipad: A next generation pda prototype. In: ICSLP, Peking (2000), http://research.microsoft.com/srg/papers/2000-xdh-icslp.pdf (checked: June 20, 2008)

  123. Comerford, L., Frank, D., Gopalakrishnan, P., Gopinath, R., Sedivy, J.: The ibm personal speech assistant. In: Proc. of IEEE ICASSP 2001, DARPA, pp. 319–321 (2001)

    Google Scholar 

  124. Aslan, I., Xu, F., Uszkoreit, H., Krüger, A., Steffen, J.: COMPASS2008: Multimodal, multilingual and crosslingual interaction for mobile tourist guide applications. In: Maybury, M., Stock, O., Wahlster, W. (eds.) INTETAIN 2005. LNCS (LNAI), vol. 3814, pp. 3–12. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  125. IBM: Developing X+V Applications Using the Multimodal Toolkit and Browser (October 2002)

    Google Scholar 

  126. IBM Pervasive Computing: Developing Multimodal Applications using XHTML+Voice (January 2003)

    Google Scholar 

  127. Marx, M., Schmandt, C.: Putting people first: specifying proper names in speech interfaces. In: UIST 1994: Proceedings of the 7th Annual ACM Symposium on User Interface Software and Technology, pp. 29–37. ACM Press, New York (1994)

    Google Scholar 

  128. Gould, J.D., Conti, J., Hovanyecz, T.: Composing letters with a simulated listening typewriter. Commun. ACM 26(4), 295–308 (1983)

    Google Scholar 

  129. Womser-Hacker, C.: Statistical experiments on computer talk. In: First International Comference on Quantitative Linguistics, pp. 251–263. Kluwer, Dordrecht (1993)

    Google Scholar 

  130. Hitzenberger, L., Womser-Hacker, C.: Experimentelle untersuchungen zu multimodalen natürlichsprachlichen dialogen in der mensch-computer-interaktion. SDV – Sprache und Datenverarbeitung 19(1), 51–61 (1995)

    Google Scholar 

  131. Coutaz, J., Salber, D., Carraux, E., Portolan, N.: Neimo, a multiworkstation usability lab for observing and analyzing multimodal interaction. In: CHI 1996: Conference Companion on Human Factors in Computing Systems, pp. 402–403. ACM Press, New York (1996)

    Google Scholar 

  132. Grasso, M.A., Ebert, D.S., Finin, T.W.: The integrality of speech in multimodal interfaces. ACM Trans. Comput.-Hum. Interact. 5(4), 303–325 (1998)

    Google Scholar 

  133. Oviatt, S., Cohen, P., Wu, L., Vergo, J., Duncan, L., Suhm, B., Bers, J., Holzman, T., Winograd, T., Landay, J., Larson, J., Ferro, D.: Designing the user interface for multimodal speech and pen-based gesture applications: State-of-the-art systems and future research directions. Human Computer Interaction 15(4), 263–322 (2000)

    Google Scholar 

  134. Treoware: Voicelauncher, http://treoware.com/voicelauncher.html (checked: June 20, 2008) Treoware

  135. CallMagazin: Hps erstes smartphone: Der ipaq 514 hört aufs wort. CallMagazin (February 16, 2007), http://www.call-magazin.de/handy-mobilfunk/handy-mobilfunk-nachrichten/hpserstes-smartphone-der-ipaq-514-hoert-aufs-wort_20628.html (checked: June 20, 2008)

  136. Microsoft: Microsoft voice command, http://www.microsoft.com/windowsmobile/voicecommand/features.mspx (checked: June 20, 2008) Microsoft

  137. Grasso, M.A., Finin, T.W.: Task integration in multimodal speech recognition environments. Crossroads 3(3), 19–22 (1997)

    Google Scholar 

  138. Oviatt, S., Cohen, P.R., Wang, M.Q.: Toward interface design for human language technology: modality and structure as determinants of linguistic complexity. Speech Commun. 15(3-4), 283–300 (1994)

    Google Scholar 

  139. IBM: Multimodal Application Design Issues (December 2003)

    Google Scholar 

  140. IBM: XHTML+Voice Programmer’s Guide. Version 1.0 edn. (February 2004)

    Google Scholar 

  141. Schnelle, D., Lyardet, F., Wei, T.: Audio Navigation Patterns. In: Proceedings of EuroPLoP 2005, pp. 237–260 (July 2005)

    Google Scholar 

  142. Schnelle, D., Lyardet, F.: Voice User Interface Design Patterns. In: Proceedings of 11th European Conference on Pattern Languages of Programs, EuroPlop 2006 (2006)

    Google Scholar 

  143. Ren, X., Zhang, G., Dai, G.: An experimental study of input modes for multimodal human-computer interaction. In: Tan, T., Shi, Y., Gao, W. (eds.) ICMI 2000. LNCS, vol. 1948, pp. 49–56. Springer, Heidelberg (2000), http://www.springerlink.com/content/wj970gnlyqam67du/fulltext.pdf (checked: June 20, 2008)

    Chapter  Google Scholar 

  144. van Welie, M.: Gui design patterns (2003), http://www.welie.com/patterns/gui/ (checked: May 16, 2007)

  145. Oviatt, S.L.: Multimodal interfaces. In: Jacko, J., Sears, A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 286–304. Lawrence Erlbaum Assoc., Mahwah (2003)

    Google Scholar 

  146. Cohen, P.R., Dalrymple, M., Moran, D.B., Pereira, F.C., Sullivan, J.W.: Synergistic use of direct manipulation and natural language. SIGCHI Bull. 20(SI), 227–233 (1989)

    Google Scholar 

  147. Kobsa, A., Allgayer, J., Reddig, C., Reithinger, N., Schmauks, D., Harbusch, K., Wahlster, W.: Combining deictic gestures and natural language for referent identification. In: Proceedings of the 11th Coference on Computational Linguistics, pp. 356–361. Association for Computational Linguistics, Morristown (1986)

    Google Scholar 

  148. Wahlster, W.: User and discourse models for multimodal communication. In: Sullivan, J.W., Tyler, S.W. (eds.) Intelligent User Interfaces, pp. 45–67. ACM Press (1991)

    Google Scholar 

  149. Shimazu, H., Arita, S., Takashima, Y.: Multi-modal definite clause grammar. In: COLING 1994, pp. 832–836 (1994)

    Google Scholar 

  150. Shimazu, H., Takashima, Y.: Multi-modal-method: A design method for building multi-modal systems. In: COLING 1996, pp. 925–930 (1996)

    Google Scholar 

  151. Bui, T., Rajman, M.: Rapid dialogue prototyping methodology. Technical Report 200401 IC/2004/01, Swiss Federal Institute of Technology (EPFL), Lausanne (January 2004)

    Google Scholar 

  152. Rajman, M., Bui, T., Rajman, A., Seydoux, F., Quarteroni, S.: Assessing the usability of a dialogue management system designed in the framework of a rapid dialogue prototyping methodology. In: Acta Acustica united with Acustica 2004 (2004)

    Google Scholar 

  153. Vo, M.T.: A Framework and Toolkit for the Construction of Multimodal Learning Interfaces. PhD thesis, School of Computer Science, Computer Science Department, Carnegie Mellon University (1998)

    Google Scholar 

  154. Cohen, P.R.: The role of natural language in a multimodal interface. In: UIST 1992: Proceedings of the 5th Annual ACM Symposium on User Interface Software and Technology, pp. 143–149. ACM Press, New York (1992)

    Google Scholar 

  155. Huls, C., Bos, E.: Studies into full integration of language and action. In: Proceedings of the International Conference on Cooperative Multimiodal Communication (CMC 1995), Eindhoven, pp. 161–174 (1995)

    Google Scholar 

  156. Bolt, R.A.:,Put-that-there“: Voice and gesture at the graphics interface. In: SIGGRAPH 1980: Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, pp. 262–270. ACM Press, New York (1980)

    Google Scholar 

  157. Bekiaris, E., Machate, J., Burmester, M.: Towards an intelligent multimodal and multimedia user interface providing a new dimension of natural hmi in the teleoperation of all home appliances by e&d users. In: Proceedings of Interfaces 1997, Montpellier, pp. 226–229 (1997)

    Google Scholar 

  158. Siroux, J., Guyomard, M., Multon, F., Remondeau, C.: Modeling and processing of oral and tactile activities in the georal system. In: Bunt, H., Beun, R.-J., Borghuis, T. (eds.) CMC 1995. LNCS (LNAI), vol. 1374, pp. 101–110. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  159. Cheyer, A.: Mviews: Multimodal tools for the video analyst. In: International Conference on Intelligent User Interfaces (IUI 1998), pp. 55–62. ACM Press, New York (1998), http://www.adam.cheyer.com/papers/iui98.pdf (checked: June 20, 2008)

  160. Portele, T., Goronzy, S., Emele, M., Kellner, A., Torge, S., te Vrugt, J.: Smartkom-home – an advanced multi-modal interface to home entertainment. In: EUROSPEECH 2003, pp. 1897–1900 (2003)

    Google Scholar 

  161. Reithinger, N., Alexandersson, J., Becker, T., Blocher, A., Engel, R., Löckelt, M., Müller, J., Pfleger, N., Poller, P., Streit, M., Tschernomas, V.: Smartkom: adaptive and flexible multimodal access to multiple applications. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 101–108. ACM Press, New York (2003)

    Google Scholar 

  162. Lisowska, A., Rajman, M., Bui, T.H.: archivus: A System for Accessing the Content of Recorded Multimodal Meetings. In: Bengio, S., Bourlard, H. (eds.) MLMI 2004. LNCS, vol. 3361, pp. 291–304. Springer, Heidelberg (2005)

    Chapter  Google Scholar 

  163. McGee, D.R., Cohen, P.R.: Creating tangible interfaces by augmenting physical objects with multimodal language. In: IUI 2001: Proceedings of the 6th International Conference on Intelligent user Interfaces, pp. 113–119. ACM Press, New York (2001)

    Google Scholar 

  164. Rauschert, I., Agrawal, P., Sharma, R., Fuhrmann, S., Brewer, I., MacEachren, A.: Designing a human-centered, multimodal gis interface to support emergency management. In: GIS 2002: Proceedings of the 10th ACM International Symposium on Advances in Geographic Information Systems, pp. 119–124. ACM Press, New York (2002)

    Google Scholar 

  165. Long, A.C., Landay, J.A., Rowe, L.A.: ”‘those look similar!’” issues in automating gesture design advice. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive user Interfaces, pp. 1–5. ACM Press, New York (2001)

    Google Scholar 

  166. Baudel, T., Beaudouin-Lafon, M.: Charade: remote control of objects using free-hand gestures. Commun. ACM 36(7), 28–35 (1993)

    Article  Google Scholar 

  167. Di Fiore, F., Vandoren, P., Van Reeth, F.: Multimodal interaction in a collaborative virtual brainstorming environment. In: Luo, Y. (ed.) CDVE 2004. LNCS, vol. 3190, pp. 47–60. Springer, Heidelberg (2004), http://research.edm.uhasselt.be/~fdifiore/research/CDVE2004/CDVE2004.pdf

    Chapter  Google Scholar 

  168. Ou, J., Fussell, S.R., Chen, X., Setlock, L.D., Yang, J.: Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 242–249. ACM Press, New York (2003)

    Google Scholar 

  169. Cheyer, A., Julia, L.: Multimodal maps: An agent-based approach. In: Bunt, H., Beun, R.-J., Borghuis, T. (eds.) CMC 1995. LNCS (LNAI), vol. 1374, pp. 111–121. Springer, Heidelberg (1998), http://www.springerlink.com/content/cafux7f8f2ymykkw/fulltext.pdf (checked: June 20, 2008)

    Chapter  Google Scholar 

  170. Summerfield, A.Q.: Use of visual information for phonetic perception. Phonetica 36, 314–331 (1979)

    Google Scholar 

  171. Oviatt, S.L.: Mutual disambiguation of recognition errors in a multimodal architecture. In: CHI 1999: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 576–583. ACM, New York (1999)

    Google Scholar 

  172. Oviatt, S.L.: Multimodal signal processing in naturalistic noisy environments. In: Yuan, B., Huang, T., Tang, X. (eds.) Proceedings of the 6th International Conference on SPoken Language Processing (ICSLP), vol. 2, pp. 696–699. Chinese Friendship Publishers, Peking (2000)

    Google Scholar 

  173. Oviatt, S.L.: Multimodal system processing in mobile environments. In: UIST 2000: Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology, pp. 21–30. ACM Press, New York (2000)

    Google Scholar 

  174. Oviatt, S.L.: Taming recognition errors with a multimodal interface. Commun. ACM 43(9), 45–51 (2000)

    Google Scholar 

  175. Kumar, S., Cohen, P.R., Coulston, R.: Multimodal interaction under exerted conditions in a natural field setting. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 227–234. ACM Press, New York (2004)

    Google Scholar 

  176. Wachsmuth, S.: Multi-modal Scene Understanding Using Probabilistic Models. PhD thesis, Technischen Fakultät, Universität Bielefeld (2001)

    Google Scholar 

  177. Yang, J., Stiefelhagen, R., Meier, U., Waibel, A.: Visual tracking for multimodal human computer interaction. In: CHI 1998: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 140–147. ACM Press / Addison-Wesley Publishing Co., New York (1998)

    Google Scholar 

  178. Jain, A.K.: Multimodal user interfaces: who’s the user? In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, p. 1. ACM Press, New York (2003)

    Google Scholar 

  179. Snelick, R., Indovina, M., Yen, J., Mink, A.: Multimodal biometrics: issues in design and testing. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 68–72. ACM Press, New York (2003)

    Google Scholar 

  180. Nasoz, F., Ozyer, O., Lisetti, C.L., Finkelstein, N.: Multimodal affective driver interfaces for future cars. In: MULTIMEDIA 2002: Proceedings of the Tenth ACM International Conference on Multimedia, pp. 319–322. ACM Press, New York (2002)

    Google Scholar 

  181. Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., Feiner, S.: Mutual disambiguation of 3d multimodal interaction in augmented and virtual reality. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 12–19. ACM Press, New York (2003)

    Google Scholar 

  182. Zhang, Q., Imamiya, A., Go, K., Mao, X.: Overriding errors in a speech and gaze multimodal architecture. In: IUI 2004: Proceedings of the 9th International Conference on Intelligent User Interface, pp. 346–348. ACM Press, New York (2004)

    Google Scholar 

  183. Tan, Y.K., Sherkat, N., Allen, T.: Error recovery in a blended style eye gaze and speech interface. In: ICMI 2003: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 196–202. ACM Press, New York (2003)

    Google Scholar 

  184. Tanaka, K.: A robust selection system using real-time multi-modal user-agent interactions. In: IUI 1999: Proceedings of the 4th International Conference on Intelligent User Interfaces, pp. 105–108. ACM Press, New York (1999)

    Google Scholar 

  185. Campana, E., Baldridge, J., Dowding, J., Hockey, B.A., Remington, R.W., Stone, L.S.: Using eye movements to determine referents in a spoken dialogue system. In: PUI 2001: Proceedings of the 2001 Workshop on Perceptive user Interfaces, pp. 1–5. ACM Press, New York (2001)

    Google Scholar 

  186. Patterson, E., Gowdy, J.: An audio-visual approach to simultaneous-speaker speech recognition. In: Proceedings of ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing, vol. 5, pp. 780–783 (2003)

    Google Scholar 

  187. Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3d pointing gestures. In: ICMI 2004: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 175–182. ACM Press, New York (2004)

    Google Scholar 

  188. Chai, J.Y., Qu, S.: A salience driven approach to robust input interpretation in multimodal conversational systems. In: HLT 2005: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 217–224. Association for Computational Linguistics, Morristown (2005)

    Google Scholar 

  189. Trabelsi, Z., Cha, S.H., Desai, D., Tappert, C.: A voice and ink xml multimodal architecture for mobile e-commerce systems. In: WMC 2002: Proceedings of the 2nd International Workshop on Mobile Commerce, pp. 100–104. ACM Press, New York (2002)

    Google Scholar 

  190. Suhm, B., Myers, B., Waibel, A.: Multimodal error correction for speech user interfaces. ACM Trans. Comput.-Hum. Interact. 8(1), 60–98 (2001)

    Google Scholar 

  191. Ainsworth, W.A., Pratt, S.R.: Feedback strategies for error correction in speech recognition systems. Int. J. Man-Mach. Stud. 36(6), 833–842 (1992)

    Article  Google Scholar 

  192. Murray, A.C., Frankish, C.R., Jones, D.M.: Data-entry by voice: Facilitating correction of misrecognitions. In: Baber, C., Noyes, J. (eds.) Interactive Speech Technology: Human Factors issues in the Application of Speech Input/Output to Computers, pp. 137–144. Taylor and Francis, Bristol (1993)

    Google Scholar 

  193. Lai, J., Vergo, J.: Medspeak: Report creation with continuous speech recognition. In: Proceedings of the Conference on Human Factors in Computing (CHI 1997), pp. 431–438. ACM Press (1997)

    Google Scholar 

  194. Papineni, K.A., Roukos, S., Ward, R.T.: Feature-based language understanding. In: Proceedings of the 5th European Conference on Speech Communication and Technology, vol. 3, pp. 1435–1438. European Speech Communication Association, Rhodes (1997)

    Google Scholar 

  195. Vergo, J.: A statistical approach to multimodal natural language interaction. In: Proceedings of the AAAI 1998 Workshop on Representations for Multimodal Human-Computer Interaction, pp. 81–85. AAAI Press (1998)

    Google Scholar 

  196. Balentine, B.: Re-engineering the speech menu. In: Gardner-Bonneau, D. (ed.) Human Factors and Voice Interacive Systems, pp. 205–235. Kluwer Akademic Publishers, Norwell (1999)

    Google Scholar 

  197. Balentine, B., Morgan, D.P.: How to Build a Speech Recognition Application. A Style Guide for Telephony Dialogues. EIG Press (2001)

    Google Scholar 

  198. Ibrahim, A., Johansson, P.: Multimodal dialogue systems: A case study for interactive tv. In: Proceedings of 7th ERCIM Workshop on User Interfaces for All, Chantilly, France, pp. 209–218 (2002)

    Google Scholar 

  199. Oviatt, S., van Gent, R.: Error reslution during multimodal human-computer interaction. In: Proc. of the International Conference on Spoken Language Processing, vol. 2, pp. 204–207 (1996)

    Google Scholar 

  200. Oviatt, S., Bernard, J., Levow, G.: Linguistic adaptation during error resolution with spoken and multimodal systems. Language and Speech (special issue on Prosody and Conversation) 41(3-4), 415–438 (1999)

    Google Scholar 

  201. Pieraccini, R., Dayanidhi, K., Bloom, J., Dahan, J.G., Phillips, M., Goodman, B.R., Prasad, K.V.: Multimodal conversational systems for automobiles. Commun. ACM 47(1), 47–49 (2004)

    Google Scholar 

  202. McCaffery, F., McTear, M.F., Murphy, M.: A multimedia interface for circuit board assembly. In: Bunt, H., Beun, R.-J., Borghuis, T. (eds.) CMC 1995. LNCS (LNAI), vol. 1374, pp. 213–230. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  203. Yankelovich, N.: How do users know what to say? Interactions 3(6), 32–43 (1996)

    Google Scholar 

  204. Wasinger, R.: Multimodal Interaction with Mobile Devices: Fusing a Broad Spectrum of Modality Combinations. PhD thesis, Naturwissenschaftlich-Technische Fakultät I der Universität des Saarlandes, Saarbrücken (2006)

    Google Scholar 

  205. Breedvelt-Schouten, I., Paternò, F., Severijns, C.A.: Reusable structures in task models. In: Harrison, H., Torres, J. (eds.) Design, Specification and Verification of Interactive Systems, pp. 115–238. Springer, New York (1997)

    Google Scholar 

  206. Sinnig, D.: The complicity of patterns and model-based engineering. Master thesis, Department of Computer Science. Concordia University, Montreal (2004)

    Google Scholar 

  207. Stimmel, C.: Hold me, thrill me, kiss me, kill me: Patterns for developing effective concept prototypes. In: Manolescu, D., Wolf, B. (eds.) Proceedings Pattern Languages of Programs, Monticello (1999)

    Google Scholar 

  208. Mahemoff, M.J., Johnston, L.J.: The planet pattern language for software internationalisation. In: Manolescu, D., Wolf, B. (eds.) Proceedings of Pattern Languages of Programs (1999), http://jerry.cs.uiuc.edu/~plop/plop99/proceedings/Mahemoff/planet.pdf (checked: June 20, 2008)

  209. Duyne, D.K.V., Landay, J., Hong, J.I.: The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience. Addison-Wesley Longman Publishing Co., Inc., Boston (2002)

    Google Scholar 

  210. Sutcliffe, A., Dimitrova, M.: Patterns, claims and multimedia. In: Sasse, M.A., Johnson, C. (eds.) Proc. Human-Computer Interaction: Interact 1999. IFIP, pp. 329–335. IOS Press, Amsterdam (1999)

    Google Scholar 

  211. Cybulski, J., Linden, T.: Composing multimedia artifacts for reuse. In: Harrison, N., Foote, B., Rohnert, H. (eds.) Pattern Languages of Program Design 4, pp. 461–488. Addison-Wesley Longman (2000)

    Google Scholar 

  212. Wolff, C.: Media design patterns. In: Womser-Hacker, C., Wolff, C. (eds.) Designing Information Systems, Konstanz, UVK, pp. 209–217 (2005)

    Google Scholar 

  213. Chung, E.S., Hong, J.I., Lin, J., Prabaker, M.K., Landay, J.A., Liu, A.L.: Development and evaluation of emerging design patterns for ubiquitous computing. In: DIS 2004: Proceedings of the 2004 Conference on Designing Interactive Systems, pp. 233–242. ACM Press, New York (2004)

    Google Scholar 

  214. Kunert, T.: User-Centered Interaction Design Patterns for Interactive Digital Television Applications. Springer (2009)

    Google Scholar 

  215. Ratzka, A.: A wizard-of-oz setting for multimodal interaction. an approach to user-based elicitation of design patterns. In: Osswald, A., Stempfhuber, M., Wolff, C. (eds.) Open Innovation. Proc. 10th International Symposium for Information Science, pp. 159–170. Universitätsverlag Konstanz, Köln (2007)

    Google Scholar 

  216. Ratzka, A.: Explorative studies on multimodal interaction in a pda- and desktop-based scenario. In: Proceedings of the International Conference on Multimodal Interfaces 2008. ACM Press (2008)

    Google Scholar 

  217. Ratzka, A.: Patternbasiertes User Interface Design für multimodale Interaktion. PhD thesis, Lehrstuhl für Informationswissenschaft, Universität Regensburg (2009/2010)

    Google Scholar 

  218. Riva, G. (ed.): Ambient Intelligence. The Evolution of Technology, Communication and Cognition Towards the Future of Human-Computer Interaction. Emerging Communication, vol. 6. IOS Press, Amsterdam (2005)

    Google Scholar 

  219. Weber, W. (ed.): Ambient Intelligence. Springer, Berlin (2005)

    Google Scholar 

  220. Nylander, S., Bylund, M., Waern, A.: Ubiquitous service access through adapted user interfaces on multiple devices. Personal Ubiquitous Comput. 9(3), 123–133 (2005)

    Google Scholar 

  221. Ishii, H., Ullmer, B.: Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of CHI 1997 Conference on Human Factors in Computing Systems, pp. 234–241. ACM Press (1997)

    Google Scholar 

  222. Cohen, P.R., McGee, D.R.: Tangible multimodal interfaces for safety-critical applications. Commun. ACM 47(1), 41–46 (2004)

    Google Scholar 

  223. Vince, J.A.: Virtual Reality Systems. Addison Wesley (1995)

    Google Scholar 

  224. Sharman, W.R., Craig, A.B.: Understanding Virtual Reality. Morgan Kaufman, San Francisco (1995)

    Google Scholar 

  225. Mayhew, D.J.: The Usability Engineering Lifecycle. Morgan Kaufmann, San Francisco (1999)

    Google Scholar 

  226. Sinnig, D., Gaffar, A., Reichart, D., Seffah, A., Forbrig, P.: Patterns in model-based engineering. In: CADUI, pp. 195–208 (2004)

    Google Scholar 

  227. Petrasch, R.: Model based user interface design: Model driven architecture und hci patterns. GI Softwaretechnik-Trends. Mitteilungen der Gesellschaft für Informatik 27(3), 5–10 (2007), http://pi.informatik.uni-siegen.de/stt/27_3/03_Technische_Beitraege/MDA_HCI_Patterns_Petrasch_Short.pdf (checked: June 27, 2008)

  228. Limbourg, Q., Vanderdonckt, J., Souchon, N.: The task-dialog and task-presentation mapping problem: Some preliminary results. In: Paternó, F. (ed.) DSV-IS 2000. LNCS, vol. 1946, pp. 227–246. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  229. Gulliksen, J., Göransson, B., Boivie, I., Persson, J., Blomkvist, S., Cajander, Å.: Key principles for user-centred systems design. In: Seffah, A., Gulliksen, J., Desmarais, M.C. (eds.) Human-Centered Software Engineering: Integrating Usability in the Software Development Lifecycle, pp. 17–36. Springer, Dordrecht (2005)

    Google Scholar 

  230. Trætteberg, H.: Model-based User Interface Design. PhD thesis, Department of Computer and Information Sciences, Norwegian University of Science and Technology (2002)

    Google Scholar 

  231. Ratzka, A.: Design patterns in the context of multi-modal interaction. In: Proceedings of the 6th Nordic Conference on Pattern Languages of Programs, VikingPLoP 2007 (2008)

    Google Scholar 

  232. Ratzka, A.: Patterns for robust and accessible multimodal interaction. In: Proceedings of the 13th European Conference on Pattern Languages of Programs, EuroPLoP 2008 (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. I:IMSK, University of Regensburg, Germany

    Andreas Ratzka

Authors
  1. Andreas Ratzka
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Engineering and Computer Science, Victoria University of Wellington, Cottn Building, Gate 6, Kelburn Parade, 6140, Wellington, New Zealand

    James Noble

  2. Siebel Center for Computer Science, 201 North Goodwin Avenue, 61801, Urbana, IL, USA

    Ralph Johnson

  3. Faculty of Computer Science, Research Group Software Architecture, University of Vienna, Währinger Straße 29, 1090, Vienna, Austria

    Uwe Zdun

  4. Department of Computer Science, University of Northern Iowa, 50613, Cedar Falls, IA, USA

    Eugene Wallingford

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Ratzka, A. (2013). User Interface Patterns for Multimodal Interaction. In: Noble, J., Johnson, R., Zdun, U., Wallingford, E. (eds) Transactions on Pattern Languages of Programming III. Lecture Notes in Computer Science, vol 7840. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-38676-3_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-38676-3_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-38675-6

  • Online ISBN: 978-3-642-38676-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation