Skip to main content
SpringerLink
Log in

Management of Multimodal User Interaction in Companion-Systems

  • Chapter
  • First Online:
Companion Technology

Part of the book series: Cognitive Technologies ((COGTECH))

  • 800 Accesses

Abstract

While interacting, human beings continuously adapt their way of communication to their surroundings and their communication partner. Although present context-aware ubiquitous systems gather a lot of information to maximize their functionality, they predominantly offer rather static ways to communicate. In order to fulfill the user’s communication needs and demands, ubiquitous sensors’ varied information could be used to dynamically adapt the user interface. Considering such an adaptive user interface management as a major and relevant component for a Companion-Technology, we also have to cope with emotional and dispositional user input as a source of implicit user requests and demands. In this chapter we demonstrate how multimodal fusion based on evidential reasoning and probabilistic fission with adaptive reasoning can act together to form a highly adaptive and model-driven interactive system component for multimodal interaction. The presented interaction management (IM) can handle uncertain or ambiguous data throughout the complete interaction cycle with a user. In addition, we present the IM’s architecture and its model-driven concept. Finally, we discuss its role within the framework of the other constituents of a Companion-Technology.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Simplified action cycle process based on [35]: form a goal → plan an action → execute an action → perceive feedback → evaluate the new state of the world → ….

References

  1. Ameri Ekhtiarabadi, A., Akan, B., Çürüklu, B., Asplund, L.: A general framework for incremental processing of multimodal inputs. In: Proceedings of the 13th International Conference on Multimodal Interface, ICMI ’11, pp. 225–228. ACM, New York (2011). doi:10.1145/2070481.2070521

    Google Scholar 

  2. Atrey, P., Hossain, M.A., El Saddik, A., Kankanhalli, M.: Multimodal fusion for multimedia analysis: a survey. Multimedia Systems 16, 345–379 (2010)

    Article  Google Scholar 

  3. Bertrand, G., Nothdurft, F., Honold, F., Schüssel, F.: CALIGRAPHI - Creation of Adaptive Dialogues Using a Graphical Interface. In: 2011 IEEE 35th Annual Computer Software and Applications Conference (COMPSAC), pp. 393–400. IEEE, New York (2011). doi:10.1109/COMPSAC.2011.58

    Google Scholar 

  4. Biundo, S., Wendemuth, A.: Companion-technology for cognitive technical systems. Künstl. Intell. (2016). doi:10.1007/s13218-015-0414-8

    Google Scholar 

  5. Blumendorf, M., Roscher, D., Albayrak, S.: Dynamic user interface distribution for flexible multimodal interaction. In: ICMI and the Workshop on Machine Learning for Multimodal Interaction. ACM, New York (2010). doi:10.1145/1891903.1891930

    Book  Google Scholar 

  6. Brandes, U., Eiglsperger, M., Herman, I., Himsolt, M., Marshall, M.: GraphML progress report: structural layer proposal. In: Mutzel, P., Jünger, M., Leipert, S. (eds.) Graph Drawing. Lecture Notes in Computer Science, vol. 2265, pp. 501–512. Springer, Berlin, Heidelberg (2002). doi:10.1007/3-540-45848-4_59

    Google Scholar 

  7. Calvary, G., Coutaz, J., Thevenin, D., Bouillon, L., Florins, M., Limbourg, Q., Souchon, N., Vanderdonckt, J., Marucci, L., Paternò, F., Santoro, C.: The Cameleon reference framework. Technical Report 1.1. CAMELEON Reference Framework Working Group (2002)

    Google Scholar 

  8. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: multimodal interaction for distributed applications. In: Proceedings of the Fifth ACM International Conference on Multimedia, MULTIMEDIA ’97, pp. 31–40. ACM, New York (1997). doi:10.1145/266180.266328

    Google Scholar 

  9. Costa, D., Duarte, C.: Adapting multimodal fission to user’s abilities. In: Proceedings of the 6th International Conference on Universal Access in Human-Computer Interaction: Design for All and EInclusion - Volume Part I, UAHCI’11, pp. 347–356. Springer, Berlin (2011)

    Google Scholar 

  10. Cutugno, F., Leano, V.A., Rinaldi, R., Mignini, G.: Multimodal framework for mobile interaction. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, AVI ’12, pp. 197–203. ACM, New York (2012). doi:10.1145/2254556.2254592

    Google Scholar 

  11. Dey, A.K., Abowd, G.D.: Towards a better understanding of context and context-awareness. In: HUC ’99: Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing, pp. 304–307. Springer, Berlin (1999)

    Google Scholar 

  12. Duarte, C., Carriço, L.: A conceptual framework for developing adaptive multimodal applications. In: IUI ’06: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 132–139. ACM, New York (2006). doi:10.1145/1111449.1111481

    Google Scholar 

  13. Dumas, B., Lalanne, D., Ingold, R.: Prototyping multimodal interfaces with the SMUIML modeling language. In: CHI 2008 Workshop on User Interface Description Languages for Next Generation User Interfaces, CHI 2008, Frienze, pp. 63–66 (2008)

    Google Scholar 

  14. Dumas, B., Lalanne, D., Guinard, D., Koenig, R., Ingold, R.: Strengths and weaknesses of software architectures for the rapid creation of tangible and multimodal interfaces. In: TEI ’08: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction, pp. 47–54. ACM, New York (2008). doi:10.1145/1347390.1347403

    Google Scholar 

  15. Dumas, B., Lalanne, D., Oviatt, S.: Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne, D., Kohlas, J. (eds.) Human Machine Interaction – Research Results of the MMI Program. Lecture Notes in Computer Science, vol. 5440/2009, chap. 1, pp. 3–26. Springer, Berlin, Heidelberg (2009). doi:10.1007/978-3-642-00437-7_1

    Google Scholar 

  16. Dumas, B., Signer, B., Lalanne, D.: Fusion in multimodal interactive systems: an HMM-based algorithm for user-induced adaptation. In: Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS ’12, pp. 15–24. ACM, New York (2012). doi:10.1145/2305484.2305490

    Google Scholar 

  17. Foster, M.E.: State of the art review: multimodal fission. Public Deliverable 6.1, University of Edinburgh (2002). COMIC Project

    Google Scholar 

  18. Geier, T., Reuter, S., Dietmayer, K., Biundo, S.: Goal-based person tracking using a first-order probabilistic model. In: Proceedings of the Ninth UAI Bayesian Modeling Applications Workshop (UAI-AW 2012) (2012)

    Google Scholar 

  19. Hina, M.D., Tadj, C., Ramdane-Cherif, A., Levy, N.: A Multi-Agent based Multimodal System Adaptive to the User’s Interaction Context. Multi-Agent Systems – Modeling, Interactions, Simulations and Case Studies, chap. 2, pp. 29–56. InTech (2011)

    Google Scholar 

  20. Holzapfel, H., Nickel, K., Stiefelhagen, R.: Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures. In: Proceedings of the 6th International Conference on Multimodal Interfaces, ICMI ’04, pp. 175–182. ACM, New York (2004). doi:10.1145/1027933.1027964

    Google Scholar 

  21. Honold, F., Schüssel, F., Weber, M.: Adaptive probabilistic fission for multimodal systems. In: Proceedings of the 24th Australian Computer-Human Interaction Conference, OzCHI ’12, pp. 222–231. ACM, New York (2012). doi:10.1145/2414536.2414575

    Google Scholar 

  22. Honold, F., Schüssel, F., Weber, M., Nothdurft, F., Bertrand, G., Minker, W.: Context models for adaptive dialogs and multimodal interaction. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 57–64. IEEE, New York (2013). doi:10.1109/IE.2013.54

    Google Scholar 

  23. Honold, F., Schüssel, F., Munding, M., Weber, M.: Tangible context modelling for rapid adaptive system testing. In: 2013 9th International Conference on Intelligent Environments (IE), pp. 278–281. IEEE, Athens (2013). doi:10.1109/IE.2013.9

    Google Scholar 

  24. Honold, F., Bercher, P., Richter, F., Nothdurft, F., Geier, T., Barth, R., Hörnle, T., Schüssel, F., Reuter, S., Rau, M., Bertrand, G., Seegebarth, B., Kurzok, P., Schattenberg, B., Minker, W., Weber, M., Biundo, S.: Companion-technology: towards user- and situation-adaptive functionality of technical systems. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 378–381. IEEE, New York (2014). doi:10.1109/IE.2014.60

    Google Scholar 

  25. Honold, F., Schüssel, F., Weber, M.: The automated interplay of multimodal fission and fusion in adaptive HCI. In: IE’14: Proceedings of the 10th International Conference on Intelligent Environments, pp. 170–177. IEEE, Shanghai (2014). doi:10.1109/IE.2014.32

    Google Scholar 

  26. Johnston, M., Bangalore, S.: Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11, 159–187 (2005). doi:10.1017/S1351324904003572

    Article  Google Scholar 

  27. Koons, D.B., Sparrell, C.J., Thorisson, K.R.: Integrating simultaneous input from speech, gaze, and hand gestures. In: Maybury, M.T. (ed.) Intelligent Multimedia Interfaces, chap. 11, pp. 257–276. American Association for AI, Menlo Park, CA (1993)

    Google Scholar 

  28. Krahnstoever, N., Kettebekov, S., Yeasin, M., Sharma, R.: A real-time framework for natural multimodal interaction with large screen displays. In: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, ICMI ’02, pp. 349–354. IEEE Computer Society, Washington, DC (2002). doi:10.1109/ICMI.2002.1167020

    Google Scholar 

  29. Lalanne, D., Nigay, L., Palanque, P., Robinson, P., Vanderdonckt, J., Ladry, J.F.: Fusion engines for multimodal input: a survey. In: Proceedings of the 2009 International Conference on Multimodal Interfaces, ICMI-MLMI ’09, pp. 153–160. ACM, New York (2009). doi:10.1145/1647314.1647343

    Google Scholar 

  30. LaViola, J.J. Jr., Buchanan, S., Pittman, C.: Multimodal Input for Perceptual User Interfaces, chap. 9, pp. 285–312. Wiley, New York (2014). doi:10.1002/9781118706237.ch9

    Google Scholar 

  31. Mansoux, B., Nigay, L., Troccaz, J.: Output multimodal interaction: the case of augmented surgery. In: Bryan-Kinns, N., Blanford, A., Curzon, P., Nigay, L. (eds.) People and Computers XX – Engage. BCS Conference Series, vol. 5, pp. 177–192. Springer/ACM, London/New York (2006). doi:10.1007/978-1-84628-664-3_14

    Google Scholar 

  32. Martin, J.C.: Tycoon: theoretical framework and software tools for multimodal interfaces. In: Lee, J. (ed.) Intelligence and Multimodality in Multimedia Interfaces. AAAI Press, Palo Alto, CA (1998)

    Google Scholar 

  33. Neal, J.G., Thielman, C.Y., Dobes, Z., Haller, S.M., Shapiro, S.C.: Natural language with integrated deictic and graphic gestures. In: Proceedings of the Workshop on Speech and Natural Language, HLT ’89, pp. 410–423. Association for Computational Linguistics, Stroudsburg, PA (1989). doi:10.3115/1075434.1075499

    Google Scholar 

  34. Nigay, L., Coutaz, J.: A generic platform for addressing the multimodal challenge. In: CHI ’95: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 98–105. ACM, New York (1995). doi:10.1145/223904.223917

    Google Scholar 

  35. Norman, D.A.: The Design of Everyday Things. iBasic Books, New York (2002)

    Google Scholar 

  36. Nothdurft, F., Honold, F., Zablotskaya, K., Diab, A., Minker, W.: Application of verbal intelligence in dialog systems for multimodal interaction. In: 2014 10th International Conference on Intelligent Environments (IE), pp. 361–364. IEEE, Shanghai (2014). doi:10.1109/IE.2014.59. (Short paper)

    Google Scholar 

  37. Oviatt, S.: Multimodal Interfaces, 2nd edn., chap. 21, pp. 413–432. CRC, Boca Raton (2007). doi:10.1201/9781410615862.ch21

    Google Scholar 

  38. Pfleger, N.: Context based multimodal fusion. In: ICMI ’04: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 265–272. ACM, New York (2004). doi:10.1145/1027933.1027977

    Google Scholar 

  39. Portillo, P.M., García, G.P., Carredano, G.A.: Multimodal fusion: a new hybrid strategy for dialogue systems. In: Proceedings of the 8th International Conference on Multimodal Interfaces, ICMI ’06, pp. 357–363. ACM, New York (2006). doi:10.1145/1180995.1181061

    Google Scholar 

  40. Reddy, B.S., Basir, O.A.: Concept-based evidential reasoning for multimodal fusion in human-computer interaction. Appl. Soft Comput. 10(2), 567–577 (2010). doi:10.1016/j.asoc.2009.08.026

    Article  Google Scholar 

  41. Roscher, D., Blumendorf, M., Albayrak, S.: A meta user interface to control multimodal interaction in smart environments. In: Proceedings of the 14th International Conference on Intelligent User Interfaces, IUI ’09, pp. 481–482. ACM, New York (2009). doi:10.1145/1502650.1502725

    Google Scholar 

  42. Rousseau, C., Bellik, Y., Vernier, F., Bazalgette, D.: A framework for the intelligent multimodal presentation of information. Signal Process. 86(12), 3696–3713 (2006). doi:10.1016/j.sigpro.2006.02.041

    Article  MATH  Google Scholar 

  43. Schüssel, F., Honold, F., Weber, M.: Influencing factors on multimodal interaction during selection tasks. J. Multimodal User Interfaces 7(4), 299–310 (2013). doi:10.1007/s12193-012-0117-5

    Article  Google Scholar 

  44. Schüssel, F., Honold, F., Weber, M.: Using the transferable belief model for multimodal input fusion in companion systems. In: Schwenker, F., Scherer, S., Morency, L.P. (eds.) Multimodal Pattern Recognition of Social Signals in Human-Computer-Interaction. Lecture Notes in Computer Science, vol. 7742, pp. 100–115. Springer, Berlin, Heidelberg (2013). doi:10.1007/978-3-642-37081-6_12

    Chapter  Google Scholar 

  45. Smets, P.: The combination of evidence in the transferable belief model. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 447–458 (1990). doi:10.1109/34.55104

    Article  Google Scholar 

  46. Smets, P.: Data fusion in the transferable belief model. In: Proceedings of the Third International Conference on Information Fusion. FUSION 2000, vol. 1, pp. PS21–PS33. IEEE, New York (2000). doi:10.1109/IFIC.2000.862713

    Google Scholar 

  47. Vanderdonckt, J., Limbourg, Q., Michotte, B., Bouillon, L., Trevisan, D., Florins, M.: USIXML: a user interface description language for specifying multimodal user interfaces. In: Proceedings of W3C Workshop on Multimodal Interaction WMI’2004, pp. 1–7 (2004)

    Google Scholar 

  48. Wahlster, W.: User and discourse models for multimodal communication. In: Sullivan, J.W., Tyler, S.W. (eds.) Intelligent User Interfaces, pp. 45–67. ACM, New York (1991). doi:10.1145/107215.128691

    Google Scholar 

  49. Wahlster, W. (ed.): SmartKom: Foundations of Multimodal Dialogue Systems. Springer, Berlin (2006). doi:10.1007/3-540-36678-4

    Google Scholar 

Download references

Acknowledgements

This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).

Author information

Authors and Affiliations

  1. Institute of Media Informatics, Ulm University, 89081, Ulm, Germany

    Felix Schüssel, Frank Honold & Michael Weber

  2. General Psychology, Ulm University, 89081, Ulm, Germany

    Nikola Bubalo & Anke Huckauf

Authors
  1. Felix Schüssel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frank Honold
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nikola Bubalo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anke Huckauf
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felix Schüssel .

Editor information

Editors and Affiliations

  1. Institute of Artificial Intelligence, Universität Ulm, Ulm, Germany

    Susanne Biundo

  2. Cognitive Systems Group, Institute for Information Technology and Communications (IIKT) and Center for Behavioral Brain Sciences (CBBS), Otto-von-Guericke Universität Magdeburg, Magdeburg, Germany

    Andreas Wendemuth

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Schüssel, F., Honold, F., Bubalo, N., Weber, M., Huckauf, A. (2017). Management of Multimodal User Interaction in Companion-Systems. In: Biundo, S., Wendemuth, A. (eds) Companion Technology. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-43665-4_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-43665-4_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-43664-7

  • Online ISBN: 978-3-319-43665-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation