Building Multimodal Interfaces Out of Executable, Model-Based Interactors and Mappings

  • Sebastian Feuerstack
  • Ednaldo Pizzolato
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6761)


Future interaction will be embedded into smart environments offering the user to choose and to combine a heterogeneous set of interaction devices and modalities based on his preferences realizing an ubiquitous and multimodal access. We propose a model-based runtime environment (the MINT Framework) that describes multimodal interaction by interactors and multimodal mappings. The interactors are modeled by using state machines and describe user interface elements for various modalities. Mappings combine these interactors with interaction devices and support the definition of multimodal relations. We prove our implementation by modeling a multimodal navigation supporting pointing and hand gestures. We additionally present the flexibility of our approach that supports modeling of common interaction paradigms such as drag-and-drop as well.


State Machine Eclipse Modeling Framework Smart Environment Multimodal Interaction Interaction Device 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F.: A transformational approach for multimodal web user interfaces based on usixml. In: ICMI 2005: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 259–266. ACM Press, New York (2005)Google Scholar
  2. 2.
    Blumendorf, M., Lehmann, G., Feuerstack, S., Albayrak, Ş.: Executable models for human-computer interaction. In: Graham, T.C.N. (ed.) DSV-IS 2008. LNCS, vol. 5136, pp. 238–251. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Demeure, A., Calvary, G., Coninx, K.: COMET(s), A software architecture style and an interactors toolkit for plastic user interfaces. In: Graham, T.C.N., Palanque, P. (eds.) DSV-IS 2008. LNCS, vol. 5136, pp. 225–237. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  4. 4.
    Bernsen, N.O.: Multimodality theory. In: Tzovaras, D. (ed.) Multimodal User Interfaces, Signals and Communication Technology, pp. 5–29. Springer, Heidelberg (2008), doi:10.1007/978-3-540-78345-9_2CrossRefGoogle Scholar
  5. 5.
    Lawson, J.-Y.L., Al-Akkad, A.-A., Vanderdonckt, J., Macq, B.: An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components. In: EICS 2009: Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 245–254. ACM, New York (2009)Google Scholar
  6. 6.
    Bouchet, J., Nigay, L., Ganille, T.: The ICARE Component-Based Approach for Multimodal Input Interaction: Application to real-time military aircraft cockpits. In: Conference Proceedings of HCI International 2005, the 11th International Conference on Human-Computer Interaction, Las Vegas, Nevada, USA. Lawrence Erlbaum Associates, Mahwah (2005)Google Scholar
  7. 7.
    Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)CrossRefGoogle Scholar
  8. 8.
    Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., López-Jaquero, V.: USIXML: A language supporting multi-path development of user interfaces. In: Bastide, R., Palanque, P.A., Roth, J. (eds.) DSV-IS 2004 and EHCI 2004. LNCS, vol. 3425, pp. 200–220. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Duke, D., Faconti, G., Harrison, M., Paternó, F.: Unifying views of interactors. In: AVI 1994: Proceedings of the Workshop on Advanced Visual Interfaces, pp. 143–152. ACM, New York (1994) ISBN:0-89791-733-2Google Scholar
  10. 10.
    Trætteberg, H.: Dialog modelling with interactors and UML statecharts - A hybrid approach. In: Jorge, J.A., Jardim Nunes, N., Falcão e Cunha, J. (eds.) DSV-IS 2003. LNCS, vol. 2844, pp. 346–361. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  11. 11.
    Bomsdorf, B.: The webTaskModel approach to web process modelling. In: Winckler, M., Johnson, H. (eds.) TAMODIA 2007. LNCS, vol. 4849, pp. 240–253. Springer, Heidelberg (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Sebastian Feuerstack
    • 1
  • Ednaldo Pizzolato
    • 1
  1. 1.Departamento de ComputacãoUniversidade Federal de São CarlosBrazil

Personalised recommendations