Abstract
Future interaction will be embedded into smart environments offering the user to choose and to combine a heterogeneous set of interaction devices and modalities based on his preferences realizing an ubiquitous and multimodal access. We propose a model-based runtime environment (the MINT Framework) that describes multimodal interaction by interactors and multimodal mappings. The interactors are modeled by using state machines and describe user interface elements for various modalities. Mappings combine these interactors with interaction devices and support the definition of multimodal relations. We prove our implementation by modeling a multimodal navigation supporting pointing and hand gestures. We additionally present the flexibility of our approach that supports modeling of common interaction paradigms such as drag-and-drop as well.
Chapter PDF
References
Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F.: A transformational approach for multimodal web user interfaces based on usixml. In: ICMI 2005: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 259–266. ACM Press, New York (2005)
Blumendorf, M., Lehmann, G., Feuerstack, S., Albayrak, Ş.: Executable models for human-computer interaction. In: Graham, T.C.N. (ed.) DSV-IS 2008. LNCS, vol. 5136, pp. 238–251. Springer, Heidelberg (2008)
Demeure, A., Calvary, G., Coninx, K.: COMET(s), A software architecture style and an interactors toolkit for plastic user interfaces. In: Graham, T.C.N., Palanque, P. (eds.) DSV-IS 2008. LNCS, vol. 5136, pp. 225–237. Springer, Heidelberg (2008)
Bernsen, N.O.: Multimodality theory. In: Tzovaras, D. (ed.) Multimodal User Interfaces, Signals and Communication Technology, pp. 5–29. Springer, Heidelberg (2008), doi:10.1007/978-3-540-78345-9_2
Lawson, J.-Y.L., Al-Akkad, A.-A., Vanderdonckt, J., Macq, B.: An open source workbench for prototyping multimodal interactions based on off-the-shelf heterogeneous components. In: EICS 2009: Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems, pp. 245–254. ACM, New York (2009)
Bouchet, J., Nigay, L., Ganille, T.: The ICARE Component-Based Approach for Multimodal Input Interaction: Application to real-time military aircraft cockpits. In: Conference Proceedings of HCI International 2005, the 11th International Conference on Human-Computer Interaction, Las Vegas, Nevada, USA. Lawrence Erlbaum Associates, Mahwah (2005)
Calvary, G., Coutaz, J., Thevenin, D., Limbourg, Q., Bouillon, L., Vanderdonckt, J.: A unifying reference framework for multi-target user interfaces. Interacting with Computers 15(3), 289–308 (2003)
Limbourg, Q., Vanderdonckt, J., Michotte, B., Bouillon, L., López-Jaquero, V.: USIXML: A language supporting multi-path development of user interfaces. In: Bastide, R., Palanque, P.A., Roth, J. (eds.) DSV-IS 2004 and EHCI 2004. LNCS, vol. 3425, pp. 200–220. Springer, Heidelberg (2005)
Duke, D., Faconti, G., Harrison, M., Paternó, F.: Unifying views of interactors. In: AVI 1994: Proceedings of the Workshop on Advanced Visual Interfaces, pp. 143–152. ACM, New York (1994) ISBN:0-89791-733-2
Trætteberg, H.: Dialog modelling with interactors and UML statecharts - A hybrid approach. In: Jorge, J.A., Jardim Nunes, N., Falcão e Cunha, J. (eds.) DSV-IS 2003. LNCS, vol. 2844, pp. 346–361. Springer, Heidelberg (2003)
Bomsdorf, B.: The webTaskModel approach to web process modelling. In: Winckler, M., Johnson, H. (eds.) TAMODIA 2007. LNCS, vol. 4849, pp. 240–253. Springer, Heidelberg (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Feuerstack, S., Pizzolato, E. (2011). Building Multimodal Interfaces Out of Executable, Model-Based Interactors and Mappings. In: Jacko, J.A. (eds) Human-Computer Interaction. Design and Development Approaches. HCI 2011. Lecture Notes in Computer Science, vol 6761. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-21602-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-642-21602-2_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-21601-5
Online ISBN: 978-3-642-21602-2
eBook Packages: Computer ScienceComputer Science (R0)