Incorporating Tilt-Based Interaction in Multimodal User Interfaces for Mobile Devices

  • Jani Mäntyjärvi
  • Fabio Paternò
  • Carmen Santoro
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4385)


Emerging ubiquitous environments raise the need to support multiple interaction modalities in diverse types of devices. Designing multimodal interfaces for ubiquitous environments using development tools creates challenges since target platforms support different resources and interfaces. Model-based approaches have been recognized as useful for managing the increasing complexity consequent to the many available interaction platforms. However, they have usually focused on graphical and/or vocal modalities. This paper presents a solution for enabling the development of tilt-based hand gesture and graphical modalities for mobile devices in a multimodal user interface development tool. The challenges related to developing gesture-based applications for various types of devices involving mobile devices are discussed in detail. The possible solution presented is based on a logical description language for hand-gesture user interfaces. Such language allows us to obtain a user interface implementation on the target mobile platform. The solution is illustrated with an example application that can be accessed from both the desktop and mobile device supporting tilt-based gesture interaction.


Model-based design of user interfaces gestural interfaces for mobile devices tilt interfaces 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abrams, M., et al.: UIML: An Appliance-Independent XML User Interface Language. In: Proceedings of the 8th WWW conference (1994)Google Scholar
  2. 2.
    Baldonado, M., et al.: The Stanford Digital Library Metadata Architecture. Int. J. Digit. Libr. 1, 108–121 (1997)CrossRefGoogle Scholar
  3. 3.
    Bastide, R., et al.: A Model-Based Approach for Real-Time Embedded Multimodal Systems in Military Aircrafts. In: Proceedings ICMI 2004, pp. 243–250. ACM Press, New York (2004)CrossRefGoogle Scholar
  4. 4.
    Berti, S., et al.: The TERESA XML Language for the Description of Interactive Systems at Multiple Abstraction Levels. In: Workshop Organized at Advanced Visual Interfaces, pp. 103–110 (2004)Google Scholar
  5. 5.
    Bouchet, J., Nigay, L., Ganille, T.: ICARE software components for rapidly developing multimodal interfaces. In: Proceedings ICMI 2004, pp. 251–258. ACM Press, New York (2004)CrossRefGoogle Scholar
  6. 6.
    Calvary, G., et al.: A Unifying Reference Framework for Multi-Target User Interfaces. Interacting with Computers 15(3), 289–308 (2003)CrossRefGoogle Scholar
  7. 7.
    Coutaz, J., et al.: Four Easy Pieces for Assessing the Usability of Multmodal Interaction: the CARE properties. In: Proceedings INTERACT 1995, pp. 115–120 (1995)Google Scholar
  8. 8.
    Eslambolchilar, P., Murray-Smith, R.: Tilt-based Automatic Zooming and Scaling in Mo-bile Devices - a state-space implementation. In: Brewster, S.A., Dunlop, M.D. (eds.) Mobile HCI 2004. LNCS, vol. 3160, pp. 120–131. Springer, Heidelberg (2004)Google Scholar
  9. 9.
    Hinckley, K., et al.: Sensing Techniques for Mobile Interaction. CHI Letters (ACM UIST 2000 Symposium on User Interface Software & Technology) 2(2), 91–100 (2000)Google Scholar
  10. 10.
    Kela, J., et al.: Accelerometer-based gesture control for a design environment. Personal and Ubiquitous Computing 10(5), 285–299 (2005)CrossRefGoogle Scholar
  11. 11.
    Kim, S.-H., et al.: An interaction and product design of gesture based TV remote control. In: CHI ’04 extended abstracts, Vienna, Austria, pp. 1548–1548 (2004)Google Scholar
  12. 12.
    Linjama, J., Kaaresoja, T.: Novel, minimalist haptic gesture interaction for mobile devices. In: Proc. Nordic-CHI, 2004, Tampere, Finland, pp. 457–458 (2004)Google Scholar
  13. 13.
    Long, A.C., Landay, J.A., Rowe, L.: Implications for a gesture design tool. In: Human Factors in Computing Systems (SIGCHI Proc.), pp. 40–47. ACM Press, New York (1999)Google Scholar
  14. 14.
    Mäntyjärvi, J., et al.: Gesture interaction for small handheld devices to support multimedia applications. Journal of Mobile Multimedia 1(2), 92–112 (2005)Google Scholar
  15. 15.
    Mori, G., Paternò, F., Santoro, C.: Design and Development of Multi-Device User Interfaces through Multiple Logical Descriptions. IEEE Transactions on Software Engineering 30(8), 507–520 (2004)CrossRefGoogle Scholar
  16. 16.
    Obrenovic, Z., Starcevic, D., Selic, B.: A Model-Driven Approach to Content Repurposing. IEEE Mutimedia, 62–71 (Januray/March 2004)Google Scholar
  17. 17.
    Paternò, F.: Model-Based Design and Evaluation of Interactive Application. Springer, Heidelberg (1999)Google Scholar
  18. 18.
    Paternò, F., Giammarino, F.: Authoring Interfaces with Combined Use of Graphics and Voice for both Stationary and Mobile Devices. In: Proceedings AVI’06, May 2006, ACM Press, New York (2006)Google Scholar
  19. 19.
    Rekimoto, J.: Tilting operations for small screen interfaces. In: ACM UIST 1996, pp. 167–168 (1996)Google Scholar
  20. 20.
    Stanciulescu, A., et al.: A Transformational Approach for Multimodal Web User Interfaces based on USIXML. In: Proceedings ICMI 2005, pp. 259–266. ACM Press, New York (2005)CrossRefGoogle Scholar
  21. 21.
    Szekely, P.: Retrospective and Challenges for Model-Based Interface Development. In: 2nd International Workshop on Computer-Aided Design of User Interfaces, Namur University Press, Namur (1996)Google Scholar
  22. 22.
    Wilson, A., Shafer, S.: Between u and i: XWand: UI for intelligent spaces. In: Proceedings of the conference on Human factors in computing systems, CHI 2003, April 2003, pp. 545–552 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Jani Mäntyjärvi
    • 1
  • Fabio Paternò
    • 2
  • Carmen Santoro
    • 2
  1. 1.VTT Technical Research Centre, Kaitoväylä 1, 90571 OuluFinland
  2. 2.ISTI-CNR, Via G. Moruzzi 1, 56124 PisaItaly

Personalised recommendations