A Survey of Multimodal Interfaces for Mobile Mapping Applications

  • Julie Doyle
  • Michela Bertolotto
  • David Wilson
Part of the Lecture Notes in Geoinformation and Cartography book series (LNGC)


The user interface is of critical importance in applications providing mapping services. It defines the visualisation and interaction modes for carrying out a variety of mapping tasks, and ease of use is essential to successful user adoption of a mapping application. This is redoubled in a mobile context, where mobile device limitations can hinder usability. In particular, interaction modes such as a pen/stylus are limited and can be quite difficult to use in a mobile context. Moreover, the majority of GIS interfaces are inherently complex and require significant user training, which can be a serious problem for novice users such as tourists. We propose an increased focus on developing multimodal interfaces for mobile GIS, allowing for two or more modes of input, as an attempt to address interaction complexity in the context of mobile mapping applications. Such interfaces allow users to choose the modes of interaction that are not only most intuitive to them, but also most suitable for their current task and environment. This chapter presents the user interaction problem and the utility of multimodal interfaces for mobile GIS. We describe our multimodal mobile GIS CoMPASS which helps to address the problem by permitting users to interact with spatial data using a combination of speech and gesture input. CoMPASS is set in the context of a representative survey across a range of comparable multimodal systems, and the effectiveness of our approach is evaluated in a user study which demonstrates that multimodal interfaces provide more intuitive and efficient interaction for mobile mapping applications.


Speech Recognition Multimodal Interface Multimodal Interaction Multimodal System Voice Command 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Benoit, C., Martin, J.C., Pelachaud, C., Schomaker, L. and Suhm, B. (2000): Audio-visual and Multimodal Speech-based Systems. In Handbook of Multimodal and Spoken Dialogue Systems: Resources, Terminology and Product Evaluation, D. Gibbon, I. Mertins and R. Moore (Eds.), pp. 102-203, Kluwer.Google Scholar
  2. Bertolotto, M. and Egenhofer, M. (1999): Progressive Vector Transmission. In 7th ACM Symposium on Advances in Geographic Information Systems, pp. 152-157, Kansas City, Kansas.Google Scholar
  3. Bolt, R. A. (1980): Put-That-There: Voice and Gesture at the Graphics Interface. In The 7 th International Conference on Computer Graphics and Interactive Techniques, pp. 262-270, Seattle, Washington.Google Scholar
  4. Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L. and Clow, J. (1997): QuickSet: Multimodal Interaction for Distributed Applications. In 5th ACM Conference on Multimedia, pp. 31-40, New York, New York.Google Scholar
  5. Cohen, P., McGee, D. and Clow, J. (2000): The Efficiency of Multimodal Interaction for a Map-based Task. In 6th International Conference on Applied Natural Language Processing, pp 331-338, Seattle, Washington.Google Scholar
  6. Constantini, E., Pianesi, F. and Prete, M. (2005): Recognising Emotions in Human and Synthetic Faces: The Role of the Upper and Lower Parts of the Face. In 10th International Conference on Intelligent User Interfaces, pp. 20-27, San Diego, California.Google Scholar
  7. Doyle, J., Weakliam, J., Bertolotto, M. and Wilson, D. (2006a): A Multimodal Interface for Personalising Spatial Data in Mobile GIS. In ICEIS – 8 th International Conference on Enterprise Information Systems, in press, Paphos, Cyprus.Google Scholar
  8. Doyle, J. (2007a): Multimodal Interface Design for Mobile GIS. A PhD thesis submitted to University College Dublin, January 2007.Google Scholar
  9. Doyle, J., Bertolotto, M. and Wilson, D. (2007b): Multimodal Interface Design for Geospatial LBS. Under review for the Journal of Location Based Services.Google Scholar
  10. Fuhrmann, S., MacEachren, A., Dou, J., Wang, K. and Cox, A. (2005): Gesture and Speech-based Maps to Support Use of GIS for Crisis Management: A User Study. In AutoCarto 2005, Las Vegas, Nevada.Google Scholar
  11. IBM ViaVoice (2006):Google Scholar
  12. Scholar
  13. Jost, M., Haussler, J., Merdes, M. and Malaka, R. (2005): Multimodal Interaction for Pedestrians: An Evaluation Study. In 10th International Conference on Intelligent User Interfaces, pp. 59-66, San Diego, California.Google Scholar
  14. Krueger, A., Butz, A., Muller, C., Stahl, C., Wasinger, R., Steinberg, K.E. and Dirschl, A. (2004): The Connected User Interface: Realizing a Personal Situated Navigation Service. In ,bt>The 9 th International Conference on Intelligent User Interfaces (IUI 04), pp. 161-168, Madeira, Portugal, ACM press.Google Scholar
  15. Lynch, D., Bertolotto, M. and Wilson, D. (2005): Spatial Annotations in a Mapping Environment. In GISRUK ‘05, GIS Research UK, pp. 524-528, Glasgow, Scotland.Google Scholar
  16. Malaka, M., Haeussler, J. and Hidir, A. (2004): SmartKom Mobile – Intelligent Ubiquitous User Interaction. In 9th International Conference on Intelligent User Interfaces, pp. 310-312, Madeira, Portugal.Google Scholar
  17. Nickel, K. and Stiefelhagen, R. (2003): Pointing Gesture Recognition based on 3D-tracking of Face, Hands and Head Orientation. In 5th International Conference on Multimodal Interfaces, pp. 140-146, Vancouver, Canada.Google Scholar
  18. OpenMap (2006): Scholar
  19. Oviatt, S. (1996a): Multimodal Interfaces for Dynamic Interactive Maps. In SIGCHI Conference on Human Factors in Computing Systems, pp. 95-102, Vancouver, Canada.Google Scholar
  20. Oviatt, S. (1999): Mutual Disambiguation of Recognition Errors in a Multimodal Architecture. In The Conference on Human Factors in Computing Systems (CHI ’99), pp. 576-583, Pittsburgh, Pennsylvania.Google Scholar
  21. Oviatt, S. (2000a): Taming Recognition Errors with a Multimodal Interface. In Communications of the ACM, Vol. 43, No. 9, pp.45-51.Google Scholar
  22. Oviatt, S. (2000b): Multimodal System Processing in Mobile Environments. In 13th Annual ACM Symposium on User Interface Software and Technology,pp. 21-30, San Diego, California.Google Scholar
  23. Oviatt, S. (2003): Multimodal Interfaces. In Handbook of Human-computer Interaction, J. Jacko and A. Sears (Eds.), pp. 286-304, New Jersey.Google Scholar
  24. Qvarfordt, P. (2003): Spoken Feedback in Multimodal Interaction: Effects on Users Experience of Qualities of Interaction. In 1st Nordic Symposium on Multimodal Communication, pp. 21-34.Google Scholar
  25. Qvarfordt, P. and Zhai, S. (2005): Conversing with the User based on Eye-Gaze Patterns. In SIGCHI Conference on Human Factors in Computing Systems, pp. 221-230, Portland, Oregon.Google Scholar
  26. Rauschert, I., Agrawal, P., Sharma, R., Fuhrmann, S., Brewer, I., MacEachren, A., Wang, H. and Cai, G. (2002a): Designing a Human-Centered, Multimodal GIS Interface to Support Emergency Management. In 10th ACM International Symposium on Advances in Geographic Information Systems, pp. 119-124, Virginia.Google Scholar
  27. Rauschert, I., Sharma, R., Fuhrmann, S., Brewer, I. And MacEachren, A. (2002b): Approaching a New Multimodal Interface. In 2nd International Conference on GIS (GIScience), Colorado.Google Scholar
  28. Rugelbak, J. and Hamnes, K. (2003): Multimodal Interaction – Will Users Tap and Speak Simultaneously? In Telektronikk ’03, pp. 118-124.Google Scholar
  29. Schapira, E. and Sharma, R. (2001): Experimental Evaluation of Vision and Speech based Multimodal Interfaces. In Workshop on Perceptive User Interfaces, pp. 1-9, Florida.Google Scholar
  30. Suhm, B., Myers, B. and Waibel, A. (2001): Multimodal Error Correction for Speech User Interfaces. In ACM Transactions on Computer-Human Interaction, Vol. 8, No. 1, pp. 60-98.CrossRefGoogle Scholar
  31. Wasinger, R. and Kreuger, A. (2004): Multimodal Interaction with Mobile Navigation Systems. In Information Technology, Vol. 46, No. 6, pp. 322-331, Oldenbourg Publishers.Google Scholar
  32. Wasinger, R., Stahl, C. and Kreuger, A. (2003a): M3I in a Pedestrian Navigation and Exploration System. In The 4 th International Symposium on Human Computer Interaction with Mobile Devices (Mobile HCI 03), pp. 481-485, Udine, Italyl, Springer-Verlag LNCS.Google Scholar
  33. Wasinger, R., Stahl, C. and Kreuger, A. (2003b): Robust Speech Interaction through the use of Multiple and Different Media Input Types. In Eurospeech 2003, pp. 1049-1052.Google Scholar
  34. Weakliam, J., Bertolotto, M. and Wilson, D. (2005a): Implicit Interaction Profiling for Recommending Spatial Content. In ACMGIS ’05, pp. 285-294, Bremen, Germany.Google Scholar
  35. Weakliam, J., Lynch, D., Doyle, J., Bertolotto, M and Wilson, D. (2005b): Delivering Personalized Context-Aware Spatial Information to Mobile Devices. In W2GIS, 5 th International Workshop on Web and Wireless Geographic Information Systems, pp. 194-205, Lausanne, Switzerland.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Julie Doyle
    • 1
  • Michela Bertolotto
    • 1
  • David Wilson
    • 2
  1. 1.School of Computer Science and InformaticsUniversity College DublinDublin
  2. 2.Department of Software and Information SystemsUniversity of North Carolina at CharlotteCharlotte

Personalised recommendations