Empirical Evaluation of Multimodal Input Interactions

  • Sanjay Ghosh
  • Anirudha Joshi
  • Sanjay Tripathi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8016)


With variety of interaction technologies like speech, pen, touch, hand or body gestures, eye gaze, etc., being now available for users, it is a challenge to design optimal and effective multimodal combinations for specific tasks. For designing that, it is important to understand how these modalities can be combined and used in a coordinated manner. We performed an experimental evaluation of combinations of different multimodal inputs, such as keyboard, speech and touch with pen etc, in an attempt to investigate, which combinations are efficient for diverse needs of the users. In our study, multimodal combination of three modalities was found to be more effective in terms of performance, accuracy and user experience than that of two modalities. Further, we also inferred the roles that each of the modalities play in a multimodal combination to achieve the usability goals.


Multimodal interaction modality combinations usability testing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Dybkaer, L., Bernsen, N.O., Minker, W.: New Challenges in Usability Evaluation-Beyond Task-Oriented Spoken Dialogue Systems. In: ICSLP, vol. 3, pp. 2261–2264 (2004)Google Scholar
  2. 2.
    Bretan, I., Karlgren, J.: Synergy Effects in Natural Language-Based Multimodal Interaction. SICS Research Report (1994)Google Scholar
  3. 3.
    Wechsung, I., Engelbrecht, K.P., Schaffer, S., Seebode, J., Metze, F., Möller, S.: Usability Evaluation of Multimodal Interfaces: Is the Whole the Sum of Its Parts? In: Jacko, J.A. (ed.) HCI International 2009, Part II. LNCS, vol. 5611, pp. 113–119. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  4. 4.
    Oviatt, S.: User-centered modeling and evaluation of multimodal interfaces. IEEE 91(9), 1457–1468 (2003)CrossRefGoogle Scholar
  5. 5.
    Ren, X., Zhang, G., Dai, G.: An experimental study of input modes for multimodal human-computer interaction. In: Tan, T., Shi, Y., Gao, W. (eds.) ICMI 2000. LNCS, vol. 1948, pp. 49–56. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  6. 6.
    Metze, F., Wechsung, I., Schaffer, S., Seebode, J., Möller, S.: Reliable Evaluation of Multimodal Dialogue Systems. In: Jacko, J.A. (ed.) Human-Computer Interaction, Part II, HCII 2009. LNCS, vol. 5611, pp. 75–83. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  7. 7.
    Bernsen, N.O., Dybkjær, L.: Evaluation of spoken multimodal conversation. In: 6th ACM International Conference on Multimodal Interfaces, pp. 38–45 (2004)Google Scholar
  8. 8.
    Serrano, M., Nigay, L.: A wizard of oz component-based approach for rapidly prototyping and testing input multimodal interfaces. J. Multimodal User Interfaces 3(3), 215–225 (2010)CrossRefGoogle Scholar
  9. 9.
    Bernhaupt, R., Palanque, P., Winckler, M., Navarre, D.: Usability Study of Multi-modal Interfaces Using Eye-Tracking. In: Baranauskas, C., Abascal, J., Barbosa, S.D.J. (eds.) INTERACT 2007. LNCS, vol. 4663, pp. 412–424. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  10. 10.
    Bernhaupt, R., Navarre, D., Palanque, P., Winckler, M.: Model-Based Evaluation: A New Way to Support Usability Evaluation of Multimodal Interactive Applications. In: Maturing Usability: Quality in Software, Interaction and Quality, pp. 96–122 (2007)Google Scholar
  11. 11.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: QuickSet: Multimodal Interaction for Distributed Applications. In: 5th ACM International Conference on Multimedia, pp. 31–40 (1997)Google Scholar
  12. 12.
    Kaster, T., Pfeiffer, M., Bauckhage, C.: Combining Speech and Haptics for Intuitive and Efficient Navigation through Image Databases. In: 5th ACM International Conference on Multimodal Interfaces, pp. 180–187 (2003)Google Scholar
  13. 13.
    Beelders, T.R., Blignaut, P.J.: The Usability of Speech and Eye Gaze as a Multimodal Interface for a Word Processor. In: Speech Technologies, pp. 386–404 (2011)Google Scholar
  14. 14.
    Microsoft Voice Recognition System, (last retrieved on February 26, 2013)
  15. 15.
    Sauro, J., Kindlund, E.A.: Method to standardize usability metrics into a single score. In: ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 401–409 (2005)Google Scholar
  16. 16.
    Hornbæk, K., Law, E.L.C.: Meta-analysis of correlations among usability measures. In: ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 617–626 (2007)Google Scholar
  17. 17.
    Cohen, P.R., Johnston, M., McGee, D., Oviatt, S.L., Clow, J., Smith, I.: The efficiency of multimodal interaction: A case study. In: International Conference on Spoken Language Processing, vol. 2, pp. 249–252 (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Sanjay Ghosh
    • 1
    • 2
  • Anirudha Joshi
    • 2
  • Sanjay Tripathi
    • 1
  1. 1.Industrial Software SystemsABB Corporate ResearchBangaloreIndia
  2. 2.Indian Institute of TechnologyBombayIndia

Personalised recommendations