Multimedia Tools and Applications

, Volume 56, Issue 3, pp 553–567 | Cite as

Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

  • Domen NovakEmail author
  • Matjaž Mihelj
  • Marko Munih


This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants’ subjective feelings.


Psychophysiology Affective computing Multitasking Human-computer interaction Mental workload 



The authors would like to thank the volunteers who participated in this study. The work was funded by the EU Information and Communication Technologies Collaborative Project MIMICS grant 215756 and additionally supported by the Slovenian Research Agency.


  1. 1.
    Backs RW (1997) Psychophysiological aspects of selective and divided attention during continuous manual tracking. Acta Psychol 96:167–191CrossRefGoogle Scholar
  2. 2.
    Backs RW, Lenneman JK, Wetzel JM, Green P (2003) Cardiac measures of driver workload during simulated driving with and without visual occlusion. Hum Factors 45:525–538CrossRefGoogle Scholar
  3. 3.
    Boiten F (1993) Component analysis of task-related respiratory patterns. Int J Psychophysiol 15:91–104CrossRefGoogle Scholar
  4. 4.
    Chen JYC (2009) Concurrent performance of military and robotics tasks and effects of cueing in a simulated multi-tasking environment. Presence–Teleop Virt 18: 1–15Google Scholar
  5. 5.
    Collet C, Averty P, Dittmar A (2009) Autonomic nervous system and subjective ratings of strain in air traffic control. Appl Ergon 40:23–32CrossRefGoogle Scholar
  6. 6.
    Ćosić K, Popović S, Jovanovic T, Kukolja D, Slamić M (2007) Physiology-driven adaptive VR system: Technology and rationale for PTSD treatment. Annual Review of CyberTherapy & Telemedicine 5:179–191Google Scholar
  7. 7.
    Derrick WL (1988) Dimensions of operator workload. Hum Factors 30:95–110Google Scholar
  8. 8.
    Detenber BH, Simons RF, Bennett GG (1998) Roll 'em!: the effects of picture motion on emotional responses. J Broadcast Electron 42:113–127CrossRefGoogle Scholar
  9. 9.
    Feldman PJ, Cohen S, Lepore SJ, Matthews KA, Kamarck TW, Marsland AL (1999) Negative emotions and acute physiological responses to stress. Ann Behav Med 21:216–222CrossRefGoogle Scholar
  10. 10.
    Gwizdka J (2010) Distribution of cognitive load in web search. J Am Soc Inf Sci Technol, in pressGoogle Scholar
  11. 11.
    Horrey WJ, Lesch MF, Garabet A (2009) Dissociation between driving performance and drivers' subjective estimates of performance and workload in dual-task conditions. J Safety Res 40:7–12CrossRefGoogle Scholar
  12. 12.
    Kahneman D (1973) Attention and Effort. Prentice Hall, Englewood CliffsGoogle Scholar
  13. 13.
    Kahol K, Panchanathan S (2008) Neuro-cognitively inspired haptic user interfaces. Multimed Tools Appl 37:15–38CrossRefGoogle Scholar
  14. 14.
    Liu C, Agrawal P, Sarkar N, Chen S (2009) Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int J Hum-Comput Int 25:506–529CrossRefGoogle Scholar
  15. 15.
    Liu J, Gwizdka J, Liu C, Belkin NJ (2010) Predicting task difficulty for different task types. In: Proceedings of the 73rd Annual Meeting of the American Society for Information Science & Technology. Pittsburgh, PA, in pressGoogle Scholar
  16. 16.
    Martinez JM, Kent JM, Coplan JD, Browne ST, Papp LA, Sullivan GM et al (2001) Respiratory variability in panic disorder. Depress Anxiety 14:232–237CrossRefGoogle Scholar
  17. 17.
    Mulder G, Mulder LJ, Meijman TF, Veldman JB, van Roon AM (2000) A psychophysiological approach to working conditions. In: Backs R, Boucsein W (eds) Engineering psychophysiology: Issues and applications. Lawrence Erlbaum Associates, Mahwah, pp 139–159Google Scholar
  18. 18.
    Nikula R (1991) Psychological correlates of nonspecific skin conductance responses. Psychophysiology 28:86–90CrossRefGoogle Scholar
  19. 19.
    Novak D, Mihelj M, Munih M (2010) Psychophysiological responses to different levels of cognitive and physical load in haptic interaction. Robotica, in pressGoogle Scholar
  20. 20.
    Octavia JR, Raymaekers C, Coninx K (2010) Adaptation in virtual environments: conceptual framework and user models. Multimed Tools Appl, in pressGoogle Scholar
  21. 21.
    Ohsuga M, Shimono F, Genno H (2001) Assessment of phasic work stress using autonomic indices. Int J Psychophysiol 40:211–220CrossRefGoogle Scholar
  22. 22.
    Parasuraman R, Cosenzo KA, De Visser E (2009) Adaptive automation for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness and mental workload. Mil Psychol 21:270–297CrossRefGoogle Scholar
  23. 23.
    Parkes AM, Coleman N (1990) Route guidance systems: a comparison of methods of presenting directional information to the driver. In: Lovesey J (ed) Contemporary Ergonomics 1990. Taylor & Francis, London, pp 480–485Google Scholar
  24. 24.
    Pashler H, Johnston JC (1998) Attentional limitations in dual-task performance. In: Pashler H (ed) Attention. Psychology Press, East Sussex, pp 155–189Google Scholar
  25. 25.
    Phillips AC, Carroll D, Hunt K, Der G (2006) The effects of the spontaneous presence of a spouse/partner and others on cardiovascular reactions to an acute psychological challenge. Psychophysiology 43:633–640CrossRefGoogle Scholar
  26. 26.
    Posner MI, Sandson J, Dhawan M, Shulman GL (1989) Is word recognition automatic? A cognitive anatomical approach. J Cogn Neurosci 1:50–60CrossRefGoogle Scholar
  27. 27.
    Ritvanen T, Louhevaara V, Helin P, Väisänen S, Hänninen O (2006) Responses of the autonomic nervous system during periods of perceived high and low work stress in younger and older female teachers. Appl Ergon 37:311–318CrossRefGoogle Scholar
  28. 28.
    Schwerdtfeger A (2004) Predicting autonomic reactivity to public speaking: don't get fixed on self-report data! Int J Psychophysiol 52:217–224CrossRefGoogle Scholar
  29. 29.
    Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (1996) Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. Eur Heart J 17:354–381Google Scholar
  30. 30.
    Veltman JA, Gaillard AW (1998) Physiological workload reactions to increasing levels of task difficulty. Ergonomics 41:656–669CrossRefGoogle Scholar
  31. 31.
    Wester AE, Böcker KB, Volkerts ER, Verster JC, Kenemans JL (2008) Event-related potentials and secondary task performance during simulated driving. Accident Anal Prev 40:1–7CrossRefGoogle Scholar
  32. 32.
    Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3:159–177CrossRefGoogle Scholar
  33. 33.
    Wilson GF, Russell CA (2003) Operator functional state classification using multiple psychophysiological features in an air traffic control task. Hum Factors 45:381–389CrossRefGoogle Scholar
  34. 34.
    Xie B, Salvendy G (2000) Review and reappraisal of modelling and predicting mental workload in single- and multi-task environments. Work Stress 14:74–99CrossRefGoogle Scholar
  35. 35.
    Zhang T, Kaber D, Hsiang S (2010) Characterisation of mental models in a virtual reality-based multitasking scenario using measures of situation awareness. Theor Issues Ergon Sci 11:99–118CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  1. 1.Faculty of Electrical EngineeringUniversity of LjubljanaLjubljanaSlovenia

Personalised recommendations