Skip to main content
SpringerLink
Log in

Cognitive Workload of Humans Using Artificial Intelligence Systems: Towards Objective Measurement Applying Eye-Tracking Technology

  • Conference paper
KI 2013: Advances in Artificial Intelligence (KI 2013)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 8077))

Included in the following conference series:

Abstract

Replying to corresponding research calls I experimentally investigate whether a higher level of artificial intelligence support leads to a lower user cognitive workload. Applying eye-tracking technology I show how the user’s cognitive workload can be measure more objectively by capturing eye movements and pupillary responses. Within a laboratory environment which adequately reflects a realistic working situation, the probands use two distinct systems with similar user interfaces but very different levels of artificial intelligence support. Recording and analyzing objective eye-tracking data (i.e. pupillary diameter mean, pupillary diameter deviation, number of gaze fixations and eye saccade speed of both left and right eyes) – all indicating cognitive workload – I found significant systematic cognitive workload differences between both test systems. My results indicated that a higher AI-support leads to lower user cognitive workload.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bernstein, A., Klein, M., Malone, T.W.: Programming the Global Brain. CACM 55(5), 41–43 (2012)

    Article  Google Scholar 

  2. Malone, T.W., Laubacher, R., Dellarocas, C.: The Collective Intelligence Genome. MIT Sloan Management Review 51(3), 21–31 (2010)

    Google Scholar 

  3. Davis, J., Lin, H.: Web 3.0 and Crowdservicing. In: AMCIS 2011 Proc. (2011)

    Google Scholar 

  4. Davis, R., Smith, R.G.: Negotiation as a Metaphor for Distributed Problem Solving. AI 20(1), 63–109 (1983)

    Google Scholar 

  5. Carneiro, D., Novais, P., Andrade, F., Zeleznikow, J., Neves, J.: Online dispute resolution: an artificial intelligence perspective. Artif. Intell. Rev. (2012) (in Press)

    Google Scholar 

  6. Park, H.J., Kim, B.K., Lim, K.Y.: Measuring the machine intelligence quotient (MIQ) of human-machine cooperative systems. IEEE TSMC, Part A 31(2), 89–96 (2001)

    Google Scholar 

  7. Wilson, M.: Six views of embodied cognition. Psychonomic Bulletin & Review 9(4), 625–636 (2002)

    Article  Google Scholar 

  8. Cain, B.: A Review of the Mental Workload Literature. Report, NATO (2007)

    Google Scholar 

  9. Ayyagari, R., Grover, V., Purvis, R.: Technostress: Technological Antecedents and Implications. MISQ 35(4), 831–858 (2011)

    Google Scholar 

  10. Tarafdar, M., Tu, Q., Ragu-Nathan, T.S.: Impact of Technostress on End-User Satisfaction and Performance. JMIS 27(3), 303–334 (2010)

    Google Scholar 

  11. Gupta, A., Li, H., Sharda, R.: Should I send this message? Understanding the impact of interruptions, social hierarchy and perceived task complexity on user performance and perceived workload. DSS 55(1), 135–145 (2013)

    Google Scholar 

  12. Ragu-Nathan, T.S., Tarafdar, M., Ragu-Nathan, B.S., Tu, Q.: The Consequences of Technostress for End Users in Organizations: Conceptual Development and Empirical Validation. ISR 19(4), 417–433 (2008)

    Article  Google Scholar 

  13. Wastell, D.G.: Learning Dysfunctions in Information Systems Development: Overcoming the Social Defenses With Transitional Objects. MISQ 23(4), 581–600 (1999)

    Article  Google Scholar 

  14. Sun, Y., Lim, K.H., Peng, J.Z.: Solving the Distinctiveness - Blindness Debate: A Unified Model for Understanding Banner Processing. JAIS 14(2), 49–71 (2013)

    Google Scholar 

  15. Ren, P., Barreto, A., Gao, Y., Adjouadi, M.: Affective Assessment by Digital Processing of the Pupil Diameter. IEEE TAC 4(1), 2–14 (2013)

    Google Scholar 

  16. Dimoka, A.: What Does the Brain Tell Us About Trust and Distrust? Evidence from a Functional Neuroimaging Study. MISQ 34(2), 373–396 (2010)

    Google Scholar 

  17. Dimoka, A., Pavlou, P.A., Davis, F.D.: NeuroIS: The Potential of Cognitive Neuroscience for Information Systems Research. ISR 22(4), 687–702 (2011)

    Article  Google Scholar 

  18. Stassen, H.G., Johannsen, G., Moray, N.: Internal representation, internal model, human performance model and mental workload. Automatica 26(4), 811–820 (1990)

    Article  Google Scholar 

  19. Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich, M.: Common Metrics for Human-Robot Interaction. In: HRI 2006 Proc., pp. 33–40 (2006)

    Google Scholar 

  20. Johannsen, G., Levis, A.H., Stassen, H.G.: Theoretical Problems in Man-machine Systems and Their Experimental Validation. Automatica 30(2), 217–231 (1992)

    Article  Google Scholar 

  21. Lopes, F., Wooldridge, M., Novais, A.Q.: Negotiation among autonomous computational agents: principles, analysis and challenges. Artif. Intell. Rev. 29(1), 1–44 (2008)

    Article  Google Scholar 

  22. Amgoud, L., Vesic, S.: A formal analysis of the role of argumentation in negotiation dialogues. J. Logic Comput. 22(5), 957–978 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  23. Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982)

    Article  Google Scholar 

  24. Hess, E.H., Polt, J.M.: Pupil Size in Relation to Mental Activity during Simple Problem-Solving. Science 143(3611), 1190–1192 (1964)

    Article  Google Scholar 

  25. Kahneman, D., Beatty, J.: Pupil Diameter and Load on Memory. Science 154(3756), 1583–1585 (1966)

    Article  Google Scholar 

  26. Bradshaw, J.: Pupil Size as a Measure of Arousal during Information Processing. Nature 216(5114), 515–516 (1967)

    Article  Google Scholar 

  27. Simpson, H.M.: Effects of a Task-Relevant Response on Pupil Size. Psychophysiology 6(2), 115–121 (1969)

    Article  Google Scholar 

  28. Kahneman, D., Beatty, J., Pollack, I.: Perceptual Deficit during a Mental Task. Science 157(3785), 218–219 (1967)

    Article  Google Scholar 

  29. Beatty, J., Wagoner, B.L.: Pupillometric signs of brain activation vary with level of cognitive processing. Science 199(4334), 1216–1218 (1978)

    Article  Google Scholar 

  30. Rayner, K.: Eye movements in reading and information processing: 20 years of research. Psychol. Bull. 124(3), 372–422 (1998)

    Article  Google Scholar 

  31. Leigh, R.J., Kennard, C.: Using saccades as a research tool in the clinical neurosciences. Brain 127(3), 460–477 (2004)

    Article  Google Scholar 

  32. Van Orden, K.F., Limbert, W., Makeig, S., Jung, T.P.: Eye Activity Correlates of Workload during a Visuospatial Memory Task. Hum. Factors 43(1), 111–121 (2001)

    Article  Google Scholar 

  33. Just, M., Carpenter, P.: Eye fixations and cognitive processes. Cognit. Psychol. 8(4), 441–480 (1976)

    Article  Google Scholar 

  34. Just, M.A., Carpenter, P.A.: A theory of reading: From eye fixations to comprehension. Psychol. Rev. 87(4), 329–354 (1980)

    Article  Google Scholar 

  35. Wierda, S.M., van Rijn, H., Taatgen, N.A., Martens, S.: Pupil dilation deconvolution reveals the dynamics of attention at high temporal resolution. PNAS 109(22), 8456–8460 (2012)

    Article  Google Scholar 

  36. Loft, S., Sanderson, P., Neal, A., Mooij, M.: Modeling and Predicting Mental Workload in En Route Air Traffic Control: Critical Review and Broader Implications. Hum. Factors 49(3), 376–399 (2007)

    Article  Google Scholar 

  37. Bailey, B.P., Iqbal, S.T.: Understanding Changes in Mental Workload during Execution of Goal-Directed Tasks and Its Application for Interruption Management. ACM TOCHI 14(4), 21:1–21:28 (2008)

    Google Scholar 

  38. Baltaci, S., Gokcay, D.: Negative Sentiment in Scenarios Elicit Pupil Dilation Response: An Auditory Study. In: ICMI 2012 Proc., pp. 529–532 (2012)

    Google Scholar 

  39. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., Bailey, B.P.: Towards an Index of Opportunity: Understanding Changes in Mental Workload during Task Execution. In: CHI 2005 Proc., pp. 311–320 (2005)

    Google Scholar 

  40. Wang, W., Li, Z., Wang, Y., Chen, F.: Indexing cognitive workload based on pupillary response under luminance and emotional changes. In: IUI 2013 Proc., pp. 247–256 (2013)

    Google Scholar 

  41. Bee, N., Prendinger, H., Nakasone, A., André, E., Ishizuka, M.: AutoSelect: What You Want Is What You Get: Real-Time Processing of Visual Attention and Affect. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Weber, M. (eds.) PIT 2006. LNCS (LNAI), vol. 4021, pp. 40–52. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  42. Ren, P., Barreto, A., Gao, Y., Adjouadi, M.: Affective Assessment of Computer Users Based on Processing the Pupil Diameter Signal. In: 2011 IEEE Eng. Med. Biol. Soc. Proc., pp. 2594–2597 (2011)

    Google Scholar 

  43. Zhai, J., Barreto, A.: Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables. In: IEEE EMBS 2006 Proc., pp. 1355–1358 (2006)

    Google Scholar 

  44. Pomplun, M., Sunkara, S.: Pupil Dilation as an Indicator of Cognitive Workload in Human-Computer Interaction. In: HCII 2003 Proc., pp. 542–546 (2003)

    Google Scholar 

  45. Longo, L.: Human-Computer Interaction and Human Mental Workload: Assessing Cognitive Engagement in the World Wide Web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part IV. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  46. Cegarra, J., Chevalier, A.: The use of Tholos software for combining measures of mental workload: Toward theoretical and methodological improvements. Behav. Res. Methods 40(4), 988–1000 (2008)

    Article  Google Scholar 

  47. Xu, J., Wang, Y., Chen, F., Choi, E.: Pupillary Response Based Cognitive Workload Measurement under Luminance Changes. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part II. LNCS, vol. 6947, pp. 178–185. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  48. Steinhauer, S.R., Siegle, G.J., Condray, R., Pless, M.: Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. Int. J. Psychophysiol. 52(1), 77–86 (2004)

    Article  Google Scholar 

  49. McBurney, P., Eijk, R.M.V., Parsons, S., Amgoud, L.: A Dialogue Game Protocol for Agent Purchase Negotiations. JAAMAS 7(3), 235–273 (2003)

    Google Scholar 

  50. Buettner, R.: The State of the Art in Automated Negotiation Models of the Behavior and Information Perspective. ITSSA 1(4), 351–356 (2006)

    Google Scholar 

  51. Buettner, R.: A Classification Structure for Automated Negotiations. In: IEEE/WIC/ACM WI-IAT 2006 Proc., pp. 523–530 (2006)

    Google Scholar 

  52. Buettner, R., Landes, J.: Web Service-based Applications for Electronic Labor Markets: A Multi-dimensional Price VCG Auction with Individual Utilities. In: ICIW 2012 Proc., pp. 168–177 (2012)

    Google Scholar 

  53. Landes, J., Buettner, R.: Argumentation–Based Negotiation? Negotiation–Based Argumentation! In: Huemer, C., Lops, P. (eds.) EC-Web 2012. LNBIP, vol. 123, pp. 149–162. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  54. Eckhardt, A., Maier, C., Buettner, R.: The Influence of Pressure to Perform and Experience on Changing Perceptions and User Performance: A Multi-Method Experimental Analysis. In: ICIS 2012 Proc. (2012)

    Google Scholar 

  55. Steinhauer, S.R., Condray, R., Kasparek, A.: Cognitive modulation of midbrain function: task-induced reduction of the pupillary light reflex. Int. J. Psychophysiol. 39(1), 21–30 (2000)

    Article  Google Scholar 

  56. Gettinger, J., Koeszegi, S.T., Schoop, M.: Shall we dance? - The effect of information presentations on negotiation processes and outcomes. DSS 53, 161–174 (2012)

    Google Scholar 

  57. Schoop, M., Köhne, F., Staskiewicz, D.: An Integrated Decision and Communication Perspective on Electronic Negotiation Support Systems - Challenges and Solutions. Journal of Decision Systems 13(4), 375–398 (2004)

    Article  Google Scholar 

  58. Dimoka, A., Davis, F.D.: Where Does TAM Reside in the Brain? The Neural Mechanisms Underlying Technology Adoption. In: ICIS 2008 Proc., Paper 169 (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Management & Information Systems, FOM University of Applied Sciences, Hopfenstraße 4, 80335, Munich, Germany

    Ricardo Buettner

Authors
  1. Ricardo Buettner
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Business Informatics I, University of Trier, 54286, Trier, Germany

    Ingo J. Timm

  2. Institute for Web Science and Technologies, University of Koblenz, Universitätsstr. 1, 56070, Koblenz, Germany

    Matthias Thimm

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Buettner, R. (2013). Cognitive Workload of Humans Using Artificial Intelligence Systems: Towards Objective Measurement Applying Eye-Tracking Technology. In: Timm, I.J., Thimm, M. (eds) KI 2013: Advances in Artificial Intelligence. KI 2013. Lecture Notes in Computer Science(), vol 8077. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40942-4_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-40942-4_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-40941-7

  • Online ISBN: 978-3-642-40942-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation