Eye Tracking-Based Workload and Performance Assessment for Skill Acquisition

  • Jesse MarkEmail author
  • Adrian Curtin
  • Amanda Kraft
  • Trevor Sands
  • William D. Casebeer
  • Matthias Ziegler
  • Hasan Ayaz
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 953)


The result of training to improve in a given skill is most often demonstrated by an increase in the relevant performance measures. However, a complementary and at times more informative measure is the mental workload imposed on the performer when doing the task. While a number of varied methods exist for measuring workload, we have chosen to explore physiological and neurological correlates for their low amount of impact and interference on subjects during an experiment. In this study, participants trained on a six-task cognitive battery over four weeks while being simultaneously recorded with remote eye tracking and a host of other neurophysiological instruments. In this preliminary analysis, we found that measures of saccades, fixations, and pupil diameters significantly correlated with task performance over time and at different difficulties, indicating the validity of our task battery as well as the specificity of workload-related eye tracking measures.


Cognitive workload Eye tracking Multimodal Task battery 



This research was supported by the Air Force Research Laboratory’s Human Performance Sensing BAA call 002 under contract number FA8650-16-c-6764. The content of the information herein does not necessarily reflect the position or the policy of the sponsor and no official endorsement should be inferred.


  1. 1.
    Parasuraman, R.: Neuroergonomics: research and practice. Theor. Issues Ergon. Sci. 4(1–2), 5–20 (2003)CrossRefGoogle Scholar
  2. 2.
    Parasuraman, R., Wilson, G.F.: Putting the brain to work: neuroergonomics past, present, and future. Hum. Factors 50(3), 468–474 (2008)CrossRefGoogle Scholar
  3. 3.
    Fedota, J., Parasuraman, R.: Neuroergonomics and human error. AU Theor. Issues Ergon. Sci. 11(5), 402–421 (2010)CrossRefGoogle Scholar
  4. 4.
    Ayaz, H., Pakir, M., Izzetoglu, K., Curtin, A., Shewokis, P.A., Bunce, S.C., Onaral, B.: Monitoring expertise development during simulated UAV piloting tasks using optical brain imaging. In: 2012 IEEE Aerospace Conference (2012)Google Scholar
  5. 5.
    Ayaz, H., Onaral, B., Izzetoglu, K., Shewokis, P.A., McKendrick, R., Parasuraman, R.: Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuroergonomic research: empirical examples and a technological development. Front. Hum. Neurosci. 7, 871 (2013)CrossRefGoogle Scholar
  6. 6.
    Afergan, D., Peck, E.M., Solovey, E.T., Jenkins, A., Hincks, S.W., Brown, E.T., Chang, R., Jacob, R.J.K.: Dynamic difficulty using brain metrics of workload. ACM, Toronto, Ontario, Canada (2014).
  7. 7.
    Hancock, P., Chignell, M.H.: Toward a theory of mental work load: stress and adaptability in human-machine systems. In: Proceedings of the International IEEE Conference on Systems, Man and Cybernetics, 378–383 (1986)Google Scholar
  8. 8.
    Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Sage (2006)Google Scholar
  9. 9.
    Debue, N., van de Leemput, C.: What does germane load mean? An empirical contribution to the cognitive load theory. Front. Psychol. 5, 1099 (2014)CrossRefGoogle Scholar
  10. 10.
    John, M.S., Kobus, D.A., Morrison, J.G.: A multi-tasking environment for manipulating and measuring neural correlates of cognitive workload. In: Proceedings of the IEEE 7th Conference on Human Factors and Power Plants (2002)Google Scholar
  11. 11.
    Miller, G.A.: The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63(2), 81 (1956)CrossRefGoogle Scholar
  12. 12.
    Solovey, E.T., Zec, M., Garcia Perez, E.A., Reimer, B., Mehler, B.: Classifying driver workload using physiological and driving performance data: two field studies. In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems. ACM (2014)Google Scholar
  13. 13.
    Mehler, B., Reimer, B., Coughlin, J.F.: Sensitivity of physiological measures for detecting systematic variations in cognitive demand from a working memory task: an on-road study across three age groups. Hum. Factors 54(3), 396–412 (2012)CrossRefGoogle Scholar
  14. 14.
    Ahlstrom, U., Friedman-Berg, F.J.: Using eye movement activity as a correlate of cognitive workload. Int. J. Ind. Ergon. 36(7), 623–636 (2006)CrossRefGoogle Scholar
  15. 15.
    Jacob, R.J.K., Karn, K.S.: Eye tracking in human-computer interaction and usability research: ready to deliver the promises. In: Hyönä, J., Radach, R., Deubel, H. (eds.) The Mind’s Eye, pp. 573–605. North-Holland, Amsterdam (2003)CrossRefGoogle Scholar
  16. 16.
    Goldberg, J.H., Kotval, X.P.: Computer interface evaluation using eye movements: methods and constructs. Int. J. Ind. Ergon. 24(6), 631–645 (1999)CrossRefGoogle Scholar
  17. 17.
    Parasuraman, R., Christensen, J., Grafton, S.: Neuroergonomics: the brain in action and at work. Neuroimage 59(1), 1–3 (2012)CrossRefGoogle Scholar
  18. 18.
    Ayaz, H., Willems, B., Bunce, S., Shewokis, P.A., Izzetoglu, K., Hah, S., Deshmukh, A., Onaral, B.: Estimation of cognitive workload during simulated air traffic control using optical brain imaging sensors. In: Schmorrow, D.D., Fidopiastis, C.M. (eds.) Foundations of Augmented Cognition. Directing the Future of Adaptive Systems: 6th International Conference, FAC 2011, Held as Part of HCI International 2011, Proceedings, pp. 549–558, Springer, Berlin, Heidelberg, Orlando, FL, USA, 9–14 July 2011CrossRefGoogle Scholar
  19. 19.
    Liu, Y., Ayaz, H., Shewokis, P.A.: Multisubject “learning” for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Front. Hum. Neurosci. 11, 389 (2017)CrossRefGoogle Scholar
  20. 20.
    Di Domenico, S.I., Rodrigo, A.H., Ayaz, H., Fournier, M.A., Ruocco, A.C.: Decision-making conflict and the neural efficiency hypothesis of intelligence: a functional near-infrared spectroscopy investigation. Neuroimage 109, 307–317 (2015)CrossRefGoogle Scholar
  21. 21.
    Durantin, G., Gagnon, J.F., Tremblay, S., Dehais, F.: Using near infrared spectroscopy and heart rate variability to detect mental overload. Behav. Brain Res. 259, 16–23 (2014)CrossRefGoogle Scholar
  22. 22.
    Rodrigo, A.H., Domenico, S.I.D., Ayaz, H., Gulrajani, S., Lam, J., Ruocco, A.C.: Differentiating functions of the lateral and medial prefrontal cortex in motor response inhibition. NeuroImage 85, Part 1(0), 423–431 (2014)CrossRefGoogle Scholar
  23. 23.
    Logan, G.D., Van Zandt, T., Verbruggen, F., Wagenmakers, E.J.: On the ability to inhibit thought and action: general and special theories of an act of control. Psychol. Rev. 121(1), 66–95 (2014)CrossRefGoogle Scholar
  24. 24.
    Shalev, L., Ben-Simon, A., Mevorach, C., Cohen, Y., Tsal, Y.: Conjunctive continuous performance task (CCPT)–a pure measure of sustained attention. Neuropsychologia 49(9), 2584–2591 (2011)CrossRefGoogle Scholar
  25. 25.
    McKendrick, R., Ayaz, H., Olmstead, R., Parasuraman, R.: Enhancing dual-task performance with verbal and spatial working memory training: continuous monitoring of cerebral hemodynamics with NIRS. NeuroImage 85, Part 3(0), 1014–1026 (2014)CrossRefGoogle Scholar
  26. 26.
    Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: N-back working memory paradigm: a meta-analysis of normative functional neuroimaging studies. Hum. Brain Mapp. 25(1), 46–59 (2005)CrossRefGoogle Scholar
  27. 27.
    Wickens, C.D.: Situation awareness and workload in aviation. Curr. Dir. Psychol. Sci. 11(4), 128–133 (2002)CrossRefGoogle Scholar
  28. 28.
    Endsley, M.R.: Design and evaluation for situation awareness enhancement. Proc. Hum. Factors Soc. Annu. Meet. 32(2), 97–101 (1988)CrossRefGoogle Scholar
  29. 29.
    Hagen, K., Ehlis, A.-C., Haeussinger, F.B., Heinzel, S., Dresler, T., Mueller, L.D., Herrmann, M.J., Fallgatter, A.J., Metzger, F.G.: Activation during the trail making test measured with functional near-infrared spectroscopy in healthy elderly subjects. NeuroImage 85, Part 1, 583–591 (2014)CrossRefGoogle Scholar
  30. 30.
    Müller, L.D., Guhn, A., Zeller, J.B.M., Biehl, S.C., Dresler, T., Hahn, T., Fallgatter, A.J., Polak, T., Deckert, J., Herrmann, M.J.: Neural correlates of a standardized version of the trail making test in young and elderly adults: a functional near-infrared spectroscopy study. Neuropsychologia 56, 271–279 (2014)CrossRefGoogle Scholar
  31. 31.
    Aklin, W.M., Lejuez, C.W., Zvolensky, M.J., Kahler, C.W., Gwadz, M.: Evaluation of behavioral measures of risk taking propensity with inner city adolescents. Behav. Res. Ther. 43(2), 215–228 (2005)CrossRefGoogle Scholar
  32. 32.
    Crowley, T.J., Raymond, K.M., Mikulich-Gilbertson, S.K., Thompson, L.L., Lejuez, C.W.: A risk-taking “set” in a novel task among adolescents with serious conduct and substance problems. J. Am. Acad. Child Adolesc. Psychiatry 45(2), 175–183 (2006)CrossRefGoogle Scholar
  33. 33.
    Ebbinghaus, H.: Memory: a contribution to experimental psychology. Ann. Neurosci. 20(4), 155–156 (2013)CrossRefGoogle Scholar
  34. 34.
    Ayaz, H., Dehais, F.: Neuroergonomics: The Brain at Work and Everyday Life, 1st edn. Elsevier, Academic Press, London (2019)Google Scholar
  35. 35.
    Gramann, K., Ferris, D.P., Gwin, J., Makeig, S.: Imaging natural cognition in action. Int. J. Psychophys. 91(1), 22–29 (2014)CrossRefGoogle Scholar
  36. 36.
    Gramann, K., Fairclough, S.H., Zander, T.O., Ayaz, H.: Editorial: trends in neuroergo-nomics. Front. Hum. Neurosci. 11(165) (2017).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Jesse Mark
    • 1
    Email author
  • Adrian Curtin
    • 1
  • Amanda Kraft
    • 2
  • Trevor Sands
    • 2
  • William D. Casebeer
    • 2
  • Matthias Ziegler
    • 2
  • Hasan Ayaz
    • 1
    • 3
    • 4
    • 5
  1. 1.School of Biomedical Engineering, Science, and Health SystemsDrexel UniversityPhiladelphiaUSA
  2. 2.Advanced Technology LaboratoriesLockheed MartinCherry HillUSA
  3. 3.Department of Family and Community HealthUniversity of PennsylvaniaPhiladelphiaUSA
  4. 4.Center for Injury Research and Prevention, Children’s Hospital of PhiladelphiaPhiladelphiaUSA
  5. 5.Drexel Business Solutions Institute, Drexel UniversityPhiladelphiaUSA

Personalised recommendations