Skip to main content
SpringerLink
Log in

Analysing the Impact of Machine Learning to Model Subjective Mental Workload: A Case Study in Third-Level Education

  • Conference paper
  • First Online:
Human Mental Workload: Models and Applications (H-WORKLOAD 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1012))

Abstract

Mental workload measurement is a complex multidisciplinary research area that includes both the theoretical and practical development of models. These models are aimed at aggregating those factors, believed to shape mental workload, and their interaction, for the purpose of human performance prediction. In the literature, models are mainly theory-driven: their distinct development has been influenced by the beliefs and intuitions of individual scholars in the disciplines of Psychology and Human Factors. This work presents a novel research that aims at reversing this tendency. Specifically, it employs a selection of learning techniques, borrowed from machine learning, to induce models of mental workload from data, with no theoretical assumption or hypothesis. These models are subsequently compared against two well-known subjective measures of mental workload, namely the NASA Task Load Index and the Workload Profile. Findings show how these data-driven models are convergently valid and can explain overall perception of mental workload with a lower error.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aghajani, H., Garbey, M., Omurtag, A.: Measuring mental workload with EEG+ fNIRS. Front. Hum. Neurosci. 11, 359 (2017)

    Article  Google Scholar 

  2. Batista, G.E., Monard, M.C.: A study of K-nearest neighbour as an imputation method. HIS 87(251–260), 48 (2002)

    Google Scholar 

  3. Bennett, K.P., Campbell, C.: Support vector machines. ACM SIGKDD Explor. Newsl. 2(2), 1–13 (2000). http://portal.acm.org/citation.cfm?doid=380995.380999

    Article  Google Scholar 

  4. Cain, B.: A review of the mental workload literature. Technical report, Defence Research and Development Canada Toronto Human System Integration Section; 2007. Report Contract No. RTO-TRHFM-121-Part-II (2004)

    Google Scholar 

  5. Carlson, K.D., Herdman, A.O.: Understanding the impact of convergent validity on research results. Organ. Res. Methods 15(1), 17–32 (2012)

    Article  Google Scholar 

  6. Chai, T., Draxler, R.R.: Root mean square error (RMSE) or mean absolute error (MAE)?-arguments against avoiding RMSE in the literature. Geosci. Model Dev. 7(3), 1247–1250 (2014)

    Article  Google Scholar 

  7. Chapman, P., Clinton, J., Khabaza, T., Reinartz, T., Wirth, R.: The crisp-dmprocess model. The CRIP–DM Consortium 310 (1999)

    Google Scholar 

  8. Cortes Torres, C.C., Sampei, K., Sato, M., Raskar, R., Miki, N.: Workload assessment with eye movement monitoring aided by non-invasive and unobtrusive micro-fabricated optical sensors. In: Adjunct Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology, pp. 53–54. ACM (2015)

    Google Scholar 

  9. Fan, J., Smith, A.P.: The impact of workload and fatigue on performance. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 90–105. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_6

    Chapter  Google Scholar 

  10. Gelman, A., Jakulin, A., Pittau, M.G., Su, Y.S.: A weakly informative default prior distribution for logistic and other regression models. Ann. Appl. Stat. 2(4), 1360–1383 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  11. Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)

    Article  MATH  Google Scholar 

  12. Hancock, P.A.: Whither workload? Mapping a path for its future development. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 3–17. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_1

    Chapter  Google Scholar 

  13. Hancock, P.A., Meshkati, N.: Human Mental Workload. Elsevier, Amsterdam (1988)

    Google Scholar 

  14. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52(C), 139–183 (1988)

    Article  Google Scholar 

  15. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Human Factors and Ergonomics Society Annual Meting, pp. 904–908 (2006)

    Article  Google Scholar 

  16. Hincks, S.W., Afergan, D., Jacob, R.J.K.: Using fNIRS for real-time cognitive workload assessment. In: Schmorrow, D.D.D., Fidopiastis, C.M.M. (eds.) AC 2016. LNCS (LNAI), vol. 9743, pp. 198–208. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39955-3_19

    Chapter  Google Scholar 

  17. Jonsson, P., Wohlin, C.: An evaluation of k-nearest neighbour imputation using Likert data. In: 2004 Proceedings of 10th International Symposium on Software Metrics, pp. 108–118, September 2004

    Google Scholar 

  18. Kotsiantis, S.B.: Supervised machine learning: a review of classification techniques. Informatica 31(2), 249–268 (2007). https://books.google.co.in/books?hl=en&lr=&id=vLiTXDHr_sYC&oi=fnd&pg=PA3&dq=survey+machine+learning&ots=CVsyuwYHjo&redir_esc=y#v=onepage&q=survey%20machine%20learning&f=false

  19. Kvålseth, T.O.: Cautionary note about R\(^2\). Am. Stat. 39(4), 279–285 (1985)

    Google Scholar 

  20. Liu, Y., Ayaz, H., Shewokis, P.A.: Multisubject “learning” for mental workload classification using concurrent EEG, fNIRS, and physiological measures. Front. Hum. Neurosci. 11, 389 (2017)

    Article  Google Scholar 

  21. Longo, L.: Formalising human mental workload as a defeasible computational concept. Ph.D. thesis, Trinity College, Dublin (2014)

    Google Scholar 

  22. Longo, L.: A defeasible reasoning framework for human mental workload representation and assessment. Behav. Inf. Technol. 34(8), 758–786 (2015)

    Article  Google Scholar 

  23. Longo, L.: Designing medical interactive systems via assessment of human mental workload. In: International Symposium on Computer-Based Medical Systems, pp. 364–365 (2015)

    Google Scholar 

  24. Longo, L.: Mental workload in medicine: foundations, applications, open problems, challenges and future perspectives. In: 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 106–111. IEEE (2016)

    Google Scholar 

  25. Longo, L.: Subjective usability, mental workload assessments and their impact on objective human performance. In: Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D.K., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10514, pp. 202–223. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67684-5_13

    Chapter  Google Scholar 

  26. Longo, L.: Experienced mental workload, perception of usability, their interaction and impact on task performance. PloS ONE 13(8), 1–36 (2018). https://doi.org/10.1371/journal.pone.0199661

    Article  Google Scholar 

  27. Longo, L.: On the reliability, validity and sensitivity of three mental workload assessment techniques for the evaluation of instructional designs: a case study in a third-level course. In: Proceedings of the 10th International Conference on Computer Supported Education, CSEDU 2018, Funchal, Madeira, Portugal, 15–17 March 2018, vol. 2, pp. 166–178 (2018). https://doi.org/10.5220/0006801801660178

  28. Longo, L., Barrett, S.: Cognitive effort for multi-agent systems. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) BI 2010. LNCS (LNAI), vol. 6334, pp. 55–66. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15314-3_6

    Chapter  Google Scholar 

  29. Longo, L., Dondio, P.: On the relationship between perception of usability and subjective mental workload of web interfaces. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 1, pp. 345–352. IEEE (2015)

    Google Scholar 

  30. Longo, L., Leva, M.C. (eds.): H-WORKLOAD 2017. CCIS, vol. 726. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0

    Book  Google Scholar 

  31. Longo, L., Rusconi, F., Noce, L., Barrett, S.: The importance of human mental workload in web-design. In: 8th International Conference on Web Information Systems and Technologies, pp. 403–409, April 2012

    Google Scholar 

  32. Mannaru, P., Balasingam, B., Pattipati, K., Sibley, C., Coyne, J.: Cognitive context detection in UAS operators using eye-gaze patterns on computer screens. In: Next-Generation Analyst IV, vol. 9851, p. 98510F. International Society for Optics and Photonics (2016)

    Google Scholar 

  33. Mayer, R.E.: Cognitive theory of multimedia learning, 2nd edn. In: Cambridge Handbooks in Psychology, pp. 43–71. Cambridge University Press, Cambridge (2014)

    Google Scholar 

  34. Meshkati, N., Loewenthal, A.: An eclectic and critical review of four primary mental workload assessment methods: a guide for developing a comprehensive model. Adv. Psychol. 52(1978), 251–267 (1988). http://www.sciencedirect.com/science/article/pii/S0166411508623912

  35. Mijović, P., Milovanović, M., Ković, V., Gligorijević, I., Mijović, B., Mačužić, I.: Neuroergonomics method for measuring the influence of mental workload modulation on cognitive state of manual assembly worker. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 213–224. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_14

    Chapter  Google Scholar 

  36. Mohammadi, M., Mazloumi, A., Kazemi, Z., Zeraati, H.: Evaluation of mental workload among ICU ward’s nurses. Health Promot. Perspect. 5(4), 280–7 (2015). http://www.ncbi.nlm.nih.gov/pubmed/26933647, http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4772798

    Article  Google Scholar 

  37. Monfort, S.S., Sibley, C.M., Coyne, J.T.: Using machine learning and real-time workload assessment in a high-fidelity UAV simulation environment. In: Next-Generation Analyst IV, vol. 9851, p. 98510B. International Society for Optics and Photonics (2016)

    Google Scholar 

  38. Moustafa, K., Luz, S., Longo, L.: Assessment of mental workload: a comparison of machine learning methods and subjective assessment techniques. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 30–50. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_3

    Chapter  Google Scholar 

  39. Nevo, B.: Face validity revisited. J. Educ. Meas. 22(4), 287–293 (1985)

    Article  Google Scholar 

  40. Ott, T., Wu, P., Paullada, A., Mayer, D., Gottlieb, J., Wall, P.: ATHENA – a zero-intrusion no contact method for workload detection using linguistics, keyboard dynamics, and computer vision. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 617, pp. 226–231. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40548-3_38

    Chapter  Google Scholar 

  41. Pham, T.T., Nguyen, T.D., Van Vo, T.: Sparse fNIRS feature estimation via unsupervised learning for mental workload classification. In: Bassis, S., Esposito, A., Morabito, F.C., Pasero, E. (eds.) Advances in Neural Networks. SIST, vol. 54, pp. 283–292. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33747-0_28

    Chapter  Google Scholar 

  42. Reid, G.B., Nygren, T.E.: The subjective workload assessment technique: a scaling procedure for measuring mental workload. In: Advances in Psychology, vol. 52, pp. 185–218. Elsevier (1988)

    Google Scholar 

  43. Rizzo, L., Dondio, P., Delany, S.J., Longo, L.: Modeling mental workload via rule-based expert system: a comparison with NASA-TLX and workload profile. In: Iliadis, L., Maglogiannis, I. (eds.) AIAI 2016. IAICT, vol. 475, pp. 215–229. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44944-9_19

    Chapter  Google Scholar 

  44. Rizzo, L., Longo, L.: Representing and inferring mental workload via defeasible reasoning: a comparison with the NASA task load index and the workload profile. In: Proceedings of the 1st Workshop on Advances In Argumentation In Artificial Intelligence Co-located with XVI International Conference of the Italian Association for Artificial Intelligence (AI * IA 2017), Bari, Italy, 16–17 November 2017, pp. 126–140 (2017)

    Google Scholar 

  45. Rizzo, L., Longo, L.: Inferential models of mental workload with defeasible argumentation and non-monotonic fuzzy reasoning: a comparative study. In: Proceedings of the 2nd Workshop on Advances In Argumentation In Artificial Intelligence Co-located with XVII International Conference of the Italian Association for Artificial Intelligence (AI*IA 2018), Trento, Italy, 20–23 November 2018, pp. 11–26 (2018)

    Google Scholar 

  46. Rubio, S., Díaz, E., Martín, J., Puente, J.M.: Evaluation of subjective mental workload: a comparison of swat, NASA-TLX, and workload profile methods. Appl. Psychol. 53(1), 61–86 (2004). https://doi.org/10.1111/j.1464-0597.2004.00161.x

    Article  Google Scholar 

  47. Smith, A.P., Smith, H.N.: Workload, fatigue and performance in the rail industry. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 251–263. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_17

    Chapter  Google Scholar 

  48. Smith, K.T.: Observations and issues in the application of cognitive workload modelling for decision making in complex time-critical environments. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 77–89. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_5

    Chapter  Google Scholar 

  49. Su, J., Luz, S.: Predicting cognitive load levels from speech data. In: Esposito, A., et al. (eds.) Recent Advances in Nonlinear Speech Processing. SIST, vol. 48, pp. 255–263. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-28109-4_26

    Chapter  Google Scholar 

  50. Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996)

    Article  Google Scholar 

  51. Walter, C., Cierniak, G., Gerjets, P., Rosenstiel, W., Bogdan, M.: Classifying mental states with machine learning algorithms using alpha activity decline. In: 2011 Proceedings of 19th European Symposium on Artificial Neural Networks, ESANN 2011, Bruges, Belgium, April 27–29 (2011). https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2011-35.pdf

  52. Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50(3), 449–455 (2008)

    Article  Google Scholar 

  53. Wickens, C.D.: Mental workload: assessment, prediction and consequences. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 18–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_2

    Chapter  Google Scholar 

  54. Wolpert, D.H.: The supervised learning no-free-lunch theorems. In: Roy, R., Köppen, M., Ovaska, S., Furuhashi, T., Hoffmann, F. (eds.) Soft Computing and Industry, pp. 25–42. Springer, London (2002). https://doi.org/10.1007/978-1-4471-0123-9_3

    Chapter  Google Scholar 

  55. Yoshida, Y., Ohwada, H., Mizoguchi, F., Iwasaki, H.: Classifying cognitive load and driving situation with machine learning. Int. J. Mach. Learn. Comput. 4(3), 210–215 (2014)

    Article  Google Scholar 

  56. Young, M.S., Brookhuis, K.A., Wickens, C.D., Hancock, P.A.: State of science: mental workload in ergonomics. Ergonomics 58(1), 1–17 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, Technological University Dublin, Dublin, Republic of Ireland

    Karim Moustafa & Luca Longo

  2. ADAPT, The Global Centre of Excellence for Digital Content Technology, Dublin, Republic of Ireland

    Karim Moustafa & Luca Longo

Authors
  1. Karim Moustafa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luca Longo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luca Longo .

Editor information

Editors and Affiliations

  1. Dublin Institute of Technology, Dublin, Ireland

    Luca Longo

  2. Dublin Institute of Technology, Dublin, Ireland

    M. Chiara Leva

Appendix

Appendix

Fig. 7.
figure 7

Scatterplots of the overall perception of mental workload reported by subjects (OP-MWL) (x-axis) and the prediction of the induced models (y-axis) for the NASA-TLX (left) and the Workload Profile (right) grouped by fold

Full size image
Fig. 8.
figure 8

Scatterplots of the overall perception of mental workload (x-axis), as reported by subjects and the prediction of the induced models (y-axis) for the 5 models produced by the regression algorithms (Extra trees: col 1; KNN: col 2; SVR: col 3; NB: col 4) employing the features of the NASA Task Load Index

Full size image
Fig. 9.
figure 9

Scatterplots of the overall perception of mental workload (x-axis), as reported by subjects and the prediction of the induced models (y-axis) for the 5 models produced by the regression algorithms (Extra trees: col 1; KNN: col 2; SVR: col 3; NB: col 4) employing the features of the Workload Profile

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Moustafa, K., Longo, L. (2019). Analysing the Impact of Machine Learning to Model Subjective Mental Workload: A Case Study in Third-Level Education. In: Longo, L., Leva, M. (eds) Human Mental Workload: Models and Applications. H-WORKLOAD 2018. Communications in Computer and Information Science, vol 1012. Springer, Cham. https://doi.org/10.1007/978-3-030-14273-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-14273-5_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-14272-8

  • Online ISBN: 978-3-030-14273-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation