Skip to main content
SpringerLink
Log in
Book cover

International Conference on Computer Supported Education

CSEDU 2018: Computer Supported Education pp 384–413Cite as

An Evaluation of the Reliability, Validity and Sensitivity of Three Human Mental Workload Measures Under Different Instructional Conditions in Third-Level Education

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1022))

Abstract

Although Cognitive Load Theory (CLT) has been researched for many years, it has been criticised for its theoretical clarity and its methodological approach. A crucial issue is the measurement of three types of cognitive load conceived in the theory, and the assessment of overall human cognitive load during learning tasks. This research study is motivated by these issues and it aims to investigate the reliability, validity and sensitivity of three existing self-reporting mental workload instruments, mainly used in Ergonomics, when applied to Education and in particular to the field of Teaching and Learning. A primary research study has been designed and performed in a typical third-level classroom in Computer Science, and the self-reporting mental workload instruments employed are the NASA Task Load Index, the Workload Profile and the Rating Scale Mental Effort. Three instructional design conditions have been designed and employed for the above purposes. The first design condition followed the traditional explicit instruction paradigm whereby a lecturer delivers instructional material mainly using a one-way approach with almost no interactions with students. The second design condition was inspired by the Cognitive Theory of Multimedia Learning whereby the same content, delivered under the first condition, was converted in a multimedia video by following a set of its design principles. The third design condition was an extension of the second condition whereby an inquiry activity was executed after the delivery of the second condition. The empirical evidence gathered in this study suggests that the three selected mental workload measures are highly reliable. Their moderate face validity is in line with the results obtained so far within Ergonomics emphasising and confirming the difficulty in creating optimally valid measures of mental workload. However, the sensitivity of these measures, as achieved in this study, is low, indicating how the three instructional design conditions, as conceived and implemented, do not impose significantly different mental workload levels on learners.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Artino Jr., A.R.: Cognitive load theory and the role of learner experience: an abbreviated review for educational practitioners. AACE J. 16(4), 425–439 (2008)

    Google Scholar 

  2. Ayres, P.: Using subjective measures to detect variations of intrinsic cognitive load within problems. Learn. Instr. 16(5), 389–400 (2006)

    Article  Google Scholar 

  3. Baddeley, A., Hitch, G.: Working Memory, vol. 8, pp. 47–90. Academic Press, Cambridge (1974)

    Google Scholar 

  4. Bleazby, J.: Autonomy, democratic community, and citizenship in philosophy for children: dewey and philosophy for children’s rejection of the individual/community dualism. Anal. Teach. 26(1), 30–52 (2006)

    Google Scholar 

  5. Brookhuis, K.A., de Waard, D.: Monitoring drivers’ mental workload in driving simulators using physiological measures. Accid. Anal. Prev. 42(3), 898–903 (2010)

    Article  Google Scholar 

  6. Brunken, R., Plass, J.L., Leutner, D.: Direct measurement of cognitive load in multimedia learning. Educ. Psychol. 38(1), 53–61 (2003)

    Article  Google Scholar 

  7. Cain, B.: A review of the mental workload literature. Technical report. Defence Research and Development Canada Toronto (2007)

    Google Scholar 

  8. Chandler, P., Sweller, J.: Cognitive load theory and the format of instruction. Cogn. Instr. 8(4), 293–332 (1991)

    Article  Google Scholar 

  9. Cierniak, G., Scheiter, K., Gerjets, P.: Explaining the split-attention effect: is the reduction of extraneous cognitive load accompanied by an increase in germane cognitive load? Comput. Hum. Behav. 25(2), 315–324 (2009)

    Article  Google Scholar 

  10. De Jong, T.: Cognitive load theory, educational research, and instructional design: some food for thought. Instr. Sci. 38(2), 105–134 (2010)

    Article  Google Scholar 

  11. Debue, N., van de Leemput, C.: What does germane load mean? An empirical contribution to the cognitive load theory. Front. Psychol. 5, 1099 (2014)

    Article  Google Scholar 

  12. DeLeeuw, K.E., Mayer, R.E.: A comparison of three measures of cognitive load: evidence for separable measures of intrinsic, extraneous, and germane load. J. Educ. Psychol. 100(1), 223 (2008)

    Article  Google Scholar 

  13. Dewey, J.: The Child and the Curriculum. University of Chicago Press, Chicago (1902). No. 5

    Google Scholar 

  14. Dixon, P.: From research to theory to practice: commentary on Chandler and Sweller. Cogn. Instr. 8(4), 343–350 (1991)

    Article  Google Scholar 

  15. Gerjets, P., Scheiter, K., Cierniak, G.: The scientific value of cognitive load theory: a research agenda based on the structuralist view of theories. Educ. Psychol. Rev. 21(1), 43–54 (2009)

    Article  Google Scholar 

  16. Goldman, S.R.: On the derivation of instructional applications from cognitive theories: commentary on Chandler and Sweller. Cogn. Instr. 8(4), 333–342 (1991)

    Article  Google Scholar 

  17. Gwizdka, J.: Distribution of cognitive load in web search. J. Am. Soc. Inf. Sci. Technol. 61(11), 2167–2187 (2010)

    Article  Google Scholar 

  18. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908. Sage Journals, San Francisco (2006)

    Article  Google Scholar 

  19. Kirschner, P.A.: Cognitive load theory: implications of cognitive load theory on the design of learning. Learn. Instr. 12(1), 1–10 (2002)

    Article  Google Scholar 

  20. Lipman, M., Sharp, A.M., Oscanyan, F.S.: Philosophy in the Classroom. Temple University Press, Philadelphia (1980)

    Google Scholar 

  21. Lipman, M.: Thinking in Education. Cambridge University Press, Cambridge (2003)

    Book  Google Scholar 

  22. Longo, L.: Human-computer interaction and human mental workload: assessing cognitive engagement in the world wide web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23768-3_43

    Chapter  Google Scholar 

  23. Longo, L.: Formalising human mental workload as non-monotonic concept for adaptive and personalised web-design. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 369–373. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-31454-4_38

    Chapter  Google Scholar 

  24. Longo, L.: Formalising human mental workload as a defeasible computational concept. Ph.D. thesis. Trinity College Dublin (2014)

    Google Scholar 

  25. Longo, L.: A defeasible reasoning framework for human mental workload representation and assessment. Behav. Inf. Technol. 34(8), 758–786 (2015)

    Article  Google Scholar 

  26. Longo, L.: Designing medical interactive systems via assessment of human mental workload. In: International Symposium on Computer-Based Medical Systems, pp. 364–365 (2015)

    Google Scholar 

  27. Longo, L.: Mental workload in medicine: foundations, applications, open problems, challenges and future perspectives. In: 2016 IEEE 29th International Symposium on Computer-Based Medical Systems (CBMS), pp. 106–111, June 2016

    Google Scholar 

  28. Longo, L.: Subjective usability, mental workload assessments and their impact on objective human performance. In: Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D.K., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10514, pp. 202–223. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67684-5_13

    Chapter  Google Scholar 

  29. Longo, L.: On the reliability, validity and sensitivity of three mental workload assessment techniques for the evaluation of instructional designs: a case study in a third-level course. In: Proceedings of the 10th International Conference on Computer Supported Education, CSEDU 2018, Funchal, Madeira, Portugal, 15–17 March 2018, vol. 2, pp. 166–178 (2018)

    Google Scholar 

  30. Longo, L., Barrett, S.: Cognitive effort for multi-agent systems. In: Yao, Y., Sun, R., Poggio, T., Liu, J., Zhong, N., Huang, J. (eds.) BI 2010. LNCS, vol. 6334, pp. 55–66. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15314-3_6

    Chapter  Google Scholar 

  31. Longo, L., Barrett, S.: A computational analysis of cognitive effort. In: Nguyen, N.T., Le, M.T., Świątek, J. (eds.) ACIIDS 2010. LNCS, vol. 5991, pp. 65–74. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12101-2_8

    Chapter  Google Scholar 

  32. Longo, L., Dondio, P.: On the relationship between perception of usability and subjective mental workload of web interfaces. In: IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, WI-IAT 2015, Singapore, 6–9 December, vol. I, pp. 345–352 (2015)

    Google Scholar 

  33. Longo, L., Kane, B., Hederman, L.: Argumentation theory in health care. In: 25th International Symposium on Computer-Based Medical Systems, Rome, Italy, pp. 1–6. IEEE (2012)

    Google Scholar 

  34. Longo, L., Rusconi, F., Noce, L., Barrett, S.: The importance of human mental workload in web-design. In: 8th International Conference on Web Information Systems and Technologies, Porto, Portugal, pp. 403–409. SciTePress, April 2012

    Google Scholar 

  35. Mayer, R.: Using multimedia for e-learning. J. Comput. Assist. Learn. 33(5), 403–423 (2017). jCAL-16-266.R1

    Article  Google Scholar 

  36. Mayer, R.E.: Multimedia learning. Psychol. Learn. Motiv. 41, 85–139 (2002)

    Article  Google Scholar 

  37. Mayer, R.E.: The Cambridge Handbook of Multimedia Learning. Cambridge University Press, Cambridge (2005)

    Book  Google Scholar 

  38. Mayer, R.E.: Multimedia Learning. Cambridge University Press, Cambridge (2009)

    Book  Google Scholar 

  39. Miller, G.A.: The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychol. Rev. 63(2), 81–97 (1956)

    Article  Google Scholar 

  40. Mousavi, S., Low, R., Sweller, J.: Reducing cognitive load by mixing auditory and visual presentation modes. J. Educ. Psychol. 87(2), 319–334 (1995)

    Article  Google Scholar 

  41. Moustafa, K., Luz, S., Longo, L.: Assessment of mental workload: a comparison of machine learning methods and subjective assessment techniques. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 30–50. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_3

    Chapter  Google Scholar 

  42. Orru, G., Gobbo, F., O’Sullivan, D., Longo, L.: An investigation of the impact of a social constructivist teaching approach, based on trigger questions, through measures of mental workload and efficiency. In: Proceedings of the 10th International Conference on Computer Supported Education, CSEDU 2018, Funchal, Madeira, Portugal, 15–17 March 2018, vol. 2, pp. 292–302 (2018)

    Google Scholar 

  43. Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)

    Article  Google Scholar 

  44. Paas, F., Van Merrienboer, J.J.G.: The efficiency of instructional conditions: an approach to combine mental effort and performance measures. Hum. Factors: J. Hum. Factors Ergon. Soc. 35(4), 737–743 (1993)

    Article  Google Scholar 

  45. Paas, F.G., Van Merriënboer, J.J., Adam, J.J.: Measurement of cognitive load in instructional research. Percept. Mot. Skills 79(1), 419–430 (1994)

    Article  Google Scholar 

  46. Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford Psychology Series. Oxford University Press, Oxford (1990)

    Book  Google Scholar 

  47. Popper, K.: Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge, Abingdon (2014)

    Book  Google Scholar 

  48. Reid, G.B., Nygren, T.E.: The subjective workload assessment technique: a scaling procedure for measuring mental workload, Chap. 8. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, Advances in Psychology, vol. 52, pp. 185–218. North-Holland (1988)

    Google Scholar 

  49. Rizzo, L., Dondio, P., Delany, S.J., Longo, L.: Modeling mental workload via rule-based expert system: a comparison with NASA-TLX and workload profile. In: Iliadis, L., Maglogiannis, I. (eds.) AIAI 2016. IAICT, vol. 475, pp. 215–229. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-44944-9_19

    Chapter  Google Scholar 

  50. Rizzo, L., Longo, L.: Representing and inferring mental workload via defeasible reasoning: a comparison with the NASA task load index and the workload profile. In: Proceedings of the 1st Workshop on Advances in Argumentation in Artificial Intelligence Co-Located with XVI International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017), Bari, Italy, 16–17 November 2017, pp. 126–140 (2017)

    Google Scholar 

  51. Roscoe, A.H., Ellis, G.A.: A subjective rating scale for assessing pilot workload in flight: a decade of practical use. Technical report TR 90019. Royal Aerospace Establishment, March 1990

    Google Scholar 

  52. Rubio, S., Diaz, E., Martin, J., Puente, J.M.: Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53(1), 61–86 (2004)

    Article  Google Scholar 

  53. Satiro, A.: Jugar a pensar con mitos: este libro forma parte del Proyecto Noria y acompaña al libro para niños de 8–9 anos: Juanita y los mitos. Octaedro (2006)

    Google Scholar 

  54. Schnotz, W., Kürschner, C.: A reconsideration of cognitive load theory. Educ. Psychol. Rev. 19(4), 469–508 (2007)

    Article  Google Scholar 

  55. Seufert, T., Jänen, I., Brünken, R.: The impact of intrinsic cognitive load on the effectiveness of graphical help for coherence formation. Comput. Hum. Behav. 23(3), 1055–1071 (2007)

    Article  Google Scholar 

  56. Sweller, J., Van Merrienboer, J., Paas, F.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10(3), 251–296 (1998)

    Article  Google Scholar 

  57. Sweller, J.: Cognitive load theory, learning difficulty, and instructional design. Learn. Instruct. 4(4), 295–312 (1994)

    Article  Google Scholar 

  58. Sweller, J.: Element interactivity and intrinsic, extraneous, and germane cognitive load. Educ. Psychol. Rev. 22(2), 123–138 (2010)

    Article  Google Scholar 

  59. Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996)

    Article  Google Scholar 

  60. Vidulich, M.A., Ward, G.F., Schueren, J.: Using the subjective workload dominance (SWORD) technique for projective workload assessment. Hum. Factors Soc. 33(6), 677–691 (1991)

    Article  Google Scholar 

  61. Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50(2), 449–454 (2008)

    Article  Google Scholar 

  62. Wilson, G.F., Eggemeier, T.F.: Mental workload measurement, Chap. 167. In: Karwowski, W. (ed.) International Encyclopedia of Ergonomics and Human Factors, 2nd edn, vol. 1. Taylor and Francis (2006)

    Google Scholar 

  63. Xie, B., Salvendy, G.: Review and reappraisal of modelling and predicting mental workload in single and multi-task environments. Work Stress 14(1), 74–99 (2000)

    Article  Google Scholar 

  64. Young, M.S., Brookhuis, K.A., Wickens, C.D., Hancock, P.A.: State of science: mental workload in ergonomics. Ergonomics 58(1), 1–17 (2015)

    Article  Google Scholar 

  65. Young, M.S., Stanton, N.A.: Mental workload: theory, measurement, and application. In: Karwowski, W. (ed.) Encyclopedia of Ergonomics and Human Factors, 2nd edn, vol. 1, pp. 818–821. Taylor & Francis (2006)

    Google Scholar 

  66. Zijlstra, F.R.H.: Efficiency in work behaviour. Doctoral thesis. Delft University, The Netherlands (1993)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing, College of Sciences and Health, Dublin Institute of Technology, Dublin, Republic of Ireland

    Luca Longo & Giuliano Orru

  2. ADAPT: The Global Centre of Excellence for Digital Content and Media Innovation, Dublin, Republic of Ireland

    Luca Longo

Authors
  1. Luca Longo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giuliano Orru
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luca Longo .

Editor information

Editors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce M. McLaren

  2. MIT, Cambridge, MA, USA

    Rob Reilly

  3. University of Denver, Denver, CO, USA

    Susan Zvacek

  4. University of Ulster, Newtownabbey, UK

    James Uhomoibhi

Appendix

Appendix

(See Tables 12, 13, 14, 15, 16, 17 and Figs. 7, 8, 9).

Table 12. The Rating Scale Mental Effort.
Full size table
Table 13. The NASA Task Load Index (NASA-TLX).
Full size table
Table 14. The Workload Profile (WP).
Full size table
Table 15. Design of the instructional condition 2 using the principles of Cognitive Theory of Multimedia Learning and its differences with condition 1 grouped by load type.
Full size table
Table 16. Dialogical activity set for the third design condition inspired by the Community Inquiry paradigm.
Full size table
Table 17. Question and scale designed for investigating the face validity of the mental workload assessment measures.
Full size table
Fig. 7.
figure 7

Density plots of the distributions of the mental workload scores by topic (T1–T4) and design condition (DC1-3) for the NASA Task Load Index

Full size image
Fig. 8.
figure 8

Density plots of the distributions of the mental workload scores by topic (T1–T4) and design condition (DC1-3) for the WorkloadProfile

Full size image
Fig. 9.
figure 9

Density plots of the distributions of the mental workload scores by topic (T1–T4) and design condition (DC1-3) for the Rating Scale Mental Effort.

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Longo, L., Orru, G. (2019). An Evaluation of the Reliability, Validity and Sensitivity of Three Human Mental Workload Measures Under Different Instructional Conditions in Third-Level Education. In: McLaren, B., Reilly, R., Zvacek, S., Uhomoibhi, J. (eds) Computer Supported Education. CSEDU 2018. Communications in Computer and Information Science, vol 1022. Springer, Cham. https://doi.org/10.1007/978-3-030-21151-6_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-21151-6_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-21150-9

  • Online ISBN: 978-3-030-21151-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation