Mental Effort, Workload, Time on Task, and Certainty: Beyond Linear Models

  • Jimmie LeppinkEmail author
  • Patricia Pérez-Fuster


Self-rated mental effort has been and continues to be the most widely used measure of cognitive load. This single-item measure is often used as a predictor variable in linear models for predicting performance or some other response variable. While an advantage of linear models is that they are fairly easy to understand, they fall short when the relations between variables of interest are clearly non-linear. The current study focused on a reanalysis of data of four recently published studies in which self-rated mental effort was measured repeatedly along with either workload, time on task—viewing time in one study, response time in another study—or self-rated certainty of correct task performance. A core outcome of the reanalysis is a preference towards mental effort as a non-linear instead of linear predictor in three of the four studies: quadratic (workload), quadratic (response time), and cubic (certainty); only in the case of viewing time, mental effort as a linear predictor was found to be the preferred solution. Implications of this finding for the interpretation of self-rated mental effort, indicators of cognitive load involving time on task, and metacognition-related response variables such as self-rated certainty are discussed.


Mental effort Workload Time on task Certainty Non-linear models 



The authors would like to thank all the authors from the original studies for designing, carrying out, and reporting of these high-quality studies, their willingness to share their data, and their great care for detail not only in reporting in their articles but in their data management and responding to our questions as well. In our view, the communication for this study and the authors’ willingness to share constitutes a fascinating example of how data sharing can help us, together, as a field, to enrich the discourse about educational research and practice.


  1. Aasman, J., Mulder, G., & Mulder, L. (1987). Operator effort and the measurement of heart-rate variability. Human Factors, 29(2), 161–170. Scholar
  2. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Csaki (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267–281). Budapest: Academiai Kiado.Google Scholar
  3. Aldekhyl, S., Cavalcanti, R. B., & Naismith, L. M. (2018). Cognitive load predicts point-of-care ultrasound simulator performance. Perspectives on Medical Education, 7(1), 23–32. Scholar
  4. Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276–292. Scholar
  5. Blissett, S., Sibbald, M., Kok, E. M., Van Merriënboer, J. J. G. (2018). Optimizing self-regulation of performance: Is mental effort a cue? Advances in Health Sciences Education, online ahead of print.
  6. Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: auditory load and modality effects. Instructional Science, 32(1–2), 115–132. Scholar
  7. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: a practical information-theoretic approach. New York: Springer.Google Scholar
  8. Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods & Research, 33(2), 261–304.CrossRefGoogle Scholar
  9. Callicott, J. H., Mattay, V. S., Bertolino, A., Finn, K., Coppola, R., Frank, J. A., Goldberg, T. E., & Weinberger, D. R. (1999). Physiological characteristics of capacity constraints in working memory as revealed by functional MRI. Cerebral Cortext, 9(1), 20–26. Scholar
  10. Claeskens, G., & Hjort, N. L. (2008). Model selection and model averaging. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  11. Clark, R. E. (1998). Motivating performance: part I—diagnosing and solving motivation problems. Performance Improvement, 37(8), 39–47. Scholar
  12. Clark, R. E., & Elen, J. (2006). When less is more: research and theory insights about instruction for complex learning. In J. Elen & R. Clark (Eds.), Handling complexity in learning environments: research and theory (pp. 283–297). Oxford: Elsevier Science Limited.Google Scholar
  13. Cox, D. R., & Snell, E. J. (1989). Analysis of binary data (2nd ed.). New York: Chapman & Hall.Google Scholar
  14. Cragg, J. G., & Uhler, R. S. (1970). The demand for automobiles. The Canadian Journal of Economics, 3(3), 386–406. Scholar
  15. De Bruin, A. B. H., Dunlosky, J., & Cavalcanti, R. B. (2017). Monitoring and regulation of learning in medical education: the need for predictive cues. Medical Education, 51(6), 575–584. Scholar
  16. Fraser, K., & McLaughlin, K. (2018). Temporal pattern of emotions and cognitive load during simulation training and debriefing. Medical Teacher, online ahead of print,
  17. Giraud, C. (2015). Introduction to high-dimensional statistics. Boca Raton: CRC.Google Scholar
  18. Hart, S. G., & Staveland, L. E. (1998). Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North Holland Press.Google Scholar
  19. Hayes, T., & McArdle, J. J. (2017). Should we impute or should we weight? Examining the performance of two CART-based techniques for addressing missing data in small sample research with nonnormal variables. Computational Statistics and Data Analysis, 115, 35–52. Scholar
  20. IBM Corporation. (2017). SPSS version 25. Retrieved from: Accessed 7 Dec 2018.
  21. Jamovi Project. (2018). Jamovi version Retrieved from: Accessed 7 Dec 2018.
  22. Kalyuga, S., & Singh, A. M. (2016). Rethinking the boundaries of cognitive load theory in complex learning. Educational Psychology Review, 28(4), 831–852. Scholar
  23. Lee, C. B., Hanham, J., & Leppink, J. (2019). Instructional design principles for high-stakes problem-solving environments. Singapore: Springer. Scholar
  24. Leppink, J., & Van den Heuvel, A. (2015). The evolution of cognitive load theory and its application to medical education. Perspectives on Medical Education, 4(3), 119–127. Scholar
  25. Leppink, J., & Van Merriënboer, J. J. G. (2015). The beast of aggregating cognitive load measures in technology-based learning. Educational Technology & Society, 18(4), 230–245.Google Scholar
  26. Maddala, G. S. (1983). Limited dependent and qualitative variables in econometrics. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  27. Martin, L. J., Turnqvist, A., Groot, B., Huang, S. Y. M., Kok, E. M., Thoma, B., & Van Merriënboer, J. J. G. (2018). Exploring the role of infographics for summarizing medical literature. Health Professions Education, online ahead of print.
  28. McFadden, D. (1974). Conditional logit analysis of qualitative choice behaviour. In P. Zarembka (Ed.), Frontiers in econometrics. Berkeley: Academic Press.Google Scholar
  29. McQuarrie, A. D. R., & Tsai, C. L. (1998). Regression and time series model selection. World Scientific.Google Scholar
  30. Nagelkerke, N. J. D. (1991). A note on a general definition of the coefficient of determination. Biometrika, 78(3), 691–692.CrossRefGoogle Scholar
  31. Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R 2 from generalized mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142.
  32. Nakagawa, S., Johnson, P. C. D., & Schielzeth, H. (2017). The coefficient of determination R 2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of the Royal Society Interface, 14(134), 1–11.Google Scholar
  33. Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skills in statistics: a cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434.CrossRefGoogle Scholar
  34. Park, B., & Brünken, R. (2014). The rhythm method: a new method for measuring cognitive load—an experimental dual-task study. Applied Cognitive Psychology, 29(2), 232–243.
  35. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592.
  36. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. New York: Wiley.CrossRefGoogle Scholar
  37. Salomon, G. (1983). The differential investment of mental effort in learning from different sources. Educational Psychologist, 18(1), 42–50. Scholar
  38. Salomon, G. (1984). Television is “easy” and print is “tough”: the differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology, 76(4), 647–658. Scholar
  39. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall.CrossRefGoogle Scholar
  40. Schmeck, A., Opfermann, M., Van Gog, T., Paas, F., & Leutner, D. (2015). Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings. Instructional Science, 43(1), 93–114. Scholar
  41. Schwarz, G. (1978). Estimating the dimensions of a model. Annals of Statistics, 6(2), 461–464.CrossRefGoogle Scholar
  42. Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hederos, K., Heene, M., Hofelich Mohr, A. J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D. M., Lei, R., Lindsay, T. A., Liverani, S., Madan, C. R., Molden, D., Molleman, E., Morey, R. D., Mulder, L. B., Nijstad, B. R., Pope, N. G., Pope, B., Prenoveau, J. M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F. D., Sherman, M. F., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E. J., Witkowiak, M., Yoon, S., & Nosek, B. A. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. Scholar
  43. Sweller, J. (2018). Measuring cognitive load. Perspectives on Medical Education, 7(1), 1–2. Scholar
  44. Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.CrossRefGoogle Scholar
  45. Szulewski, A., Roth, N., & Howes, D. W. (2015). The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Academic Medicine, 90(7), 981–987. Scholar
  46. Szulewski, A., Gegenfurtner, A., Howes, D. W., Sivilotti, M. L. A., & Van Merriënboer, J. J. G. (2017). Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Advances in Health Sciences Education, 22(4), 951–968. Scholar
  47. Tan, F. E. S. (2010). Best practices in analysis of longitudinal data: a multilevel approach. In J. W. Osborne (Ed.), Best practices in quantitative methods (pp. 451–470). London: Sage.Google Scholar
  48. Tjur, T. (2009). Coefficients of determination in logistic regression models—a new proposal: the coefficient of discrimination. The American Statistician, 63(4), 366–372. Scholar
  49. Van Buuren, S. (2012). Flexible imputation of missing data. New York: Chapman & Hall.CrossRefGoogle Scholar
  50. Van Gog, T., Kirschner, F., Kester, L., & Paas, F. (2012). Timing and frequency of mental effort measurement: evidence in favour of repeated measures. Applied Cognitive Psychology, 26(6), 833–839. Scholar
  51. Verbeke, G., & Molenberghs, G. (2000). Linear mixed models for longitudinal data. New York: Springer.Google Scholar
  52. Weakliem, D. (1999). A critique of the Bayesian information criterion for model selection. Sociological Methods & Research, 27(3), 359–397. Scholar
  53. Yuan, K. H., Yang-Wallentin, F., & Bentler, P. M. (2012). ML versus MI for missing data with violation of distribution conditions. Sociological Methods & Research, 41(4), 598–629. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.University of YorkYorkUK
  2. 2.Maastricht UniversityMaastrichtThe Netherlands
  3. 3.University of ValenciaValenciaSpain

Personalised recommendations