Mental Effort, Workload, Time on Task, and Certainty: Beyond Linear Models
Self-rated mental effort has been and continues to be the most widely used measure of cognitive load. This single-item measure is often used as a predictor variable in linear models for predicting performance or some other response variable. While an advantage of linear models is that they are fairly easy to understand, they fall short when the relations between variables of interest are clearly non-linear. The current study focused on a reanalysis of data of four recently published studies in which self-rated mental effort was measured repeatedly along with either workload, time on task—viewing time in one study, response time in another study—or self-rated certainty of correct task performance. A core outcome of the reanalysis is a preference towards mental effort as a non-linear instead of linear predictor in three of the four studies: quadratic (workload), quadratic (response time), and cubic (certainty); only in the case of viewing time, mental effort as a linear predictor was found to be the preferred solution. Implications of this finding for the interpretation of self-rated mental effort, indicators of cognitive load involving time on task, and metacognition-related response variables such as self-rated certainty are discussed.
KeywordsMental effort Workload Time on task Certainty Non-linear models
The authors would like to thank all the authors from the original studies for designing, carrying out, and reporting of these high-quality studies, their willingness to share their data, and their great care for detail not only in reporting in their articles but in their data management and responding to our questions as well. In our view, the communication for this study and the authors’ willingness to share constitutes a fascinating example of how data sharing can help us, together, as a field, to enrich the discourse about educational research and practice.
- Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & F. Csaki (Eds.), Proceedings of the Second International Symposium on Information Theory (pp. 267–281). Budapest: Academiai Kiado.Google Scholar
- Blissett, S., Sibbald, M., Kok, E. M., Van Merriënboer, J. J. G. (2018). Optimizing self-regulation of performance: Is mental effort a cue? Advances in Health Sciences Education, online ahead of print. https://doi.org/10.1007/s10459-018-9838-x.
- Brünken, R., Plass, J. L., & Leutner, D. (2004). Assessment of cognitive load in multimedia learning with dual-task methodology: auditory load and modality effects. Instructional Science, 32(1–2), 115–132. https://doi.org/10.1023/B:TRUC.0000021812.96911.c5.CrossRefGoogle Scholar
- Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: a practical information-theoretic approach. New York: Springer.Google Scholar
- Callicott, J. H., Mattay, V. S., Bertolino, A., Finn, K., Coppola, R., Frank, J. A., Goldberg, T. E., & Weinberger, D. R. (1999). Physiological characteristics of capacity constraints in working memory as revealed by functional MRI. Cerebral Cortext, 9(1), 20–26. https://doi.org/10.1093/cercor/9.1.20.CrossRefGoogle Scholar
- Clark, R. E., & Elen, J. (2006). When less is more: research and theory insights about instruction for complex learning. In J. Elen & R. Clark (Eds.), Handling complexity in learning environments: research and theory (pp. 283–297). Oxford: Elsevier Science Limited.Google Scholar
- Cox, D. R., & Snell, E. J. (1989). Analysis of binary data (2nd ed.). New York: Chapman & Hall.Google Scholar
- Fraser, K., & McLaughlin, K. (2018). Temporal pattern of emotions and cognitive load during simulation training and debriefing. Medical Teacher, online ahead of print, https://doi.org/10.1080/0142159X.2018.1459531.
- Giraud, C. (2015). Introduction to high-dimensional statistics. Boca Raton: CRC.Google Scholar
- Hart, S. G., & Staveland, L. E. (1998). Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In P. A. Hancock & N. Meshkati (Eds.), Human mental workload (pp. 139–183). Amsterdam: North Holland Press.Google Scholar
- Hayes, T., & McArdle, J. J. (2017). Should we impute or should we weight? Examining the performance of two CART-based techniques for addressing missing data in small sample research with nonnormal variables. Computational Statistics and Data Analysis, 115, 35–52. https://doi.org/10.1016/j.csda.2017.05.006.CrossRefGoogle Scholar
- IBM Corporation. (2017). SPSS version 25. Retrieved from: https://www-01.ibm.com/support/docview.wss?uid=swg24043678. Accessed 7 Dec 2018.
- Jamovi Project. (2018). Jamovi version 0.9.5.0. Retrieved from: https://www.jamovi.org. Accessed 7 Dec 2018.
- Leppink, J., & Van Merriënboer, J. J. G. (2015). The beast of aggregating cognitive load measures in technology-based learning. Educational Technology & Society, 18(4), 230–245.Google Scholar
- Martin, L. J., Turnqvist, A., Groot, B., Huang, S. Y. M., Kok, E. M., Thoma, B., & Van Merriënboer, J. J. G. (2018). Exploring the role of infographics for summarizing medical literature. Health Professions Education, online ahead of print. https://doi.org/10.1016/j.hpe.2018.03.005.
- McFadden, D. (1974). Conditional logit analysis of qualitative choice behaviour. In P. Zarembka (Ed.), Frontiers in econometrics. Berkeley: Academic Press.Google Scholar
- McQuarrie, A. D. R., & Tsai, C. L. (1998). Regression and time series model selection. World Scientific.Google Scholar
- Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R 2 from generalized mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012-00261.x.
- Nakagawa, S., Johnson, P. C. D., & Schielzeth, H. (2017). The coefficient of determination R 2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of the Royal Society Interface, 14(134), 1–11.Google Scholar
- Park, B., & Brünken, R. (2014). The rhythm method: a new method for measuring cognitive load—an experimental dual-task study. Applied Cognitive Psychology, 29(2), 232–243. https://doi.org/10.1002/acp.3100.
- Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581
- Schmeck, A., Opfermann, M., Van Gog, T., Paas, F., & Leutner, D. (2015). Measuring cognitive load with subjective rating scales during problem solving: differences between immediate and delayed ratings. Instructional Science, 43(1), 93–114. https://doi.org/10.1007/s11251-014-9328-3.CrossRefGoogle Scholar
- Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hederos, K., Heene, M., Hofelich Mohr, A. J., Högden, F., Hui, K., Johannesson, M., Kalodimos, J., Kaszubowski, E., Kennedy, D. M., Lei, R., Lindsay, T. A., Liverani, S., Madan, C. R., Molden, D., Molleman, E., Morey, R. D., Mulder, L. B., Nijstad, B. R., Pope, N. G., Pope, B., Prenoveau, J. M., Rink, F., Robusto, E., Roderique, H., Sandberg, A., Schlüter, E., Schönbrodt, F. D., Sherman, M. F., Sommer, S. A., Sotak, K., Spain, S., Spörlein, C., Stafford, T., Stefanutti, L., Tauber, S., Ullrich, J., Vianello, M., Wagenmakers, E. J., Witkowiak, M., Yoon, S., & Nosek, B. A. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646.CrossRefGoogle Scholar
- Szulewski, A., Roth, N., & Howes, D. W. (2015). The use of task-evoked pupillary response as an objective measure of cognitive load in novices and trained physicians: a new tool for the assessment of expertise. Academic Medicine, 90(7), 981–987. https://doi.org/10.1097/ACM.0000000000000677.CrossRefGoogle Scholar
- Szulewski, A., Gegenfurtner, A., Howes, D. W., Sivilotti, M. L. A., & Van Merriënboer, J. J. G. (2017). Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Advances in Health Sciences Education, 22(4), 951–968. https://doi.org/10.1007/s10459-016-9725-2.CrossRefGoogle Scholar
- Tan, F. E. S. (2010). Best practices in analysis of longitudinal data: a multilevel approach. In J. W. Osborne (Ed.), Best practices in quantitative methods (pp. 451–470). London: Sage.Google Scholar
- Verbeke, G., & Molenberghs, G. (2000). Linear mixed models for longitudinal data. New York: Springer.Google Scholar