Skip to main content
SpringerLink
Log in

Does the Modality Effect Exist? and if So, Which Modality Effect?

  • Published:
Journal of Psycholinguistic Research Aims and scope Submit manuscript

Abstract

The modality effect is a central issue in multimedia learning [see Mayer (Cambridge University Press, 2005a), for a review]. Sweller’s Cognitive Load Theory (CLT), for example, presumes that an illustrated text is better understood when presented visually rather than orally. The predictive power of CLT lies in how it links in to Baddeley’s (1986) model of working memory and Penney’s (Mem Cognit 17:398–442, 1989) Separate-Streams Hypothesis. Ginns’s (Learn Instr 4:313–331, 2005) recent meta-analysis also supports the modality effect (d = 0.72, based on 43 independent effects). This article replicates the meta-analysis of the modality effect based on 86 independent effects (with within-study subgroups as the unit of analysis and with mean of the outcomes as the dependent measure), with results showing a reduction of the overall effect size by almost half (d = 0.38), and even more when Duval and Tweedie’s Trim and Fill method is used to correct publication bias (d = 0.20). This article also widens the scope of the analysis of moderator variables (e.g. Pace of presentation, Type of visualization, Research group) as well as their potentially confounded effects. Finally, it is argued that, for theoretical reasons, the so-called modality effect cannot be based on Penney’s or Baddeley’s theories and must be explained in a different way.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • * = References with an asterisk are considered in the present meta-analysis.

  • *Abel R. R., Kulhavy R. W. (1986) Maps, mode of text presentation, and children’s prose learning. American Educational Research Journal 23(2): 263–274

    Google Scholar 

  • Altmann G. T. M., Kamide Y. (1999) Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73: 247–264

    Article  PubMed  Google Scholar 

  • Altmann G. T. M., Kamide Y. (2007) The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language 57(4): 502–518

    Article  Google Scholar 

  • *Atkinson R. K. (2002) Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology 94: 416–427

    Article  Google Scholar 

  • Baddeley A. D. (1986) Working memory. Oxford University Press, Oxford

    Google Scholar 

  • Baddeley A. D. (1992) Working memory. Science 255: 556–559

    Article  PubMed  Google Scholar 

  • Borenstein M., Hedges L. V., Higgins J. P. T., Rothstein H. R. (2009) Introduction to meta-analysis. Wiley, UK

    Book  Google Scholar 

  • *Brünken R., Leutner D. (2001) Aufmerksamkeitsverteilung oder Aufmerksamkeitsfokussierung? Empirische Ergebnisse zur “Split-Attention-Hypothese” beim Lernen mit Multimedia. Unterrichtswissenschaft 29: 357–366

    Google Scholar 

  • Brünken R., Plass J. L., Leutner D. (2003) Direct measurement of cognitive load in multimedia learning. Educational Psychologist 38: 53–61

    Article  Google Scholar 

  • *Brünken R., Plass J. L., Leutner D. (2004) Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science 32: 115–132

    Article  Google Scholar 

  • *Brünken R., Steinbacher S., Plass J. L., Leutner D. (2002) Assessment of cognitive load in multimedia learning using dual-task methodology. Experimental Psychology 49: 109–119

    Article  PubMed  Google Scholar 

  • Carney R. N., Levin J. R. (2002) Pictorial illustrations still improve students’ learning from text. Educational Psychology Review 14(1): 5–26

    Article  Google Scholar 

  • Chandler P., Sweller J. (1991) Cognitive load theory and the format of instruction. Cognition and Instruction 8: 293–332

    Article  Google Scholar 

  • *Chung, W. (2006). The effects of presentation pace and modality on learning a multimedia science lesson. Dissertation submitted to the Faculty of the Graduate School of the University of Texas at Austin (UMI Microform 3221810, ProQuest Information and Learning Company).

  • *Craig S. D., Gholson B., Driscoll D. M. (2002) Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features, and redundancy. Journal of Educational Psychology 94(2): 428–434

    Article  Google Scholar 

  • *Douglas, E. S. (2007). Modality effects on adult learning. Dissertation submitted to the Department of Educational Psychology of the University of Nevada, Las Vegas (UMI Microform 3261073, ProQuest Information and Learning Company).

  • Duis S. S., Dean R. S., Derks P. (1994) The modality effect: A result of methodology?. International Journal of Neuroscience 78: 1–7

    Article  PubMed  Google Scholar 

  • *Dunsworth Q., Atkinson R. K. (2007) Fostering multimedia learning of science: Exploring the role of an animated agent’s image. Computers and Education 49: 677–690

    Article  Google Scholar 

  • Ginns P. (2005) Meta-analysis of the modality effect. Learning and Instruction 4: 313–331

    Article  Google Scholar 

  • Glass G. V., McGaw B., Smith M. L. (1981) Meta-analysis in social research. Sage, Beverly Hills

    Google Scholar 

  • *Gyselinck V., Jamet E., Dubois V. (2008) The role of working memory components in multimedia comprehension. Applied Cognitive Psychology 22: 353–374

    Article  Google Scholar 

  • Hannus M., Hyönä J. (1999) Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology 24: 95–123

    Article  PubMed  Google Scholar 

  • Hedges L. (1981) Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6: 107–128

    Article  Google Scholar 

  • Higgins J., Thompson S. G., Deeks J. J., Altman D. G. (2003) Measuring inconsistency in meta-analyses. BMJ 327: 557–560

    Article  PubMed  Google Scholar 

  • Höffler T. N., Leutner D. (2007) Instructional animation versus static pictures: A meta-analysis. Learning and Instruction 17: 722–738

    Article  Google Scholar 

  • *Jeung H.-J., Chandler P., Sweller J. (1997) The role of visual indicators in dual sensory mode instruction. Educational Psychology 17(3): 329–343

    Article  Google Scholar 

  • Jones D. M., Macken W. J., Nicholls A. P. (2004) The phonological store of working memory: Is it phonological and is it a store?. Journal of Experimental Psychology: Learning, Memory, and Cognition 30(3): 656–674

    Article  PubMed  Google Scholar 

  • *Joseph, J. H. (1978). The instructional effectiveness of integrating abstract and realistic visualization. Thesis submitted to the Graduate School, Division of Curriculum and Instruction, of the Pennsylvania State University.

  • Kalyuga S. (2005) Prior knowledge principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 325–337

    Google Scholar 

  • Kalyuga S., Ayres P., Chandler P., Sweller J. (2003) The expertise reversal effect. Educational Psychologist 38: 23–31

    Article  Google Scholar 

  • *Kalyuga S., Chandler P., Sweller J. (1999) Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology 13: 351–371

    Article  Google Scholar 

  • *Kalyuga S., Chandler P., Sweller J. (2000) Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology 92(1): 126–136

    Article  Google Scholar 

  • Kalyuga S., Chandler P., Tuovinen J., Sweller J. (2001) When problem solving is superior to studying worked examples. Journal of Educational Psychology 93: 579–588

    Article  Google Scholar 

  • Kamide Y., Altmann G. T. M., Haywood S. L. (2003) The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language 49: 133–159

    Article  Google Scholar 

  • Knoeferle P., Crocker M. W. (2006) The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science 30: 481–529

    Article  PubMed  Google Scholar 

  • *Leahy W., Chandler P., Sweller J. (2003) When auditory presentations should and should not be a component of multimedia instruction. Applied Cognitive Psychology 17: 401–418

    Article  Google Scholar 

  • Levin J. R., Divine-Hawkins P. (1974) Visual imagery as a prose-learning process. Journal of Reading Behavior 6(1): 23–30

    Google Scholar 

  • Lipsey M. W. (2003) Those confounded moderators in meta-analysis: Good, bad, and ugly. The Annals of the American Political and Social Science 587(1): 69–81

    Article  Google Scholar 

  • Low R., Sweller J. (2005) The modality principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 147–158

    Google Scholar 

  • *Main R. E., Griffiths B. (1977) Evaluation of audio and pictorial instructional supplements. AV Communication Review 25(2): 167–179

    Google Scholar 

  • *Mann B. L. (1997) Evaluation of presentation modalities in a hypermedia system. Computers & Education 28(2): 133–143

    Article  Google Scholar 

  • *Mann B., Newhouse P., Pagram J., Campbell A., Schulz H. (2002) A comparison of temporal speech and text cueing in educational multimedia. Journal of Computer Assisted Learning 18: 296–308

    Article  Google Scholar 

  • Mayer, R. E. (ed.) (2005a) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge

    Google Scholar 

  • Mayer R. E. (2005b) Principles for managing essential processing in multimedia learning: Segmenting, pretraining, and modality principles. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 169–182

    Google Scholar 

  • Mayer R. E. (2005) Cognitive theory of multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 31–48

    Google Scholar 

  • Mayer R. E., Bove W., Bryman A., Mars R., Tapangco L. (1996) When less is more: Meaningful learning from visual and verbal summaries of science textbook lessons. Journal of Educational Psychology 88(1): 64–73

    Article  Google Scholar 

  • *Mayer R. E., Dow G. T., Mayer S. (2003) Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds?. Journal of Educational Psychology 95(4): 806–813

    Article  Google Scholar 

  • *Mayer R. E., Moreno R. (1998) A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology 90(2): 312–320

    Article  Google Scholar 

  • Mayer R. E., Moreno R. (2002) Animation as an aid to multimedia learning. Educational Psychology Review 14(1): 87–99

    Article  Google Scholar 

  • Mayer R. E., Steinhoff K., Bower G., Mars R. (1995) A generative theory of textbook design: Using annotated illustrations to foster meaningful learning of science text. Educational Technology Research and Development 43: 31–43

    Article  Google Scholar 

  • *McNeill, A. L. (2004). The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment. Dissertation submitted to the Faculty of Virginia Polytechnic Institute and State University (UMI Microform 3142239, ProQuest Information and Learning Company).

  • *Mein, N. N. (2005). Impact of audio text, visual text and cueing on cognitive load and performance. Graduate School of Wayne State University, Detroit, Michigan (UMI Microform 3166768, ProQuest Information and Learning Company).

  • Moreno R. (2006) Does the modality principle hold for different media?: A test of the method-affects-learning hypothesis. Journal of Computer Assisted Learning 22(3): 149–158

    Article  Google Scholar 

  • *Moreno R., Mayer R. E. (1999) Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology 91(2): 358–368

    Article  Google Scholar 

  • *Moreno R., Mayer R. (2002) Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Education Psychology 94: 598–610

    Article  Google Scholar 

  • *Moreno R., Mayer R., Spires H. A., Lester J. C. (2001) The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents?. Cognition and Instruction 19: 177–213

    Article  Google Scholar 

  • *Mousavi S. Y., Low R., Sweller J. (1995) Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology 87(2): 319–334

    Article  Google Scholar 

  • *Nugent G. C. (1982) Pictures, audio, and print: Symbolic representation and effect on learning. Educational Communication and Technology Journal 30(3): 163–174

    Google Scholar 

  • *O’Neil H. F., Mayer R. E., Herl H., Niemi C., Olin K., Thurman R. A. (2000) Instructional strategies for virtual aviation training environments. In: O’Neil H. F., Andrews D. H. (eds) Aircrew training and assessment. Erlbaum, Mahwah, pp 105–130

    Google Scholar 

  • Paivio A. (1986) Mental representations: A dual-coding approach. Oxford University Press, New York

    Google Scholar 

  • Penney C. G. (1989) Modality effects and the structure of short-term memory. Memory and Cognition 17: 398–442

    Article  Google Scholar 

  • Rayner K., Rotello C. M., Stewart A. J., Keir J., Duffy S. A. (2001) Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied 7: 219–226

    Article  PubMed  Google Scholar 

  • Rickheit G., Strohner H., Müsseler J., Nattkemper D. (1987) Recalling oral and written discourse. Journal of Educational Psychology 97: 438–444

    Article  Google Scholar 

  • Sadoski M., Paivio A. (2001) Imagery and text—a dual coding theory of reading and writing. Erlbaum, Mahwah

    Google Scholar 

  • *Schneider V. I., Healy A. F., Barshi I. (2004) Effects of instruction modality and readback on accuracy in following navigation commands. Journal of Experimental Psychology: Applied 10(4): 245–257

    Article  PubMed  Google Scholar 

  • Schnotz W. (2005) An integrated model of text and picture comprehension. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 49–69

    Google Scholar 

  • Schnotz W., Kürschner C. (2007) A reconsideration of cognitive load theory. Educational Psychological Review 19: 469–508

    Article  Google Scholar 

  • *Segers E., Verhoeven L., Hulstijn-Hendrikse H. (2008) Cognitive processes in children’s multimedia text learning. Applied Cognitive Psychology 22: 375–387

    Article  Google Scholar 

  • *Seufert T., Schütze M., Brünken R. (2009) Memory characteristics and modality in multimedia learning: An aptitude-treatment-interaction study. Learning and Instruction 19(1): 28–42

    Article  Google Scholar 

  • *Sewell E. H. Jr, Moore R. L. (1980) Cartoon embellishments in informative presentations. Educational Communication and Technology Journal 28(1): 39–46

    Google Scholar 

  • *Stromfors, C. M. (2005). The impact of modality and working memory capacity on achievement in a multimedia environment. Dissertation submitted to the Arizona State University (UMI Microform 3178274, ProQuest Information and Learning Company).

  • Sweller J. (2005a) The redundancy principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 159–167

    Google Scholar 

  • Sweller, J. (2005b). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge 1009 handbook of multimedia learning (pp. 19–30). Cambridge: Cambridge University Press.

  • *Tabbers, H. K. (2002). The modality of text in multimedia instructions: refining the design guidelines. Unpublished doctoral dissertation, Educational Technology Expertise Centre, Open University of the Netherlands, Heerlen. http://www.ou.nl/Docs/Expertise/OTEC/Publicaties/huib%20tabbers/doctoral%20dissertation%20Huib%20Tabbers%20-%20web%20version.pdf. Accessed 03/25/2009.

  • *Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2000). Multimedia instructions and cognitive load theory: split-attention and modality effects. Paper presented at the National Convention of the Association for Educational Communications and Technology, Long Beach, CA.

  • *Tabbers H. K., Martens R. L., van Merriënboer J. J. G. (2001) The modality effect in multimedia instructions. In: Moore J. D., Stenning K. (eds) Proceedings of the twenty-third annual conference of the Cognitive Science Society. Lawrence Erlbaum Associates, Mahwah, pp 1024–1029

    Google Scholar 

  • Tabbers H. K., Martens R. L., van Merriënboer J. J. G. (2004) Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology 74: 71–81

    Article  PubMed  Google Scholar 

  • *Tindall-Ford S., Chandler P., Sweller J. (1997) When two sensory modes are better than one. Journal of Experimental Psychology: Applied 3: 257–287

    Article  Google Scholar 

  • *Tindall-Ford S., Sweller J. (2006) Altering the modality of instructions to facilitate imagination: Interactions between the modality and imagination effects. Instructional Science 34: 343–365

    Article  Google Scholar 

  • Underwood G., Jebbet L., Roberts K. T. (2004) Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. Quarterly Journal of Experimental Psychology 57: 165–182

    Article  PubMed  Google Scholar 

  • van Merriënboer J. J. G. (1997) Training complex cognitive skills: A four-component instructional design model for technical training. Educational Technology Publications, Englewood Cliffs, NJ

    Google Scholar 

  • van Merriënboer J. J. G., Kester L. (2005) The four-component instructional design model: Multimedia principles in environments for complex learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 71–93

    Google Scholar 

  • *Van Gerven, P. W. M. (2002). Efficient complex skills training into old age: Exploring the benefits of cognitive load theory. Dissertation submitted to the Open University of the Netherlands, Heerlen. http://www.personeel.unimaas.nl/p.vangerven/Index_files/Pascal%20van%20Gerven%20-%20Dissertation.pdf. Accessed 03/25/2009.

  • *Wouters P., Paas F., van Merriënboer J. J. G. (2009) Observational learning from animated models: Effects of modality and reflection on transfer. Contemporary Educational Psychology 34: 1–8

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Département de linguistique, Université du Québec à Montréal, C.P. 8888, Succursale Centre-Ville, Montréal, QC, H3C 3P8, Canada

    Joachim Reinwein

Authors
  1. Joachim Reinwein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joachim Reinwein.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reinwein, J. Does the Modality Effect Exist? and if So, Which Modality Effect?. J Psycholinguist Res 41, 1–32 (2012). https://doi.org/10.1007/s10936-011-9180-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10936-011-9180-4

Keywords

Search

Navigation