Advertisement

Journal of Psycholinguistic Research

, Volume 41, Issue 1, pp 1–32 | Cite as

Does the Modality Effect Exist? and if So, Which Modality Effect?

  • Joachim Reinwein
Article

Abstract

The modality effect is a central issue in multimedia learning [see Mayer (Cambridge University Press, 2005a), for a review]. Sweller’s Cognitive Load Theory (CLT), for example, presumes that an illustrated text is better understood when presented visually rather than orally. The predictive power of CLT lies in how it links in to Baddeley’s (1986) model of working memory and Penney’s (Mem Cognit 17:398–442, 1989) Separate-Streams Hypothesis. Ginns’s (Learn Instr 4:313–331, 2005) recent meta-analysis also supports the modality effect (d = 0.72, based on 43 independent effects). This article replicates the meta-analysis of the modality effect based on 86 independent effects (with within-study subgroups as the unit of analysis and with mean of the outcomes as the dependent measure), with results showing a reduction of the overall effect size by almost half (d = 0.38), and even more when Duval and Tweedie’s Trim and Fill method is used to correct publication bias (d = 0.20). This article also widens the scope of the analysis of moderator variables (e.g. Pace of presentation, Type of visualization, Research group) as well as their potentially confounded effects. Finally, it is argued that, for theoretical reasons, the so-called modality effect cannot be based on Penney’s or Baddeley’s theories and must be explained in a different way.

Keywords

Modality effect Meta-analysis Illustrated text Working memory Cognitive load theory 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. * = References with an asterisk are considered in the present meta-analysis.Google Scholar
  2. *Abel R. R., Kulhavy R. W. (1986) Maps, mode of text presentation, and children’s prose learning. American Educational Research Journal 23(2): 263–274Google Scholar
  3. Altmann G. T. M., Kamide Y. (1999) Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73: 247–264PubMedCrossRefGoogle Scholar
  4. Altmann G. T. M., Kamide Y. (2007) The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language 57(4): 502–518CrossRefGoogle Scholar
  5. *Atkinson R. K. (2002) Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology 94: 416–427CrossRefGoogle Scholar
  6. Baddeley A. D. (1986) Working memory. Oxford University Press, OxfordGoogle Scholar
  7. Baddeley A. D. (1992) Working memory. Science 255: 556–559PubMedCrossRefGoogle Scholar
  8. Borenstein M., Hedges L. V., Higgins J. P. T., Rothstein H. R. (2009) Introduction to meta-analysis. Wiley, UKCrossRefGoogle Scholar
  9. *Brünken R., Leutner D. (2001) Aufmerksamkeitsverteilung oder Aufmerksamkeitsfokussierung? Empirische Ergebnisse zur “Split-Attention-Hypothese” beim Lernen mit Multimedia. Unterrichtswissenschaft 29: 357–366Google Scholar
  10. Brünken R., Plass J. L., Leutner D. (2003) Direct measurement of cognitive load in multimedia learning. Educational Psychologist 38: 53–61CrossRefGoogle Scholar
  11. *Brünken R., Plass J. L., Leutner D. (2004) Assessment of cognitive load in multimedia learning with dual-task methodology: Auditory load and modality effects. Instructional Science 32: 115–132CrossRefGoogle Scholar
  12. *Brünken R., Steinbacher S., Plass J. L., Leutner D. (2002) Assessment of cognitive load in multimedia learning using dual-task methodology. Experimental Psychology 49: 109–119PubMedCrossRefGoogle Scholar
  13. Carney R. N., Levin J. R. (2002) Pictorial illustrations still improve students’ learning from text. Educational Psychology Review 14(1): 5–26CrossRefGoogle Scholar
  14. Chandler P., Sweller J. (1991) Cognitive load theory and the format of instruction. Cognition and Instruction 8: 293–332CrossRefGoogle Scholar
  15. *Chung, W. (2006). The effects of presentation pace and modality on learning a multimedia science lesson. Dissertation submitted to the Faculty of the Graduate School of the University of Texas at Austin (UMI Microform 3221810, ProQuest Information and Learning Company).Google Scholar
  16. *Craig S. D., Gholson B., Driscoll D. M. (2002) Animated pedagogical agents in multimedia educational environments: Effects of agent properties, picture features, and redundancy. Journal of Educational Psychology 94(2): 428–434CrossRefGoogle Scholar
  17. *Douglas, E. S. (2007). Modality effects on adult learning. Dissertation submitted to the Department of Educational Psychology of the University of Nevada, Las Vegas (UMI Microform 3261073, ProQuest Information and Learning Company).Google Scholar
  18. Duis S. S., Dean R. S., Derks P. (1994) The modality effect: A result of methodology?. International Journal of Neuroscience 78: 1–7PubMedCrossRefGoogle Scholar
  19. *Dunsworth Q., Atkinson R. K. (2007) Fostering multimedia learning of science: Exploring the role of an animated agent’s image. Computers and Education 49: 677–690CrossRefGoogle Scholar
  20. Ginns P. (2005) Meta-analysis of the modality effect. Learning and Instruction 4: 313–331CrossRefGoogle Scholar
  21. Glass G. V., McGaw B., Smith M. L. (1981) Meta-analysis in social research. Sage, Beverly HillsGoogle Scholar
  22. *Gyselinck V., Jamet E., Dubois V. (2008) The role of working memory components in multimedia comprehension. Applied Cognitive Psychology 22: 353–374CrossRefGoogle Scholar
  23. Hannus M., Hyönä J. (1999) Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemporary Educational Psychology 24: 95–123PubMedCrossRefGoogle Scholar
  24. Hedges L. (1981) Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational Statistics 6: 107–128CrossRefGoogle Scholar
  25. Higgins J., Thompson S. G., Deeks J. J., Altman D. G. (2003) Measuring inconsistency in meta-analyses. BMJ 327: 557–560PubMedCrossRefGoogle Scholar
  26. Höffler T. N., Leutner D. (2007) Instructional animation versus static pictures: A meta-analysis. Learning and Instruction 17: 722–738CrossRefGoogle Scholar
  27. *Jeung H.-J., Chandler P., Sweller J. (1997) The role of visual indicators in dual sensory mode instruction. Educational Psychology 17(3): 329–343CrossRefGoogle Scholar
  28. Jones D. M., Macken W. J., Nicholls A. P. (2004) The phonological store of working memory: Is it phonological and is it a store?. Journal of Experimental Psychology: Learning, Memory, and Cognition 30(3): 656–674PubMedCrossRefGoogle Scholar
  29. *Joseph, J. H. (1978). The instructional effectiveness of integrating abstract and realistic visualization. Thesis submitted to the Graduate School, Division of Curriculum and Instruction, of the Pennsylvania State University.Google Scholar
  30. Kalyuga S. (2005) Prior knowledge principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 325–337Google Scholar
  31. Kalyuga S., Ayres P., Chandler P., Sweller J. (2003) The expertise reversal effect. Educational Psychologist 38: 23–31CrossRefGoogle Scholar
  32. *Kalyuga S., Chandler P., Sweller J. (1999) Managing split-attention and redundancy in multimedia instruction. Applied Cognitive Psychology 13: 351–371CrossRefGoogle Scholar
  33. *Kalyuga S., Chandler P., Sweller J. (2000) Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology 92(1): 126–136CrossRefGoogle Scholar
  34. Kalyuga S., Chandler P., Tuovinen J., Sweller J. (2001) When problem solving is superior to studying worked examples. Journal of Educational Psychology 93: 579–588CrossRefGoogle Scholar
  35. Kamide Y., Altmann G. T. M., Haywood S. L. (2003) The time-course of prediction in incremental sentence processing: Evidence from anticipatory eye movements. Journal of Memory and Language 49: 133–159CrossRefGoogle Scholar
  36. Knoeferle P., Crocker M. W. (2006) The coordinated interplay of scene, utterance, and world knowledge: Evidence from eye tracking. Cognitive Science 30: 481–529PubMedCrossRefGoogle Scholar
  37. *Leahy W., Chandler P., Sweller J. (2003) When auditory presentations should and should not be a component of multimedia instruction. Applied Cognitive Psychology 17: 401–418CrossRefGoogle Scholar
  38. Levin J. R., Divine-Hawkins P. (1974) Visual imagery as a prose-learning process. Journal of Reading Behavior 6(1): 23–30Google Scholar
  39. Lipsey M. W. (2003) Those confounded moderators in meta-analysis: Good, bad, and ugly. The Annals of the American Political and Social Science 587(1): 69–81CrossRefGoogle Scholar
  40. Low R., Sweller J. (2005) The modality principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 147–158Google Scholar
  41. *Main R. E., Griffiths B. (1977) Evaluation of audio and pictorial instructional supplements. AV Communication Review 25(2): 167–179Google Scholar
  42. *Mann B. L. (1997) Evaluation of presentation modalities in a hypermedia system. Computers & Education 28(2): 133–143CrossRefGoogle Scholar
  43. *Mann B., Newhouse P., Pagram J., Campbell A., Schulz H. (2002) A comparison of temporal speech and text cueing in educational multimedia. Journal of Computer Assisted Learning 18: 296–308CrossRefGoogle Scholar
  44. Mayer, R. E. (ed.) (2005a) The Cambridge handbook of multimedia learning. Cambridge University Press, CambridgeGoogle Scholar
  45. Mayer R. E. (2005b) Principles for managing essential processing in multimedia learning: Segmenting, pretraining, and modality principles. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 169–182Google Scholar
  46. Mayer R. E. (2005) Cognitive theory of multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 31–48Google Scholar
  47. Mayer R. E., Bove W., Bryman A., Mars R., Tapangco L. (1996) When less is more: Meaningful learning from visual and verbal summaries of science textbook lessons. Journal of Educational Psychology 88(1): 64–73CrossRefGoogle Scholar
  48. *Mayer R. E., Dow G. T., Mayer S. (2003) Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds?. Journal of Educational Psychology 95(4): 806–813CrossRefGoogle Scholar
  49. *Mayer R. E., Moreno R. (1998) A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology 90(2): 312–320CrossRefGoogle Scholar
  50. Mayer R. E., Moreno R. (2002) Animation as an aid to multimedia learning. Educational Psychology Review 14(1): 87–99CrossRefGoogle Scholar
  51. Mayer R. E., Steinhoff K., Bower G., Mars R. (1995) A generative theory of textbook design: Using annotated illustrations to foster meaningful learning of science text. Educational Technology Research and Development 43: 31–43CrossRefGoogle Scholar
  52. *McNeill, A. L. (2004). The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment. Dissertation submitted to the Faculty of Virginia Polytechnic Institute and State University (UMI Microform 3142239, ProQuest Information and Learning Company).Google Scholar
  53. *Mein, N. N. (2005). Impact of audio text, visual text and cueing on cognitive load and performance. Graduate School of Wayne State University, Detroit, Michigan (UMI Microform 3166768, ProQuest Information and Learning Company).Google Scholar
  54. Moreno R. (2006) Does the modality principle hold for different media?: A test of the method-affects-learning hypothesis. Journal of Computer Assisted Learning 22(3): 149–158CrossRefGoogle Scholar
  55. *Moreno R., Mayer R. E. (1999) Cognitive principles of multimedia learning: The role of modality and contiguity. Journal of Educational Psychology 91(2): 358–368CrossRefGoogle Scholar
  56. *Moreno R., Mayer R. (2002) Learning science in virtual reality multimedia environments: Role of methods and media. Journal of Education Psychology 94: 598–610CrossRefGoogle Scholar
  57. *Moreno R., Mayer R., Spires H. A., Lester J. C. (2001) The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents?. Cognition and Instruction 19: 177–213CrossRefGoogle Scholar
  58. *Mousavi S. Y., Low R., Sweller J. (1995) Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology 87(2): 319–334CrossRefGoogle Scholar
  59. *Nugent G. C. (1982) Pictures, audio, and print: Symbolic representation and effect on learning. Educational Communication and Technology Journal 30(3): 163–174Google Scholar
  60. *O’Neil H. F., Mayer R. E., Herl H., Niemi C., Olin K., Thurman R. A. (2000) Instructional strategies for virtual aviation training environments. In: O’Neil H. F., Andrews D. H. (eds) Aircrew training and assessment. Erlbaum, Mahwah, pp 105–130Google Scholar
  61. Paivio A. (1986) Mental representations: A dual-coding approach. Oxford University Press, New YorkGoogle Scholar
  62. Penney C. G. (1989) Modality effects and the structure of short-term memory. Memory and Cognition 17: 398–442CrossRefGoogle Scholar
  63. Rayner K., Rotello C. M., Stewart A. J., Keir J., Duffy S. A. (2001) Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied 7: 219–226PubMedCrossRefGoogle Scholar
  64. Rickheit G., Strohner H., Müsseler J., Nattkemper D. (1987) Recalling oral and written discourse. Journal of Educational Psychology 97: 438–444CrossRefGoogle Scholar
  65. Sadoski M., Paivio A. (2001) Imagery and text—a dual coding theory of reading and writing. Erlbaum, MahwahGoogle Scholar
  66. *Schneider V. I., Healy A. F., Barshi I. (2004) Effects of instruction modality and readback on accuracy in following navigation commands. Journal of Experimental Psychology: Applied 10(4): 245–257PubMedCrossRefGoogle Scholar
  67. Schnotz W. (2005) An integrated model of text and picture comprehension. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 49–69Google Scholar
  68. Schnotz W., Kürschner C. (2007) A reconsideration of cognitive load theory. Educational Psychological Review 19: 469–508CrossRefGoogle Scholar
  69. *Segers E., Verhoeven L., Hulstijn-Hendrikse H. (2008) Cognitive processes in children’s multimedia text learning. Applied Cognitive Psychology 22: 375–387CrossRefGoogle Scholar
  70. *Seufert T., Schütze M., Brünken R. (2009) Memory characteristics and modality in multimedia learning: An aptitude-treatment-interaction study. Learning and Instruction 19(1): 28–42CrossRefGoogle Scholar
  71. *Sewell E. H. Jr, Moore R. L. (1980) Cartoon embellishments in informative presentations. Educational Communication and Technology Journal 28(1): 39–46Google Scholar
  72. *Stromfors, C. M. (2005). The impact of modality and working memory capacity on achievement in a multimedia environment. Dissertation submitted to the Arizona State University (UMI Microform 3178274, ProQuest Information and Learning Company).Google Scholar
  73. Sweller J. (2005a) The redundancy principle in multimedia learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 159–167Google Scholar
  74. Sweller, J. (2005b). Implications of cognitive load theory for multimedia learning. In R. E. Mayer (Ed.), The Cambridge 1009 handbook of multimedia learning (pp. 19–30). Cambridge: Cambridge University Press.Google Scholar
  75. *Tabbers, H. K. (2002). The modality of text in multimedia instructions: refining the design guidelines. Unpublished doctoral dissertation, Educational Technology Expertise Centre, Open University of the Netherlands, Heerlen. http://www.ou.nl/Docs/Expertise/OTEC/Publicaties/huib%20tabbers/doctoral%20dissertation%20Huib%20Tabbers%20-%20web%20version.pdf. Accessed 03/25/2009.
  76. *Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. G. (2000). Multimedia instructions and cognitive load theory: split-attention and modality effects. Paper presented at the National Convention of the Association for Educational Communications and Technology, Long Beach, CA.Google Scholar
  77. *Tabbers H. K., Martens R. L., van Merriënboer J. J. G. (2001) The modality effect in multimedia instructions. In: Moore J. D., Stenning K. (eds) Proceedings of the twenty-third annual conference of the Cognitive Science Society. Lawrence Erlbaum Associates, Mahwah, pp 1024–1029Google Scholar
  78. Tabbers H. K., Martens R. L., van Merriënboer J. J. G. (2004) Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology 74: 71–81PubMedCrossRefGoogle Scholar
  79. *Tindall-Ford S., Chandler P., Sweller J. (1997) When two sensory modes are better than one. Journal of Experimental Psychology: Applied 3: 257–287CrossRefGoogle Scholar
  80. *Tindall-Ford S., Sweller J. (2006) Altering the modality of instructions to facilitate imagination: Interactions between the modality and imagination effects. Instructional Science 34: 343–365CrossRefGoogle Scholar
  81. Underwood G., Jebbet L., Roberts K. T. (2004) Inspecting pictures for information to verify a sentence: Eye movements in general encoding and in focused search. Quarterly Journal of Experimental Psychology 57: 165–182PubMedCrossRefGoogle Scholar
  82. van Merriënboer J. J. G. (1997) Training complex cognitive skills: A four-component instructional design model for technical training. Educational Technology Publications, Englewood Cliffs, NJGoogle Scholar
  83. van Merriënboer J. J. G., Kester L. (2005) The four-component instructional design model: Multimedia principles in environments for complex learning. In: Mayer R. E. (ed.) The Cambridge handbook of multimedia learning. Cambridge University Press, Cambridge, pp 71–93Google Scholar
  84. *Van Gerven, P. W. M. (2002). Efficient complex skills training into old age: Exploring the benefits of cognitive load theory. Dissertation submitted to the Open University of the Netherlands, Heerlen. http://www.personeel.unimaas.nl/p.vangerven/Index_files/Pascal%20van%20Gerven%20-%20Dissertation.pdf. Accessed 03/25/2009.
  85. *Wouters P., Paas F., van Merriënboer J. J. G. (2009) Observational learning from animated models: Effects of modality and reflection on transfer. Contemporary Educational Psychology 34: 1–8CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Département de linguistiqueUniversité du Québec à MontréalMontréalCanada

Personalised recommendations