Journal of Science Education and Technology

, Volume 24, Issue 1, pp 43–55 | Cite as

Using Eye Tracking to Investigate Semantic and Spatial Representations of Scientific Diagrams During Text-Diagram Integration

  • Yu-Cin Jian
  • Chao-Jung Wu


We investigated strategies used by readers when reading a science article with a diagram and assessed whether semantic and spatial representations were constructed while reading the diagram. Seventy-one undergraduate participants read a scientific article while tracking their eye movements and then completed a reading comprehension test. Our results showed that the text-diagram referencing strategy was commonly used. However, some readers adopted other reading strategies, such as reading the diagram or text first. We found all readers who had referred to the diagram spent roughly the same amount of time reading and performed equally well. However, some participants who ignored the diagram performed more poorly on questions that tested understanding of basic facts. This result indicates that dual coding theory may be a possible theory to explain the phenomenon. Eye movement patterns indicated that at least some readers had extracted semantic information of the scientific terms when first looking at the diagram. Readers who read the scientific terms on the diagram first tended to spend less time looking at the same terms in the text, which they read after. Besides, presented clear diagrams can help readers process both semantic and spatial information, thereby facilitating an overall understanding of the article. In addition, although text-first and diagram-first readers spent similar total reading time on the text and diagram parts of the article, respectively, text-first readers had significantly less number of saccades of text and diagram than diagram-first readers. This result might be explained as text-directed reading.


Eye movement Scientific diagrams Text-diagram reading strategies Semantic representation Spatial representation 



This study is supported by the grant NSC101-2511-S-003-015 from the National Science Council in Taiwan.


  1. Afflerbach P, Pearson P, Paris SG (2008) Clarifying differences between reading skills and reading strategies. Read Teach 61(5):364–373. doi: 10.1598/RT.61.5.1 CrossRefGoogle Scholar
  2. Ainsworth S (1999) The functions of multiple representations. Comput Educ 33:131–152. doi: 10.1016/S0360-1315(99)00029-9 CrossRefGoogle Scholar
  3. Andrews S, Miller B, Rayner K (2004) Eye movements and morphological segmentation of compound words: there is a mouse in mousetrap. Eur J Cogn Psychol 16(1/2):285–311. doi: 10.1080/09541440340000123
  4. Boucheix JM, Lowe RK (2010) An eye tracking comparison of external pointing cues and internal continuous cues in learning with complex animations. Learn Instr 20:123–135. doi: 10.1016/j.learninstruc.2009.02.015 CrossRefGoogle Scholar
  5. Brusnighan MB, Folk JR (2012) Combining contextual and morphemic cues is beneficial during incidental vocabulary acquisition: semantic transparency in novel compound word processing. Read Res Q 47(2):172–190. doi: 10.1002/RRQ.015 Google Scholar
  6. Carney RN, Levin JR (2002) Pictorial illustrations still improve students’ learning from text. Educ Psychol Rev 14(1):5–26. doi: 10.1023/A%3A1013176309260 CrossRefGoogle Scholar
  7. Cook MP (2006) Visual representations in science education: the influence of prior knowledge and cognitive load theory on instructional design principles. Sci Educ 90:1073–1091. doi: 10.1002/sce.20164 CrossRefGoogle Scholar
  8. De Koning BB, Tabbers HK, Rikers RMJP, Paas F (2007) Attention cueing as a mean to enhance learning from an animation. Appl Cognit Psychol 21:731–746. doi: 10.1002/acp.1346 CrossRefGoogle Scholar
  9. De Koning BB, Tabbers HK, Rikers RMJP, Paas F (2010) Attention guidance in learning from a complex animation: seeing is understanding? Learn Instr 20(2):111–122. doi: 10.1016/j.learninstruc.2009.02.010 CrossRefGoogle Scholar
  10. Ferk V, Vrtacnik M, Blejec A, Gril A (2003) Students’ understanding of molecular structure representations. Int J Sci Educ 25(10):1227–1245. doi: 10.1080/0950069022000038231 CrossRefGoogle Scholar
  11. Hannus M, Hyönä J (1999) Utilization of illustrations during learning of science textbook passages among low- and high-ability children. Contemp Educ Psychol 24:95–123. doi: 10.1006/ceps.1998.0987 CrossRefGoogle Scholar
  12. Hegarty M (1992) Mental animation: Inferring motion from static displays of mechanical systems. J Exp Psychol Learn Mem Cognit 18:1084–1102. doi: 10.1037//0278-7393.18.5.1084 CrossRefGoogle Scholar
  13. Hegarty M, Just MA (1993) Constructing mental models of machines from text and diagrams. J Mem Lang 32:717–742. doi: 10.1006/jmla.1993.1036 CrossRefGoogle Scholar
  14. Hegarty M, Carpenter PA, Just MA (1991) Diagrams in the comprehension of scientific texts. In: Barr R, Kamil ML, Mosenthal PB, Pearson PD (eds) Handbook of reading research, vol II. Longman, New York, pp 641–668Google Scholar
  15. Heiser J, Tversky B (2006) Arrows in comprehending and producing mechanical diagrams. Cognit Sci 30:581–592. doi: 10.1207/s15516709cog0000_70 CrossRefGoogle Scholar
  16. Holsanova J, Holmberg N, Holmqvist K (2009) Reading information graphics: the role of spatial contiguity and dual attentional guidance. Appl Cognit Psychol 23:1215–1226. doi: 10.1002/acp.1525 CrossRefGoogle Scholar
  17. Hyönä J (2010) The use of eye movements in the study of multimedia learning. Learn Instr 20(2):172–176. doi: 10.1016/j.learninstruc.2009.02.013 CrossRefGoogle Scholar
  18. Jian YC, Chen ML, Ko HW (2013) Context effects in processing of Chinese academic words: an eye-tracking investigation. Read Res Quart 48(4):403–413. doi: 10.1002/rrq.56
  19. Jian YC, Wu CJ (2011) Using eye tracking to investigate the function of diagram in scientific article (in Chinese). Oral presentation at the 28th conference on Chinese Science Education, Kaohsiung, TaiwanGoogle Scholar
  20. Jian YC, Wu CJ (2012) The effect of arrows in an illustration when reading scientific text: evidence from eye movements (in Chinese). Chinese J Psychol 54:385–402Google Scholar
  21. Johnson CI, Mayer RE (2012) An eye movement analysis of the spatial contiguity effect in multimedia learning. J Exp Psychol Appl 18:178–191. doi: 10.1037/a0026923 CrossRefGoogle Scholar
  22. Kress G, van Leeuwen T (1996) Reading images: the grammar of the visual design. Routledge, New YorkGoogle Scholar
  23. LeDoux J (1994) Emotion, memory and the brain. Sci Am 270:50–57CrossRefGoogle Scholar
  24. Ledoux K, Gordon PC, Camblin CC, Swaab TY (2007) Coreference and lexical repetition: mechanisms of discourse integration. Mem Cognit 35(4):801–815. doi: 10.3758/BF03193316 CrossRefGoogle Scholar
  25. Liversedge SP, Pickering MJ, Clayes EL, Branigan HP (2003) Thematic processing of adjuncts: evidence from an eye-tracking experiment. Psychon Bull Rev 10(3):667–675. doi: 10.3758/BF03196530 CrossRefGoogle Scholar
  26. Mason L, Pluchino P, Tornatora MC, Ariasi N (2013a) An eye-tracking study of learning from science text with concrete and abstract illustrations. J Exp Educ 81:356–384. doi: 10.1080/00220973.2012.727885 CrossRefGoogle Scholar
  27. Mason L, Tornatora MC, Pluchino P (2013b) Do fourth graders integrated text and picture in processing and learning from an illustrated science text? Evidence from eye-movement patterns. Comput Educ 60:95–109. doi: 10.1016/j.compedu.2012.07.011 CrossRefGoogle Scholar
  28. Mayer RE (1989) Systematic thinking fostered by illustrations in scientific text. J Educ Psychol 81:240–246. doi: 10.1037/0022-0663.81.2.240 CrossRefGoogle Scholar
  29. Mayer RE (2001) Multimedia learning. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  30. Mayer RE (2005) Cognitive theory of multimedia learning. In: Mayer RE (ed) Cambridge handbook of multimedia learning. Cambridge University Press, New York, pp 31–48CrossRefGoogle Scholar
  31. Mayer RE, Gallini JK (1990) When is an illustration worth ten thousand words? J Educ Psychol 82:715–726. doi: 10.1037/0022-0663.82.4.715 CrossRefGoogle Scholar
  32. Mayer RE, Moreno R (2003) Nine ways to reduce cognitive load in multimedia learning. Educ Psychol 38:43–52. doi: 10.1207/S15326985EP3801_6 CrossRefGoogle Scholar
  33. Ozcelik E, Arslan-Ari I, Cagiltay K (2010) Why does signaling enhance multimedia learning? Evidence from eye movements. Comput Hum Behav 26:110–117CrossRefGoogle Scholar
  34. Paivio A (1990) Dual coding theory. In: Paivio A (ed) Mental representations: a dual coding approach. Oxford University Press, New York, pp 53–83CrossRefGoogle Scholar
  35. Peeck J (1993) Increasing picture effects in learning from illustrated text. Learn Instr 3:227–238. doi: 10.1016/0959-4752(93)90006-L CrossRefGoogle Scholar
  36. Pozzer LL, Roth WM (2003) Prevalence, function, and structure of photographs in highschool biology textbooks. J Res Sci Teach 40(10):1089–1114. doi: 10.1002/tea.10122 CrossRefGoogle Scholar
  37. Ratwani RM, Trafton JG, Boehm-Davis DA (2008) Thinking graphically: connecting vision and cognition during graph comprehension. J Exp Psychol Appl 14(1):36–49. doi: 10.1037/1076-898X.14.1.36 CrossRefGoogle Scholar
  38. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422. doi: 10.1037/0033-2909.124.3.372
  39. Rayner K, Rotello CM, Stewart AJ, Keir J, Duffy SA (2001) Integrating text and pictorial information: eye movements when looking at print advertisements. J Exp Psychol Appl 7:219–226. doi: 10.1037//1076-898X.7.3.219 CrossRefGoogle Scholar
  40. Roth WM, Pozzer-Ardenghi L, Han J (2005) Critical graphicacy: Understanding visual representation practices in school science. Springer, DordrechtGoogle Scholar
  41. Schnotz W (2002) Toward an integrated view of learning from text and visual displays. Educ Psychol Rev 14(1):101–120. doi: 10.1023/A:1013136727916 CrossRefGoogle Scholar
  42. She HC, Chen YZ (2009) The impact of multimedia effect on science learning: evidence from eye movements. Comput Educ 53:1297–1307. doi: 10.1016/j.compedu.2009.06.012 CrossRefGoogle Scholar
  43. Slough S, McTigue E (2010) Introduction to the integration of verbal and visual information in science texts. Read Psychol 31(3):206–212. doi: 10.1080/02702710903241397 CrossRefGoogle Scholar
  44. Slough S, McTigue EM, Kim S, Jennings S (2010) Science textbook’s use of graphical representation: a descriptive analysis of four sixth-grade science texts. Read Psychol 31:301–325. doi: 10.1080/02702710903256502 CrossRefGoogle Scholar
  45. Unsworth L (2001) Teaching multiliteracies across the curriculum: changing contexts of text and image in classroom practice. Open University Press, BuckinghamGoogle Scholar
  46. Yang Yu-Fen (2006) Reading strategies or comprehension monitoring strategies? Read Psychol 27(4):313–343. doi: 10.1080/02702710600846852 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Educational Psychology and CounselingNational Taiwan Normal UniversityTaipeiTaiwan

Personalised recommendations