Skip to main content
SpringerLink
Log in

Tracking students’ visual attention on manga-based interactive e-book while reading: an eye-movement approach

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This study employed an eye tracking technology to explore university students’ visual attention and learning performance while learning Japanese using an interactive manga-based e-book. The developed e-book consisted of 8 pages accompanied by 13 annotations with both text and graphical formats. The subjects consisted of 60 students whose eye movements were tracked and recorded by the eye tracking system. These students came from the applied foreign language department in a northern university in Taiwan, of which 30 were assigned to high prior knowledge (PK) group and the other 30 were assigned to low PK group. Eye tracking measurements, including total contact time, number of fixations, latency of first fixation, and number of clicks on the defined regions of interest of the two groups were compared to indicate their visual attention. The results revealed that overall students spent more time on reading text and annotation than graphic information. The high PK students showed longer fixation durations on the texts, while the low PK students showed longer fixation durations on the graphics and annotations. Meanwhile, the low PK students used more clicks to look up underlined annotations whenever they didn’t know words or phrases on the e-book. In addition, with respect to the latency of the first fixation, the graphic captured the attention faster than the text because of the size and its appeal to the students. Further analysis of saccade paths indicated that the low PK students showed more inter-scanning transitions not only between the text dialog and the annotation zone but also within annotation zone. Finally, the results of reading comprehension pretest and posttest found that there was a significant difference in learning outcomes between each PK group.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Adams J (2001) A critical study of comics. The International Journal of Art & Design Education 20(2):133–143

    Article  Google Scholar 

  2. Alexander PA, Jetton TL (2000) Learning from text: a multidimensional and developmental perspective. In: Kamil ML, Mosenthal PB, Pearson PD, Barr R (eds) Handbook of reading research, vol 3. Erlbaum, Mahwah, pp 285–310

    Google Scholar 

  3. Black RW (2005) Access and affiliation: the literacy and composition practices of English-language learners in an on-line fanfiction community. Journal of Adolescent & Adult Literacy 49(2):118–128

    Article  Google Scholar 

  4. Bryce M, Davis J, Barber C (2008) The cultural biographies and social lives of manga: lessons from the mangaverse. Journal of Media Arts Culture Collection 5(2). Retrieved from) http://scan.net.au/scan/journal/display.php?journal_id=114

  5. Calvo MG, Lang PJ (2004) Gaze patterns when looking at emotional pictures: motivationally biased attention. Motiv Emot 28(3):221–243

    Article  Google Scholar 

  6. Canham M, Hegarty M (2010) Effects of knowledge and display design on comprehension of complex graphics. Learn Instr 20(2):155–166

    Article  Google Scholar 

  7. Cary S (2004) Going graphic: Comics at work in the multilingual classroom. Heinemann, Portsmouth

    Google Scholar 

  8. Chen YC, Hwang RH, Wang CY (2012) Development and evaluation of a web 2.0 annotation system as a learning tool in an e-learning environment. Comput Educ 58(4):1094–1105

    Article  Google Scholar 

  9. Chi MTH, VanLehn KA (2012) Seeing deep structure from the interactions of surface features. Educ Psychol 47(3):177–188

    Article  Google Scholar 

  10. Chou IC (2014) Investigating EFL students’ E-book reading attitudes in first and second language. US-China Foreign. Language 12(1):64–74

    Google Scholar 

  11. Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, New Jersey

    MATH  Google Scholar 

  12. Coiro J (2011) Predicting reading comprehension on the internet: contributions of offline reading skills, online reading skills, and prior knowledge. J Lit Res 43(4):352–392

    Article  Google Scholar 

  13. Cook M, Carter GN, Wiebe E (2008) The interpretation of cellular transport graphics by students with low and high prior knowledge. Int J Sci Educ 30(2):241–263

    Article  Google Scholar 

  14. Doty DE, Popplewell SR, Byers GO (2001) Interactive CD-ROM storybooks young reader’s reading comprehension. J Res Comput Educ 3(4):374–384

    Article  Google Scholar 

  15. Epelboim J, Suppes P (1997) Eye movements during geometrical problem solving. Paper presented at the 19th Annual Conference of the Cognitive Science Society

  16. Grant ER, Spivey MJ (2003) Eye movements and problem solving guiding attention guides thought. Psychol Sci 14(5):462–466

    Article  Google Scholar 

  17. Greene JA, Costa LJ, Robertson J, Pan Y, Deekens VM (2010) Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Comput Educ 55:1027–1043

    Article  Google Scholar 

  18. Hegarty M, Mayer RE, Green C (1992) Comprehension of arithmetic word problems: evidence from students’ eye fixations. J Educ Psychol 84(1):76–84

    Article  Google Scholar 

  19. Hegarty M, Mayer RE, Monk CA (1995) Comprehension of arithmetic word problems: a comparison of successful and unsuccessful problem solvers. J Educ Psychol 87(1):18–32

    Article  Google Scholar 

  20. Hewig J, Trippe RH, Hecht H, Straube T, Miltner WHR (2008) Gender differences for specific body regions when looking at men and women. J Nonverbal Behav 32(2):67–78

    Article  Google Scholar 

  21. Higgins NC, Cocks P (1999) The effects of animation cues on vocabulary development. Journal of Reading Psychology 20:1–10

    Article  Google Scholar 

  22. Ho HN, Tsai MJ, Wang CY, Tsai CC (2014) Prior knowledge and online inquiry-based science reading: evidence from eye tracking. Int J Sci Math Educ 12:525–554

    Article  Google Scholar 

  23. Huang TH (2014) A study on the effects of interactive e-books on Taiwan high/vocational school students’ reading comprehension. Social Science Research Network (SSRN), Retrieved from SSRN https://ssrn.com/abstract=2462830

    Google Scholar 

  24. Humphrey K, Underwood G (2009) Domain knowledge moderates the influence of visual saliency in scene recognition. Br J Psychol 100(2):377–398

    Article  Google Scholar 

  25. Hwang WY, Hsu GL (2011) The effects of pre-reading and haring mechanisms on learning with the use of annotations. Turkish Online Journal of Educational Technology 10(2):234–249

    Google Scholar 

  26. Hwang WY, Wang CY, Sharples M (2007) A study of multimedia annotation of web-based materials. Comput Educ 48(4):680–699

    Article  Google Scholar 

  27. Hwang WY, Chen NS, Shadiev R, Li JS (2011a) Effects of reviewing annotations and homework solutions on math learning achievement. Br J Educ Technol 42(6):1016–1028

    Article  Google Scholar 

  28. Hwang WY, Shadiev R, Huang SM (2011b) A study of a multimedia web annotation system and its effect on the EFL writing and speaking performance of junior high school students. ReCALL 23:160–180

    Article  Google Scholar 

  29. Hwang WY, Liu YF, Chen HR, Huang JW, Li JY (2015) Role of parents and annotation sharing in children’s learning behavior and achievement using E-readers. Educational Technology & Society 18(1):292–307

    Google Scholar 

  30. Hyönä, J. Jr., Lorch, R.F., & Kaakinen, J.K. (2002). Individual differences in reading to summarize expository text: evidence from eye fixation patterns. J Educ Psychol, 94, 44–55

  31. Jarodzka H, Scheiter K, Gerjets P, van Gog T (2010) In the eyes of the beholder: how experts and novices interpret dynamic stimuli. Learn Instr 20(2):146–154

    Article  Google Scholar 

  32. Jarodzka H, Holmqvist K, Gruber H (2017) Eye tracking in educational science: theoretical frameworks and research agendas. J Eye Mov Res 10(1):1–18

    Google Scholar 

  33. Just MA, Carpenter PA (1980) A theory of reading: from eye fixations to comprehension. Psychol Rev 87:329–354

    Article  Google Scholar 

  34. Just MA, Carpenter PA (1985) Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability. Psychol Review 92(2):137-172

  35. Khurana S (2005) So you want to be a superhero? How the art of making comics in an afterschool setting can develop young people‘s creativity, literacy and identity. Afterschool Matters 4:1–9

    Google Scholar 

  36. Knoblich G, Ohlsson S, Raney GE (2001) An eye movement study of insight problem solving. Mem Cogn 29(7):1000–1009

    Article  Google Scholar 

  37. Korat O, Shamir A (2004) Do Hebrew electronic books differs from Dutch electronic books? A replication of a Dutch content analysis. J Comput Assist Learn 20(4):257–268

    Article  Google Scholar 

  38. Kristof R, Satran A (1995) Interactivity by design: creating & communicating with new media. Adobe Press, Mountain View

    Google Scholar 

  39. Lai ML, Tsai MJ, Yang FY, Hsu CY, Liu TC, Lee SWY, Lee MH, Chiou GL, Liang JC, Tsai CC (2013) A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review 10:90–115

    Article  Google Scholar 

  40. Lin IY (2009) The effect of E-books on EFL learners’ reading attitude. Master thesis, National Taiwan Normal University, Taiwan. Retrieved from http://ndltd.ncl.edu.tw/cgi-bin/gs32/gsweb.cgi?o=dnclcde&s=id=%22097NTNU5238009%22.&searchmode=basic

  41. Lin YT, Wu CC, Hou TY, Lin YC, Yang FY, Chang CH (2016) Tracking students’ cognitive processes during program debugging-an eye-movement approach. IEEE Trans Educ 59(3):175–186

    Article  Google Scholar 

  42. Liu HC, Andre T, Greenbowe T (2008) The impact of learner’s prior knowledge on their use of chemistry computer simulations: a case study. J Sci Educ Technol 17:466–482

    Article  Google Scholar 

  43. Liversedge S, Paterson K, Pickering M (1998) Eye movements and measures of reading time. In: Underwood G (ed) Eye guidance in reading and scene perception. Elsevier, Oxford, pp 55–75

    Chapter  Google Scholar 

  44. Mason L, Pluchino P, Tornatora MC, Ariasi N (2013) An eye-tracking study of learning from science text with concrete and abstract illustrations. J Exp Educ 81(3):356–384

    Article  Google Scholar 

  45. Mayer RE (1997) Multimedia learning: are we asking the right question? Educ Psychol 32(1):1–19

    Article  Google Scholar 

  46. Mayer RE (2005) The Cambridge handbook of multimedia learning. Cambridge University Press, New York

    Book  Google Scholar 

  47. Mayer RE (2008) Applying the science of learning: evidence-based principles for the design of multimedia instruction. Am Psychol 63(8):760–769

    Article  Google Scholar 

  48. Mayer RE (2009) Multimedia learning, 2nd edn. Cambridge University Press, New York

    Book  Google Scholar 

  49. Mayer RE (2011) Applying the science of learning. Pearson, Upper Saddle River

    Google Scholar 

  50. Mayer RE, Sims VK (1994) For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. J Educ Psychol 86:389–401

    Article  Google Scholar 

  51. Mintzes J, Wandersee JH, Novak JD (eds) (2000) Assessing science understanding (pp. 304–341). Academic Press, San Diego

    Google Scholar 

  52. Moist KM, Bartholomew M (2007) When pigs fly: anime, auteurism, and Miyazaki’s Porco Rosso. SAGE Publications, London

    Google Scholar 

  53. Murakami S, Bryce M (2009) Manga as an educational medium. The International Journal of the Humanities 7(10):45–55

    Google Scholar 

  54. Ogawa E (2013) Educational manga: an effective medium for english learning. In: Sonda N, Krause A (eds) Paper presented at JALT2012 conference proceedings. Tokyo, JALT

    Google Scholar 

  55. Paivio A (1986) Mental representations: a dual coding approach. Oxford University Press, Oxford

    Google Scholar 

  56. Paivio A (2006) Mind and its evolution: A dual coding theoretical interpretation. Lawrence Erlbaum Associates, Inc, Mahwah

    Google Scholar 

  57. Pan TW, Hsu MC, Tsai MJ (2013) Effect of graphic design on E-book reading: a pilot eye-tracking study. In: Tan SC et al (eds) Paper presented at the workshop proceedings of the 21st international conference on computers in education, Indonesia

  58. Pass F, van Merriënboer JJG (1994) Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. J Educ Psychol 86(1):122-133

  59. Paas F, Renkl A, Sweller J (2003) Cognitive load theory and instructional design: recent developments. Educ Psychol 38:1-4

  60. Peverly ST, Vekaria PC, Reddington LA, Sumowski JF, Johnson KR, Ramsay CM (2013) The relationship of handwriting speed, working memory, language comprehension and outlines to lecture note-taking and test-taking among college students. Appl Cogn Psychol 27(1):115–126

    Article  Google Scholar 

  61. Plass JL, Moreno R, Brunken R (eds) (2010) Cognitive load theory. Cambridge University Press, New York

    Google Scholar 

  62. Potter MC, Wyble B, Hagmann CE, McCourt ES (2014) Detecting meaning in RSVP at 13 ms per picture. Atten Percept Psychophys 76:270–279

    Article  Google Scholar 

  63. Rawlins GJE (1993) Publishing over the new decade. J Am Soc Inf Sci 44(8):474–479

    Article  Google Scholar 

  64. Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422

    Article  Google Scholar 

  65. Rayner K (2009) Eye movements and attention in reading, scene perception, and visual search. Q J Exp Psychol 62(8):1457–1506

    Article  Google Scholar 

  66. Rayner K, Rotello CM, Stewart AJ, Keir J, Duffy SA (2001) Integrating text and pictorial information: eye movements when looking at print advertisements. J Exp Psychol Appl 7:219–226

    Article  Google Scholar 

  67. Rayner K, Smith T, Malcolm GL, Henderson JM (2009) Eye movements and visual encoding during scene perception. Psychol Sci 20(1):6–10

    Article  Google Scholar 

  68. Rayner K, Slattery TJ, Bélanger NN (2010) Eye movements, the perceptual span, and reading speed. Psychon Bull Rev 17(6):834–839

    Article  Google Scholar 

  69. Reingold EM, Sheridan H (2011) Eye movements and visual expertise in chess and medicine. In: Liversedge SP, Gilchrist ID, Everling S (eds) Oxford handbook on eye movements. Oxford University Press, Oxford, pp 528–550

    Google Scholar 

  70. Riley P (2003) Drawing the threads together. In: Little D, Ridley J, Ushioda E (eds) Learner autonomy in the foreign language classroom: teacher, learner, curriculum and assessment. Dublin, Authentik, pp 237–252

    Google Scholar 

  71. Sadoski M, Willson VL (2006) Effects of a theoretically-based large scale reading intervention in a multicultural urban school district. Am Educ Res J 43:137–154

    Article  Google Scholar 

  72. Sanders MS, McCormick EJ (1987) Human factors in engineering and design. McGraw-Hill, New York

    Google Scholar 

  73. She HC, Chen YZ (2009) The impact of multimedia effect on science learning: evidence from eye movements. Comput Educ 53(4):1297–1307

    Article  Google Scholar 

  74. Shimada A, Okubo F, Yin CJ, Oi M, Kojima K, Yamada M, Ogata H (2015) Analysis of preview behavior in E-book system. In: Ogata H et al (eds) Paper presented at the 23rd international conference on computers in education (ICCE 2015), Hangzhou, China

  75. Smith, S.D., Caruso, J.B., & Kim, J. (2010). The ECAR study of undergraduate students and information technology. ECAR Research Study, 6, 1–120. Retrieved from http://library.educause.edu/~/media/files/library/2010/10/ers1006w-pdf.pdf

  76. Sweller J (1988) Cognitive load during problem solving: effects on learning. Cogn Sci 12(2):257–285

    Article  Google Scholar 

  77. Thompson RA, Zamboanga BL (2003) Prior knowledge and its relevance to student achievement in introduction to psychology. Teach Psychol 30:96–101

    Article  Google Scholar 

  78. Tsai JL, Yen MH, Wang CA (2005) Eye movement recording and the application in research of reading Chinese. Research in Applied Psychology 28:91–104

    Google Scholar 

  79. Tsai MJ, Hou HT, Lai ML, Liu WY, Yang FY (2011) Visual attention for solving multiple-choice science problem: an eye-tracking analysis. Comput Educ 58:375–385

    Article  Google Scholar 

  80. Tsai MJ, Huang LJ, Hou HT, Hsu CY, Chiou GL (2016) Visual behavior, flow and achievement in game-based learning. Comput Educ 98:115–129

    Article  Google Scholar 

  81. Van Gog T, Scheiter K (2010) Eye tracking as a tool to study and enhance multimedia learning. Learn Instr 20(2):95–99

    Article  Google Scholar 

  82. Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25(2):325–331

    Article  Google Scholar 

  83. Wolfe, P. (2010). Brain matters: Translating research into classroom practice (2nd Ed.) Alexandria: Association for Supervision & curriculum development

  84. Wu AH, Hsu PF, Chiu HJ, Tsai MJ (2014) Visual behavior and cognitive load on E-book vocabulary learning. In: Liu CC et al (eds) Paper presented at the workshop proceedings of the 22nd international conference on computers in education, Japan

  85. Yang FY, Chang CY, Chien WR, Chien YT, Tseng YH (2013) Tracking learners’ visual attention during a multimedia presentation in a real classroom. Comput Educ 62:208–220

    Article  Google Scholar 

  86. Yin CJ, Okubo F, Shimada A, Oi M, Hirokawa S, Ogata H (2015) Identifying and analyzing the learning behaviors of students using e-books. In: Ogata H et al (eds) Paper presented at the workshop proceedings of the 23rd international conference on computers in education (ICCE 2015), Hangzhou, China

  87. Zheng HT, Wang Z, Ma N, Chen J, Xiao X, Sangaiah AK (2017) Weakly-supervised image captioning based on rich contextual information. Multimedia Tools and Applications:1–17

Download references

Acknowledgements

The author is grateful to the Department of International Cooperation and Science Education as well as the Ministry of Science and Technology of the Republic of China for their financial support to carry out this work, under Grant number: MOST 104-2511-S-149-001-. The author also wish to thank the Aim for the Top University (ATU) project of National Taiwan Normal University (NTNU), sponsored by the Ministry of Education of the Republic of China.

Author information

Authors and Affiliations

  1. Department of Computer Science and Information Engineering, Chang Jung Christian University, Gueiren, Tainan City, 71101, Taiwan

    Chun-Chia Wang

  2. Department of Information Technology, Overseas Chinese University, Taichung, 40721, Taiwan

    Jason C. Hung

  3. Department of Information Communication, Asia University, Wufeng, Taichung, 41354, Taiwan

    Shih-Nung Chen

  4. Department of Information and Library Science, Tamkang University, Tamsui, New Taipei City, 25137, Taiwan

    Hsuan-Pu Chang

Authors
  1. Chun-Chia Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jason C. Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shih-Nung Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hsuan-Pu Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason C. Hung.

Ethics declarations

Conflict of interest

No conflicts of interest have been declared.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, CC., Hung, J.C., Chen, SN. et al. Tracking students’ visual attention on manga-based interactive e-book while reading: an eye-movement approach. Multimed Tools Appl 78, 4813–4834 (2019). https://doi.org/10.1007/s11042-018-5754-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5754-6

Keywords

Search

Navigation