Advertisement

Tracking students’ visual attention on manga-based interactive e-book while reading: an eye-movement approach

  • Chun-Chia Wang
  • Jason C. Hung
  • Shih-Nung Chen
  • Hsuan-Pu Chang
Article
  • 112 Downloads

Abstract

This study employed an eye tracking technology to explore university students’ visual attention and learning performance while learning Japanese using an interactive manga-based e-book. The developed e-book consisted of 8 pages accompanied by 13 annotations with both text and graphical formats. The subjects consisted of 60 students whose eye movements were tracked and recorded by the eye tracking system. These students came from the applied foreign language department in a northern university in Taiwan, of which 30 were assigned to high prior knowledge (PK) group and the other 30 were assigned to low PK group. Eye tracking measurements, including total contact time, number of fixations, latency of first fixation, and number of clicks on the defined regions of interest of the two groups were compared to indicate their visual attention. The results revealed that overall students spent more time on reading text and annotation than graphic information. The high PK students showed longer fixation durations on the texts, while the low PK students showed longer fixation durations on the graphics and annotations. Meanwhile, the low PK students used more clicks to look up underlined annotations whenever they didn’t know words or phrases on the e-book. In addition, with respect to the latency of the first fixation, the graphic captured the attention faster than the text because of the size and its appeal to the students. Further analysis of saccade paths indicated that the low PK students showed more inter-scanning transitions not only between the text dialog and the annotation zone but also within annotation zone. Finally, the results of reading comprehension pretest and posttest found that there was a significant difference in learning outcomes between each PK group.

Keywords

Eye tracking Prior knowledge Visual attention Hypermedia system Cognitive analysis 

Notes

Acknowledgements

The author is grateful to the Department of International Cooperation and Science Education as well as the Ministry of Science and Technology of the Republic of China for their financial support to carry out this work, under Grant number: MOST 104-2511-S-149-001-. The author also wish to thank the Aim for the Top University (ATU) project of National Taiwan Normal University (NTNU), sponsored by the Ministry of Education of the Republic of China.

Compliance with ethical standards

Conflict of interest

No conflicts of interest have been declared.

References

  1. 1.
    Adams J (2001) A critical study of comics. The International Journal of Art & Design Education 20(2):133–143CrossRefGoogle Scholar
  2. 2.
    Alexander PA, Jetton TL (2000) Learning from text: a multidimensional and developmental perspective. In: Kamil ML, Mosenthal PB, Pearson PD, Barr R (eds) Handbook of reading research, vol 3. Erlbaum, Mahwah, pp 285–310Google Scholar
  3. 3.
    Black RW (2005) Access and affiliation: the literacy and composition practices of English-language learners in an on-line fanfiction community. Journal of Adolescent & Adult Literacy 49(2):118–128CrossRefGoogle Scholar
  4. 4.
    Bryce M, Davis J, Barber C (2008) The cultural biographies and social lives of manga: lessons from the mangaverse. Journal of Media Arts Culture Collection 5(2). Retrieved from) http://scan.net.au/scan/journal/display.php?journal_id=114
  5. 5.
    Calvo MG, Lang PJ (2004) Gaze patterns when looking at emotional pictures: motivationally biased attention. Motiv Emot 28(3):221–243CrossRefGoogle Scholar
  6. 6.
    Canham M, Hegarty M (2010) Effects of knowledge and display design on comprehension of complex graphics. Learn Instr 20(2):155–166CrossRefGoogle Scholar
  7. 7.
    Cary S (2004) Going graphic: Comics at work in the multilingual classroom. Heinemann, PortsmouthGoogle Scholar
  8. 8.
    Chen YC, Hwang RH, Wang CY (2012) Development and evaluation of a web 2.0 annotation system as a learning tool in an e-learning environment. Comput Educ 58(4):1094–1105CrossRefGoogle Scholar
  9. 9.
    Chi MTH, VanLehn KA (2012) Seeing deep structure from the interactions of surface features. Educ Psychol 47(3):177–188CrossRefGoogle Scholar
  10. 10.
    Chou IC (2014) Investigating EFL students’ E-book reading attitudes in first and second language. US-China Foreign. Language 12(1):64–74MathSciNetGoogle Scholar
  11. 11.
    Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Lawrence Erlbaum Associates, New JerseyMATHGoogle Scholar
  12. 12.
    Coiro J (2011) Predicting reading comprehension on the internet: contributions of offline reading skills, online reading skills, and prior knowledge. J Lit Res 43(4):352–392CrossRefGoogle Scholar
  13. 13.
    Cook M, Carter GN, Wiebe E (2008) The interpretation of cellular transport graphics by students with low and high prior knowledge. Int J Sci Educ 30(2):241–263CrossRefGoogle Scholar
  14. 14.
    Doty DE, Popplewell SR, Byers GO (2001) Interactive CD-ROM storybooks young reader’s reading comprehension. J Res Comput Educ 3(4):374–384CrossRefGoogle Scholar
  15. 15.
    Epelboim J, Suppes P (1997) Eye movements during geometrical problem solving. Paper presented at the 19th Annual Conference of the Cognitive Science SocietyGoogle Scholar
  16. 16.
    Grant ER, Spivey MJ (2003) Eye movements and problem solving guiding attention guides thought. Psychol Sci 14(5):462–466CrossRefGoogle Scholar
  17. 17.
    Greene JA, Costa LJ, Robertson J, Pan Y, Deekens VM (2010) Exploring relations among college students’ prior knowledge, implicit theories of intelligence, and self-regulated learning in a hypermedia environment. Comput Educ 55:1027–1043CrossRefGoogle Scholar
  18. 18.
    Hegarty M, Mayer RE, Green C (1992) Comprehension of arithmetic word problems: evidence from students’ eye fixations. J Educ Psychol 84(1):76–84CrossRefGoogle Scholar
  19. 19.
    Hegarty M, Mayer RE, Monk CA (1995) Comprehension of arithmetic word problems: a comparison of successful and unsuccessful problem solvers. J Educ Psychol 87(1):18–32CrossRefGoogle Scholar
  20. 20.
    Hewig J, Trippe RH, Hecht H, Straube T, Miltner WHR (2008) Gender differences for specific body regions when looking at men and women. J Nonverbal Behav 32(2):67–78CrossRefGoogle Scholar
  21. 21.
    Higgins NC, Cocks P (1999) The effects of animation cues on vocabulary development. Journal of Reading Psychology 20:1–10CrossRefGoogle Scholar
  22. 22.
    Ho HN, Tsai MJ, Wang CY, Tsai CC (2014) Prior knowledge and online inquiry-based science reading: evidence from eye tracking. Int J Sci Math Educ 12:525–554CrossRefGoogle Scholar
  23. 23.
    Huang TH (2014) A study on the effects of interactive e-books on Taiwan high/vocational school students’ reading comprehension. Social Science Research Network (SSRN), Retrieved from SSRN https://ssrn.com/abstract=2462830 Google Scholar
  24. 24.
    Humphrey K, Underwood G (2009) Domain knowledge moderates the influence of visual saliency in scene recognition. Br J Psychol 100(2):377–398CrossRefGoogle Scholar
  25. 25.
    Hwang WY, Hsu GL (2011) The effects of pre-reading and haring mechanisms on learning with the use of annotations. Turkish Online Journal of Educational Technology 10(2):234–249Google Scholar
  26. 26.
    Hwang WY, Wang CY, Sharples M (2007) A study of multimedia annotation of web-based materials. Comput Educ 48(4):680–699CrossRefGoogle Scholar
  27. 27.
    Hwang WY, Chen NS, Shadiev R, Li JS (2011a) Effects of reviewing annotations and homework solutions on math learning achievement. Br J Educ Technol 42(6):1016–1028CrossRefGoogle Scholar
  28. 28.
    Hwang WY, Shadiev R, Huang SM (2011b) A study of a multimedia web annotation system and its effect on the EFL writing and speaking performance of junior high school students. ReCALL 23:160–180CrossRefGoogle Scholar
  29. 29.
    Hwang WY, Liu YF, Chen HR, Huang JW, Li JY (2015) Role of parents and annotation sharing in children’s learning behavior and achievement using E-readers. Educational Technology & Society 18(1):292–307Google Scholar
  30. 30.
    Hyönä, J. Jr., Lorch, R.F., & Kaakinen, J.K. (2002). Individual differences in reading to summarize expository text: evidence from eye fixation patterns. J Educ Psychol, 94, 44–55Google Scholar
  31. 31.
    Jarodzka H, Scheiter K, Gerjets P, van Gog T (2010) In the eyes of the beholder: how experts and novices interpret dynamic stimuli. Learn Instr 20(2):146–154CrossRefGoogle Scholar
  32. 32.
    Jarodzka H, Holmqvist K, Gruber H (2017) Eye tracking in educational science: theoretical frameworks and research agendas. J Eye Mov Res 10(1):1–18Google Scholar
  33. 33.
    Just MA, Carpenter PA (1980) A theory of reading: from eye fixations to comprehension. Psychol Rev 87:329–354CrossRefGoogle Scholar
  34. 34.
    Just MA, Carpenter PA (1985) Cognitive coordinate systems: Accounts of mental rotation and individual differences in spatial ability. Psychol Review 92(2):137-172Google Scholar
  35. 35.
    Khurana S (2005) So you want to be a superhero? How the art of making comics in an afterschool setting can develop young people‘s creativity, literacy and identity. Afterschool Matters 4:1–9Google Scholar
  36. 36.
    Knoblich G, Ohlsson S, Raney GE (2001) An eye movement study of insight problem solving. Mem Cogn 29(7):1000–1009CrossRefGoogle Scholar
  37. 37.
    Korat O, Shamir A (2004) Do Hebrew electronic books differs from Dutch electronic books? A replication of a Dutch content analysis. J Comput Assist Learn 20(4):257–268CrossRefGoogle Scholar
  38. 38.
    Kristof R, Satran A (1995) Interactivity by design: creating & communicating with new media. Adobe Press, Mountain ViewGoogle Scholar
  39. 39.
    Lai ML, Tsai MJ, Yang FY, Hsu CY, Liu TC, Lee SWY, Lee MH, Chiou GL, Liang JC, Tsai CC (2013) A review of using eye-tracking technology in exploring learning from 2000 to 2012. Educational Research Review 10:90–115CrossRefGoogle Scholar
  40. 40.
    Lin IY (2009) The effect of E-books on EFL learners’ reading attitude. Master thesis, National Taiwan Normal University, Taiwan. Retrieved from http://ndltd.ncl.edu.tw/cgi-bin/gs32/gsweb.cgi?o=dnclcde&s=id=%22097NTNU5238009%22.&searchmode=basic
  41. 41.
    Lin YT, Wu CC, Hou TY, Lin YC, Yang FY, Chang CH (2016) Tracking students’ cognitive processes during program debugging-an eye-movement approach. IEEE Trans Educ 59(3):175–186CrossRefGoogle Scholar
  42. 42.
    Liu HC, Andre T, Greenbowe T (2008) The impact of learner’s prior knowledge on their use of chemistry computer simulations: a case study. J Sci Educ Technol 17:466–482CrossRefGoogle Scholar
  43. 43.
    Liversedge S, Paterson K, Pickering M (1998) Eye movements and measures of reading time. In: Underwood G (ed) Eye guidance in reading and scene perception. Elsevier, Oxford, pp 55–75CrossRefGoogle Scholar
  44. 44.
    Mason L, Pluchino P, Tornatora MC, Ariasi N (2013) An eye-tracking study of learning from science text with concrete and abstract illustrations. J Exp Educ 81(3):356–384CrossRefGoogle Scholar
  45. 45.
    Mayer RE (1997) Multimedia learning: are we asking the right question? Educ Psychol 32(1):1–19CrossRefGoogle Scholar
  46. 46.
    Mayer RE (2005) The Cambridge handbook of multimedia learning. Cambridge University Press, New YorkCrossRefGoogle Scholar
  47. 47.
    Mayer RE (2008) Applying the science of learning: evidence-based principles for the design of multimedia instruction. Am Psychol 63(8):760–769CrossRefGoogle Scholar
  48. 48.
    Mayer RE (2009) Multimedia learning, 2nd edn. Cambridge University Press, New YorkCrossRefGoogle Scholar
  49. 49.
    Mayer RE (2011) Applying the science of learning. Pearson, Upper Saddle RiverGoogle Scholar
  50. 50.
    Mayer RE, Sims VK (1994) For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. J Educ Psychol 86:389–401CrossRefGoogle Scholar
  51. 51.
    Mintzes J, Wandersee JH, Novak JD (eds) (2000) Assessing science understanding (pp. 304–341). Academic Press, San DiegoGoogle Scholar
  52. 52.
    Moist KM, Bartholomew M (2007) When pigs fly: anime, auteurism, and Miyazaki’s Porco Rosso. SAGE Publications, LondonGoogle Scholar
  53. 53.
    Murakami S, Bryce M (2009) Manga as an educational medium. The International Journal of the Humanities 7(10):45–55Google Scholar
  54. 54.
    Ogawa E (2013) Educational manga: an effective medium for english learning. In: Sonda N, Krause A (eds) Paper presented at JALT2012 conference proceedings. Tokyo, JALTGoogle Scholar
  55. 55.
    Paivio A (1986) Mental representations: a dual coding approach. Oxford University Press, OxfordGoogle Scholar
  56. 56.
    Paivio A (2006) Mind and its evolution: A dual coding theoretical interpretation. Lawrence Erlbaum Associates, Inc, MahwahGoogle Scholar
  57. 57.
    Pan TW, Hsu MC, Tsai MJ (2013) Effect of graphic design on E-book reading: a pilot eye-tracking study. In: Tan SC et al (eds) Paper presented at the workshop proceedings of the 21st international conference on computers in education, IndonesiaGoogle Scholar
  58. 58.
    Pass F, van Merriënboer JJG (1994) Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. J Educ Psychol 86(1):122-133Google Scholar
  59. 59.
    Paas F, Renkl A, Sweller J (2003) Cognitive load theory and instructional design: recent developments. Educ Psychol 38:1-4Google Scholar
  60. 60.
    Peverly ST, Vekaria PC, Reddington LA, Sumowski JF, Johnson KR, Ramsay CM (2013) The relationship of handwriting speed, working memory, language comprehension and outlines to lecture note-taking and test-taking among college students. Appl Cogn Psychol 27(1):115–126CrossRefGoogle Scholar
  61. 61.
    Plass JL, Moreno R, Brunken R (eds) (2010) Cognitive load theory. Cambridge University Press, New YorkGoogle Scholar
  62. 62.
    Potter MC, Wyble B, Hagmann CE, McCourt ES (2014) Detecting meaning in RSVP at 13 ms per picture. Atten Percept Psychophys 76:270–279CrossRefGoogle Scholar
  63. 63.
    Rawlins GJE (1993) Publishing over the new decade. J Am Soc Inf Sci 44(8):474–479CrossRefGoogle Scholar
  64. 64.
    Rayner K (1998) Eye movements in reading and information processing: 20 years of research. Psychol Bull 124(3):372–422CrossRefGoogle Scholar
  65. 65.
    Rayner K (2009) Eye movements and attention in reading, scene perception, and visual search. Q J Exp Psychol 62(8):1457–1506CrossRefGoogle Scholar
  66. 66.
    Rayner K, Rotello CM, Stewart AJ, Keir J, Duffy SA (2001) Integrating text and pictorial information: eye movements when looking at print advertisements. J Exp Psychol Appl 7:219–226CrossRefGoogle Scholar
  67. 67.
    Rayner K, Smith T, Malcolm GL, Henderson JM (2009) Eye movements and visual encoding during scene perception. Psychol Sci 20(1):6–10CrossRefGoogle Scholar
  68. 68.
    Rayner K, Slattery TJ, Bélanger NN (2010) Eye movements, the perceptual span, and reading speed. Psychon Bull Rev 17(6):834–839CrossRefGoogle Scholar
  69. 69.
    Reingold EM, Sheridan H (2011) Eye movements and visual expertise in chess and medicine. In: Liversedge SP, Gilchrist ID, Everling S (eds) Oxford handbook on eye movements. Oxford University Press, Oxford, pp 528–550Google Scholar
  70. 70.
    Riley P (2003) Drawing the threads together. In: Little D, Ridley J, Ushioda E (eds) Learner autonomy in the foreign language classroom: teacher, learner, curriculum and assessment. Dublin, Authentik, pp 237–252Google Scholar
  71. 71.
    Sadoski M, Willson VL (2006) Effects of a theoretically-based large scale reading intervention in a multicultural urban school district. Am Educ Res J 43:137–154CrossRefGoogle Scholar
  72. 72.
    Sanders MS, McCormick EJ (1987) Human factors in engineering and design. McGraw-Hill, New YorkGoogle Scholar
  73. 73.
    She HC, Chen YZ (2009) The impact of multimedia effect on science learning: evidence from eye movements. Comput Educ 53(4):1297–1307CrossRefGoogle Scholar
  74. 74.
    Shimada A, Okubo F, Yin CJ, Oi M, Kojima K, Yamada M, Ogata H (2015) Analysis of preview behavior in E-book system. In: Ogata H et al (eds) Paper presented at the 23rd international conference on computers in education (ICCE 2015), Hangzhou, ChinaGoogle Scholar
  75. 75.
    Smith, S.D., Caruso, J.B., & Kim, J. (2010). The ECAR study of undergraduate students and information technology. ECAR Research Study, 6, 1–120. Retrieved from http://library.educause.edu/~/media/files/library/2010/10/ers1006w-pdf.pdf
  76. 76.
    Sweller J (1988) Cognitive load during problem solving: effects on learning. Cogn Sci 12(2):257–285CrossRefGoogle Scholar
  77. 77.
    Thompson RA, Zamboanga BL (2003) Prior knowledge and its relevance to student achievement in introduction to psychology. Teach Psychol 30:96–101CrossRefGoogle Scholar
  78. 78.
    Tsai JL, Yen MH, Wang CA (2005) Eye movement recording and the application in research of reading Chinese. Research in Applied Psychology 28:91–104Google Scholar
  79. 79.
    Tsai MJ, Hou HT, Lai ML, Liu WY, Yang FY (2011) Visual attention for solving multiple-choice science problem: an eye-tracking analysis. Comput Educ 58:375–385CrossRefGoogle Scholar
  80. 80.
    Tsai MJ, Huang LJ, Hou HT, Hsu CY, Chiou GL (2016) Visual behavior, flow and achievement in game-based learning. Comput Educ 98:115–129CrossRefGoogle Scholar
  81. 81.
    Van Gog T, Scheiter K (2010) Eye tracking as a tool to study and enhance multimedia learning. Learn Instr 20(2):95–99CrossRefGoogle Scholar
  82. 82.
    Van Gog T, Kester L, Nievelstein F, Giesbers B, Paas F (2009) Uncovering cognitive processes: different techniques that can contribute to cognitive load research and instruction. Comput Hum Behav 25(2):325–331CrossRefGoogle Scholar
  83. 83.
    Wolfe, P. (2010). Brain matters: Translating research into classroom practice (2nd Ed.) Alexandria: Association for Supervision & curriculum developmentGoogle Scholar
  84. 84.
    Wu AH, Hsu PF, Chiu HJ, Tsai MJ (2014) Visual behavior and cognitive load on E-book vocabulary learning. In: Liu CC et al (eds) Paper presented at the workshop proceedings of the 22nd international conference on computers in education, JapanGoogle Scholar
  85. 85.
    Yang FY, Chang CY, Chien WR, Chien YT, Tseng YH (2013) Tracking learners’ visual attention during a multimedia presentation in a real classroom. Comput Educ 62:208–220CrossRefGoogle Scholar
  86. 86.
    Yin CJ, Okubo F, Shimada A, Oi M, Hirokawa S, Ogata H (2015) Identifying and analyzing the learning behaviors of students using e-books. In: Ogata H et al (eds) Paper presented at the workshop proceedings of the 23rd international conference on computers in education (ICCE 2015), Hangzhou, ChinaGoogle Scholar
  87. 87.
    Zheng HT, Wang Z, Ma N, Chen J, Xiao X, Sangaiah AK (2017) Weakly-supervised image captioning based on rich contextual information. Multimedia Tools and Applications:1–17Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Chun-Chia Wang
    • 1
  • Jason C. Hung
    • 2
  • Shih-Nung Chen
    • 3
  • Hsuan-Pu Chang
    • 4
  1. 1.Department of Computer Science and Information EngineeringChang Jung Christian UniversityTainan CityTaiwan
  2. 2.Department of Information TechnologyOverseas Chinese UniversityTaichungTaiwan
  3. 3.Department of Information CommunicationAsia UniversityTaichungTaiwan
  4. 4.Department of Information and Library ScienceTamkang UniversityNew Taipei CityTaiwan

Personalised recommendations