Applications of Eye Tracking in Language Processing and Other Areas

  • Abhijit MishraEmail author
  • Pushpak Bhattacharyya
Part of the Cognitive Intelligence and Robotics book series (CIR)


This chapter presents an overview on applications of eye-tracking technology in natural language processing (NLP) and other areas. While traditional NLP techniques mostly rely on the textual properties, recent research has shown that human behavioral data collected through technologies like eye tracking, electroencephalography (EEG), and magnetoencephalography (MEG) along with textual representations help improve performances of NLP systems. Our survey focuses particularly on eye-tracking technology and summarizes various methods proposed to include eye-movement data in different components of NLP pipeline viz. annotation, classification, and evaluation. A significant portion of this chapter is also devoted to more than three decades of eye-movement research and development in the fields of psychology, psycholinguistics, neuroscience, industrial engineering, marketing, user experience design, to build a perspective on why eye-movement information can be effective toward solving important problems in different research areas.


  1. Abdelali, A., Durrani, N., & Guzmán, F. (2016). iAppraise: A manual machine translation evaluation environment supporting eye-tracking. In NAACL HLT 2016 (p. 17).Google Scholar
  2. Barrett, M., Bingel, J., Keller, F., & Søgaard, A. (2016). Weakly supervised part-of-speech tagging using eye-tracking data. In The 54th Annual Meeting of the Association for Computational Linguistics (p. 579).Google Scholar
  3. Berg-Kirkpatrick, T., Gillick, D., & Klein, D. (2011). Jointly learning to extract and compress. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (Vol. 1, pp. 481–490). Association for Computational Linguistics.Google Scholar
  4. Bicknell, K., & Levy, R. (2010). A rational model of eye movement control in reading. In Proceedings of the 48th Annual Meeting of the ACL (pp. 1168–1178). ACL.Google Scholar
  5. Bojko, A. (2005). Eye tracking in user experience testing: How to make the most of it. In Proceedings of the UPA 2005 Conference.Google Scholar
  6. Brill, E. (1995). Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4), 543–565.MathSciNetGoogle Scholar
  7. Carl, M., Dragsted, B., Elming, J., Hardt, D., & Jakobsen, A. L. (2011). The process of post-editing: a pilot study. In Proceedings of the 8th International NLPSC Workshop. Special Theme: Human-Machine Interaction in Translation (Vol. 41, pp. 131–142).Google Scholar
  8. Chapman, P. R., & Underwood, G. (1998). Visual search of dynamic scenes: Event types and the role of experience in viewing driving situations. Eye guidance in reading and scene perception (pp. 369–393). Oxford: Elsevier.Google Scholar
  9. Christodoulopoulos, C., Goldwater, S., & Steedman, M. (2010). Two decades of unsupervised POS induction: How far have we come? In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (pp. 575–584). Association for Computational Linguistics.Google Scholar
  10. Clarke, J., & Lapata, M. (2006). Constraint-based sentence compression an integer programming approach. In Proceedings of the COLING/ACL on Main Conference Poster Sessions (pp. 144–151). Association for Computational Linguistics.Google Scholar
  11. Cohn, T., & Lapata, M. (2008). Sentence compression beyond word deletion. In Proceedings of the 22nd International Conference on Computational Linguistics (Vol. 1, pp. 137–144). Association for Computational Linguistics.Google Scholar
  12. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(Aug), 2493–2537.Google Scholar
  13. Corcoran, P. M., Nanu, F., Petrescu, S., & Bigioi, P. (2012). Real-time eye gaze tracking for gaming design and consumer electronics systems. IEEE Transactions on Consumer Electronics, 58(2), 347–355.Google Scholar
  14. Cristino, F., Mathôt, S., Theeuwes, J., & Gilchrist, I. D. (2010). Scanmatch: A novel method for comparing fixation sequences. Behavior Research Methods, 42(3), 692–700.CrossRefGoogle Scholar
  15. Cutting, D., Kupiec, J., Pedersen, J., & Sibun, P. (1992). A practical part-of-speech tagger. In Proceedings of the Third Conference on Applied Natural Language Processing (pp. 133–140). Association for Computational Linguistics.Google Scholar
  16. Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2), 193–210.CrossRefGoogle Scholar
  17. Djamasbi, S. (2014). Eye tracking and web experience. AIS Transactions on Human-Computer Interaction, 6(2), 37–54.CrossRefGoogle Scholar
  18. Doherty, S., O’Brien, S., & Carl, M. (2010). Eye tracking as an mt evaluation technique. Machine Translation, 24(1), 1–13.CrossRefGoogle Scholar
  19. Dragsted, B., & Carl, M. (2013). Towards a classification of translation styles based on eye-tracking and keylogging data. Journal of Writing Research, 5(1), 133–158Google Scholar
  20. Dragsted, B. (2010). Coordination of reading and writing processes in translation. Translation and Cognition, 15, 41.CrossRefGoogle Scholar
  21. Duchowski, A. T. (2002). A breadth-first survey of eye-tracking applications. Behavior Research Methods, Instruments, & Computers, 34(4), 455–470.CrossRefGoogle Scholar
  22. Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). Swift: A dynamical model of saccade generation during reading. Psychological Review, 112(4), 777.CrossRefGoogle Scholar
  23. Filippova, K., & Strube, M. (2008). Dependency tree based sentence compression. In Proceedings of the Fifth International Natural Language Generation Conference (pp. 25–32). Association for Computational Linguistics.Google Scholar
  24. Findlay, J. M., & Walker, R. (1999). A model of saccade generation based on parallel processing and competitive inhibition. Behavioral and Brain Sciences, 22(04), 661–674.Google Scholar
  25. Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14(2), 178–210.CrossRefGoogle Scholar
  26. Frisson, S., & Pickering, M. J. (1999). The processing of metonymy: Evidence from eye movements. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(6), 1366.Google Scholar
  27. Gibson, E. (1998). Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1), 1–76.CrossRefGoogle Scholar
  28. Jacob, R., & Karn, K. S. (2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind, 2(3), 4.Google Scholar
  29. Joshi, S., Kanojia, D., & Bhattacharyya, P. (2013). More than meets the eye: Study of human cognition in sense annotation. In NAACL HLT 2013. Atlanta, USA.Google Scholar
  30. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329.CrossRefGoogle Scholar
  31. Kennedy, A., Hill, R., & Pynte, J. (2003). The Dundee corpus. In Proceedings of the 12th European Conference on Eye Movement.Google Scholar
  32. Klerke, S., Goldberg, Y., & Søgaard, A. (2016). Improving sentence compression by learning to predict gaze. arXiv:1604.03357.Google Scholar
  33. Kliegl, R., Grabner, E., Rolfs, M., & Engbert, R. (2004). Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cognitive Psychology, 16(1–2), 262–284.CrossRefGoogle Scholar
  34. Knight, K., & Marcu, D. (2000). Statistics-based summarization-step one: Sentence compression. AAAI/IAAI, 2000, 703–710.Google Scholar
  35. Knight, K., & Marcu, D. (2002). Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence, 139(1), 91–107.CrossRefGoogle Scholar
  36. Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1), 1–27.MathSciNetCrossRefGoogle Scholar
  37. Li, S., Graça, J. V., & Taskar, B. (2012). Wiki-ly supervised part-of-speech tagging. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 1389–1398). Association for Computational Linguistics.Google Scholar
  38. Lohse, G. L. (1997). Consumer eye movement patterns on yellow pages advertising. Journal of Advertising, 26(1), 61–73.CrossRefGoogle Scholar
  39. Mishra, A., Bhattacharyya, P., & Carl, M. (2013). Automatically predicting sentence translation difficulty. In ACL (2), (pp. 346–351) (CRITT, I.).Google Scholar
  40. Mishra, A., Joshi, A., & Bhattacharyya, P. (2014). A cognitive study of subjectivity extraction in sentiment annotation. In ACL 2014 (p. 142).Google Scholar
  41. Navigli, R. (2009). Word sense disambiguation: A survey. ACM Computing Surveys (CSUR), 41(2), 10.CrossRefGoogle Scholar
  42. Needleman, S. B., & Wunsch, C. D. (1970). A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of Molecular Biology, 48(3), 443–453.CrossRefGoogle Scholar
  43. O’Brien, S. (2009). Eye tracking in translation process research: Methodological challenges and solutions. Methodology, Technology and Innovation in Translation Process Research, 38, 251–266.Google Scholar
  44. O’Sullivan, C., Dingliana, J., & Howlett, S. (2002). Gaze-contingent algorithms for interactive graphics. In J. Hyönä, R. Radach, & H. Deubel (Eds.), The mind’s eyes: cognitive and applied aspects of eye movement research. Oxford: Elsevier Science.Google Scholar
  45. Özyurt, J., DeSouza, P., West, P., Rutschmann, R., & Greenlee, M. (2001). Comparison of cortical activity and oculomotor performance in the gap and step paradigms. In European Conference on Visual Perception (ECVP) (Vol. 30)Google Scholar
  46. Palinko, O., Kun, A. L., Shyrokov, A., & Heeman, P. (2010). Estimating cognitive load using remote eye tracking in a driving simulator. In Proceedings of the 2010 Symposium on Eye-tracking Research & Applications (pp. 141–144). ACM.Google Scholar
  47. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (pp. 311–318). Association for Computational Linguistics.Google Scholar
  48. Pavlović, N., & Jensen, K. (2009). Eye tracking translation directionality. Translation Research Projects, 2, 93.Google Scholar
  49. Pickering, M. J., & Frisson, S. (2001). Processing ambiguous verbs: Evidence from eye movements. Journal of Experimental Psychology Learning Memory and Cognition, 27(2), 556–573.CrossRefGoogle Scholar
  50. Posner, M. I., Snyder, C. R., & Davidson, B. J. (1980). Attention and the detection of signals. Journal of Experimental Psychology: General, 109(2), 160.CrossRefGoogle Scholar
  51. Ratnaparkhi, A., et al. (1996). A maximum entropy model for part-of-speech tagging. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (Vol. 1, pp. 133–142). Philadelphia, USA.Google Scholar
  52. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124(3), 372.CrossRefGoogle Scholar
  53. Rayner, K., & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14(3), 191–201.CrossRefGoogle Scholar
  54. Rayner, K., Rotello, C. M., Stewart, A. J., Keir, J., & Duffy, S. A. (2001). Integrating text and pictorial information: Eye movements when looking at print advertisements. Journal of Experimental Psychology: Applied, 7(3), 219.Google Scholar
  55. Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105(1), 125.CrossRefGoogle Scholar
  56. Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The E-Z reader model of eye-movement control in reading: Comparisons to other models. Behavioral and Brain Sciences, 26(04), 445–476.CrossRefGoogle Scholar
  57. Reichle, E. D., Pollatsek, A., & Rayner, K. (2006). E-Z reader: A cognitive-control, serial-attention model of eye-movement behavior during reading. Cognitive Systems Research, 7(1), 4–22.CrossRefGoogle Scholar
  58. Robinson, D. A. (1968). The oculomotor control system: A review. Proceedings of the IEEE, 56(6), 1032–1049.CrossRefGoogle Scholar
  59. Rosbergen, E., Wedel, M., & Pieters, F. G. M. (1990). Analyzing visual attention to repeated print advertising using scanpath theory (Technical Report No. 97B32). University Library Groningen, SOM Research School.Google Scholar
  60. Sajjad, H., Guzmán, F., Durrani, N., Abdelali, A., Bouamor, H., Temnikova, I., et al. (2016). Eyes don’t lie: Predicting machine translation quality using eye movement. In Proceedings of NAACL-HLT (pp. 1082–1088).Google Scholar
  61. Sarter, N. B., Mumaw, R. J., & Wickens, C. D. (2007). Pilots’ monitoring strategies and performance on automated flight decks: An empirical study combining behavioral and eye-tracking data. Human Factors: The Journal of the Human Factors and Ergonomics Society, 49(3), 347–357.CrossRefGoogle Scholar
  62. Scott, G. G., O’Donnell, P. J., & Sereno, S. C. (2012). Emotion words affect eye fixations during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 38(3), 783.Google Scholar
  63. Sibert, L. E., & Jacob, R. J. (2000). Evaluation of eye gaze interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 281–288). ACM.Google Scholar
  64. Stymne, S., Danielsson, H., Bremin, S., Hu, H., Karlsson, J., Lillkull, A. P., et al. (2012). Eye tracking as a tool for machine translation error analysis. LREC (pp. 1121–1126).Google Scholar
  65. Surakka, V., Illi, M., & Isokoski, P. (2004). Gazing and frowning as a new human-computer interaction technique. ACM Transactions on Applied Perception (TAP), 1(1), 40–56.CrossRefGoogle Scholar
  66. Tomanek, K., Hahn, U., Lohmann, S., & Ziegler, J. (2010). A cognitive cost model of annotations based on eye-tracking data. In Proceedings of the 48th Annual Meeting of the ACL (pp. 1158–1167). ACL.Google Scholar
  67. Underwood, G., Clews, S., & Everatt, J. (1990). How do readers know where to look next? Local information distributions influence eye fixations. The Quarterly Journal of Experimental Psychology, 42(1), 39–65.CrossRefGoogle Scholar
  68. Van Schijndel, M., & Schuler, W. (2015). Hierarchic syntax improves reading time prediction. In HLT-NAACL (pp. 1597–1605).Google Scholar
  69. Von der Malsburg, T., & Vasishth, S. (2011). What is the scanpath signature of syntactic reanalysis? Journal of Memory and Language, 65(2), 109–127.CrossRefGoogle Scholar
  70. Von der Malsburg, T., Kliegl, R., & Vasishth, S. (2015). Determinants of scanpath regularity in reading. Cognitive Science, 39(7), 1675–1703.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.India Research LabIBM ResearchBangaloreIndia
  2. 2.Indian Institute of Technology PatnaPatnaIndia

Personalised recommendations