Investigating the Potential of EEG for Implicit Detection of Unknown Words for Foreign Language Learning

  • Christina SchneegassEmail author
  • Thomas Kosch
  • Albrecht Schmidt
  • Heinrich Hussmann
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11748)


Ubiquitous technologies change the way we learn new languages. They provide easy access to multilingual media content within everyday scenarios to enhance language skills or improve vocabulary. The detection of learners’ vocabulary deficiencies is an integral part of effective language learning. However, retrieving translations during everyday media consumption causes inattention and hinders fluent learning. In this paper, we investigate Electroencephalography (EEG) to assess single-word incomprehension within non-native learning contents. In a user study (N = 10), we employed Rapid Serial Visual Presentation (RSVP) to display text while recording Event-Related Potentials (ERPs) for different word difficulties. Our results show that incomprehension of words can be detected in participants’ neural responses, as is confirmed by post hoc ratings. We conclude with use case scenarios in which our approach can facilitate seamless vocabulary deficiency detection in everyday life.


EEG Language proficiency detection Implicit interaction 


  1. 1.
    An, X., Kuang, D., Guo, X., Zhao, Y., He, L.: A deep learning method for classification of EEG data based on motor imagery. In: Huang, D.-S., Han, K., Gromiha, M. (eds.) ICIC 2014. LNCS, vol. 8590, pp. 203–210. Springer, Cham (2014). Scholar
  2. 2.
    Ardal, S., Donald, M.W., Meuter, R., Muldrew, S., Luce, M.: Brain responses to semantic incongruity in bilinguals. Brain Lang. 39(2), 187–205 (1990)CrossRefGoogle Scholar
  3. 3.
    Baillet, S., Mosher, J.C., Leahy, R.M.: Electromagnetic brain mapping. IEEE Signal Process. Mag. 18(6), 14–30 (2001)CrossRefGoogle Scholar
  4. 4.
    Bentin, S., McCarthy, G., Wood, C.C.: Event-related potentials, lexical decision and semantic priming. Electroencephalogr. Clin. Neurophysiol. 60(4), 343–355 (1985)CrossRefGoogle Scholar
  5. 5.
    Berzak, Y., Katz, B., Levy, R.: Assessing language proficiency from eye movements in reading. arXiv preprint arXiv:1804.07329 (2018)
  6. 6.
    Bleichner, M.G., et al.: Exploring miniaturized EEG electrodes for brain-computer interfaces. An EEG you do not see? Physiol. Rep. 3(4), 1–9 (2015) CrossRefGoogle Scholar
  7. 7.
    Borovsky, A., Kutas, M., Elman, J.: Learning to use words: event-related potentials index single-shot contextual word learning. Cognition 116(2), 289–296 (2010)CrossRefGoogle Scholar
  8. 8.
    Chen, C.H., Chien, Y.H.: Effects of RSVP display design on visual performance in accomplishing dual tasks with small screens. Int. J. Des. 1(1), 27–35 (2007)MathSciNetGoogle Scholar
  9. 9.
    Cheng, P., Autthasan, P., Pijarana, B., Chuangsuwanich, E., Wilaiprasitporn, T.: Towards asynchronous motor imagery-based brain-computer interfaces: a joint training scheme using deep learning. arXiv preprint arXiv:1808.10852 (2018)
  10. 10.
    Coulson, S., King, J.W., Kutas, M.: Expect the unexpected: event-related brain response to morphosyntactic violations. Lang. Cogn. Process. 13(1), 21–58 (1998)CrossRefGoogle Scholar
  11. 11.
    Cutmore, T.R., James, D.A.: Identifying and reducing noise in psychophysiological recordings. Int. J. Psychophysiol. 32(2), 129–150 (1999)CrossRefGoogle Scholar
  12. 12.
    Davidson, D.: Brain activity during second language processing (ERP). In: The Encyclopedia of Applied Linguistics (2012)Google Scholar
  13. 13.
    De, M.V., Debener, S.: Mobile EEG: towards brain activity monitoring during natural action and cognition. Int. J. Psychol. 91(1), 1–2 (2014)Google Scholar
  14. 14.
    Debener, S., Emkes, R., De Vos, M., Bleichner, M.: Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear. Sci. Rep. 5, 16743 (2015)CrossRefGoogle Scholar
  15. 15.
    Debener, S., Minow, F., Emkes, R., Gandras, K., De Vos, M.: How about taking a low-cost, small, and wireless EEG for a walk? Psychophysiology 49(11), 1617–1621 (2012)CrossRefGoogle Scholar
  16. 16.
    Dingler, T., Rzayev, R., Schwind, V., Henze, N.: RSVP on the go: implicit reading support on smart watches through eye tracking. In: Proceedings of the 2016 ACM International Symposium on Wearable Computers, pp. 116–119. ACM (2016)Google Scholar
  17. 17.
    Doherty, S.: Translations—the impact of translation technologies on the process and product of translation. Int. J. Commun. 10, 23 (2016)Google Scholar
  18. 18.
    Council of Europe, Council for Cultural Co-operation. Education Committee. Modern Languages Division: Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press, Cambridge (2001)Google Scholar
  19. 19.
    van Erp, J., Lotte, F., Tangermann, M.: Brain-computer interfaces: beyond medical applications. Computer 45(4), 26–34 (2012)CrossRefGoogle Scholar
  20. 20.
    Fan, S.P., Liberman, Z., Keysar, B., Kinzler, K.D.: The exposure advantage: early exposure to a multilingual environment promotes effective communication. Psychol. Sci. 26(7), 1090–1097 (2015)CrossRefGoogle Scholar
  21. 21.
    Fisch, B.: Fisch and Spehlmann’s EEG Primer: Basic Principles of Digital and Analog EEG, 3rd edn. Elsevier, Amsterdam (1999)Google Scholar
  22. 22.
    Fragoso, V., Gauglitz, S., Zamora, S., Kleban, J., Turk, M.: Translatar: a mobile augmented reality translator. In: 2011 IEEE Workshop on Applications of Computer Vision (WACV), pp. 497–502. IEEE (2011)Google Scholar
  23. 23.
    Frey, J., Mühl, C., Lotte, F., Hachet, M.: Review of the use of electroencephalography as an evaluation method for human-computer interaction. arXiv preprint arXiv:1311.2222 (2013)
  24. 24.
    Gannon, E., He, J., Gao, X., Chaparro, B.: RSVP reading on a smart watch. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 60, pp. 1130–1134. SAGE Publications, Los Angeles (2016)CrossRefGoogle Scholar
  25. 25.
    Hagoort, P., Hald, L., Bastiaansen, M., Petersson, K.M.: Integration of word meaning and world knowledge in language comprehension. Science 304(5669), 438–441 (2004)CrossRefGoogle Scholar
  26. 26.
    Hahne, A.: What’s different in second-language processing? Evidence from event-related brain potentials. J. Psycholinguist. Res. 30(3), 251–266 (2001)CrossRefGoogle Scholar
  27. 27.
    Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908. Sage Publications, Los Angeles (2006)CrossRefGoogle Scholar
  28. 28.
    Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in Psychology, vol. 52, pp. 139–183. Elsevier (1988)Google Scholar
  29. 29.
    Hassib, M., Pfeiffer, M., Schneegass, S., Rohs, M., Alt, F.: Emotion actuator: embodied emotional feedback through electroencephalography and electrical muscle stimulation. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 6133–6146. ACM (2017)Google Scholar
  30. 30.
    Hassib, M., Schneegass, S., Eiglsperger, P., Henze, N., Schmidt, A., Alt, F.: EngageMeter: a system for implicit audience engagement sensing using electroencephalography. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 5114–5119. ACM (2017)Google Scholar
  31. 31.
    Herron, C., Morris, M., Secules, T., Curtis, L.: A comparison study of the effects of video-based versus text-based instruction in the foreign language classroom. Fr. Rev. 68, 775–795 (1995)Google Scholar
  32. 32.
    Holcomb, P.J., Neville, H.J.: Auditory and visual semantic priming in lexical decision: a comparison using event-related brain potentials. Lang. Cogn. Process. 5(4), 281–312 (1990)CrossRefGoogle Scholar
  33. 33.
    Jasper, H.H.: The ten-twenty electrode system of the international federation. Electroencephalogr. Clin. Neurophysiol. 10, 370–375 (1958)CrossRefGoogle Scholar
  34. 34.
    Jirayucharoensak, S., Pan-Ngum, S., Israsena, P.: EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci. World J. 2014, 10 p. (2014)Google Scholar
  35. 35.
    Karolus, J., Wozniak, P.W., Chuang, L.L., Schmidt, A.: Robust gaze features for enabling language proficiency awareness. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2998–3010. ACM (2017)Google Scholar
  36. 36.
    Knittel, J., Dingler, T.: Mining subtitles for real-time content generation for second-screen applications. In: Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video, pp. 93–103. ACM (2016)Google Scholar
  37. 37.
    Kok, A.: Event-related-potential (ERP) reflections of mental resources: a review and synthesis. Biol. Psychol. 45(1–3), 19–56 (1997)CrossRefGoogle Scholar
  38. 38.
    Kolk, H.H., Chwilla, D.J., Van Herten, M., Oor, P.J.: Structure and limited capacity in verbal working memory: a study with event-related potentials. Brain Lang. 85(1), 1–36 (2003)CrossRefGoogle Scholar
  39. 39.
    Kosch, T., Funk, M., Schmidt, A., Chuang, L.: Identifying cognitive assistance with mobile electroencephalography: a case study with in-situ projections for manual assembly. In: Proceedings of the 10th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM (2018). Scholar
  40. 40.
    Kosch, T., Hassib, M., Buschek, D., Schmidt, A.: Look into my eyes: using pupil dilation to estimate mental workload for task complexity adaptation. In: Proceedings of the 2018 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2018. ACM, New York (2018).
  41. 41.
    Kosch, T., Hassib, M., Wozniak, P., Buschek, D., Alt, F.: Your eyes tell: leveraging smooth pursuit for assessing cognitive workload. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI 2018. ACM, New York (2018).
  42. 42.
    Kutas, M.: Event-related brain potentials (ERPs) elicited during rapid serial visual presentation of congruous and incongruous sentences. Electroencephalogr. Clin. Neurophysiol. 40(Suppl.), 406–411 (1987)Google Scholar
  43. 43.
    Kutas, M., Dale, A.: Electrical and magnetic readings of mental functions. Cogn. Neurosci. 53, 197–242 (1997)Google Scholar
  44. 44.
    Kutas, M., Federmeier, K.D.: Electrophysiology reveals semantic memory use in language comprehension. Trends Cogn. Sci. 4(12), 463–470 (2000)CrossRefGoogle Scholar
  45. 45.
    Kutas, M., Hillyard, S.A.: Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207(4427), 203–205 (1980)CrossRefGoogle Scholar
  46. 46.
    Legge, G.E., Mansfield, J.S., Chung, S.T.: Psychophysics of reading: XX. Linking letter recognition to reading speed in central and peripheral vision. Vis. Res. 41(6), 725–743 (2001)CrossRefGoogle Scholar
  47. 47.
    Lennon, C., Burdick, H.: The Lexile framework as an approach for reading measurement and success (2004). Electronic publication on
  48. 48.
    Liberman, Z., Woodward, A.L., Keysar, B., Kinzler, K.D.: Exposure to multiple languages enhances communication skills in infancy. Dev. Sci. 20(1), e12420 (2017)CrossRefGoogle Scholar
  49. 49.
    Mori, S., Nishida, H., Yamada, H.: Optical Character Recognition. Wiley, New York (1999)Google Scholar
  50. 50.
    Mueller, J.L.: Electrophysiological correlates of second language processing. Second Lang. Res. 21(2), 152–174 (2005)MathSciNetCrossRefGoogle Scholar
  51. 51.
    Mullen, T.R., et al.: Real-time neuroimaging and cognitive monitoring using wearable dry EEG. IEEE Trans. Biomed. Eng. 62(11), 2553–2567 (2015)CrossRefGoogle Scholar
  52. 52.
    Neuman, S.B., Koskinen, P.: Captioned television as comprehensible input: effects of incidental word learning from context for language minority students. Read. Res. Q. 27, 95–106 (1992)Google Scholar
  53. 53.
    Osterhout, L., McLaughlin, J., Kim, A., Greenwald, R., Inoue, K.: Sentences in the brain: event-related potentials as real-time reflections of sentence comprehension and language learning. In: The On-Line Study of Sentence Comprehension: Eyetracking, ERP, and Beyond, pp. 271–308 (2004)Google Scholar
  54. 54.
    Pearson, P.D., Hiebert, E.H., Kamil, M.L.: Vocabulary assessment: what we know and what we need to learn. Read. Res. Q. 42(2), 282–296 (2007)CrossRefGoogle Scholar
  55. 55.
    Proakis, J.G.: Digital Signal Processing: Principles Algorithms and Applications. Pearson Education, New Delhi (2001)Google Scholar
  56. 56.
    Quinn, E., Nation, I.S.P.: Speed Reading: A Course For Learners of English. Oxford University Press, Oxford (1974) Google Scholar
  57. 57.
    Quinn, E., Nation, I.S.P., Millett, S.: Asian and Pacific Speed Readings for ESL Learners. English Language Institute Occasional Publication, p. 24 (2007)Google Scholar
  58. 58.
    Roy, R.N., Charbonnier, S., Campagne, A., Bonnet, S.: Efficient mental workload estimation using task-independent EEG features. J. Neural Eng. 13(2), 026019 (2016)CrossRefGoogle Scholar
  59. 59.
    Rubin, G.S., Turano, K.: Reading without saccadic eye movements. Vis. Res. 32(5), 895–902 (1992)CrossRefGoogle Scholar
  60. 60.
    Sahami Shirazi, A., Funk, M., Pfleiderer, F., Glück, H., Schmidt, A.: MediaBrain: annotating videos based on brain-computer interaction. In: Mensch & Computer 2012: interaktiv informiert-allgegenwärtig und allumfassend!? (2012)Google Scholar
  61. 61.
    Sakunkoo, N., Sakunkoo, P.: GliFlix: using movie subtitles for language learning. In: Proceedings of the 26th Symposium on User Interface Software and Technology. ACM (2013)Google Scholar
  62. 62.
    Secules, T., Herron, C., Tomasello, M.: The effect of video context on foreign language learning. Mod. Lang. J. 76(4), 480–490 (1992)CrossRefGoogle Scholar
  63. 63.
    Shirazi, A.S., Hassib, M., Henze, N., Schmidt, A., Kunze, K.: What’s on your mind?: mental task awareness using single electrode brain computer interfaces. In: Proceedings of the 5th Augmented Human International Conference, p. 45. ACM (2014)Google Scholar
  64. 64.
    Stokes, T.A., Welk, A.K., Zielinska, O.A., Gillan, D.J.: The oddball effect and inattentional blindness: How unexpected events influence our perceptions of time. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 61, pp. 1753–1757. SAGE Publications, Los Angeles (2017)CrossRefGoogle Scholar
  65. 65.
    Van Cauwenberge, A., Schaap, G., Van Roy, R.: “TV no longer commands our full attention”: effects of second-screen viewing and task relevance on cognitive load and learning from news. Comput. Hum. Behav. 38, 100–109 (2014)CrossRefGoogle Scholar
  66. 66.
    Van Hell, J.G., Tokowicz, N.: Event-related brain potentials and second language learning: syntactic processing in late L2 learners at different L2 proficiency levels. Second Lang. Res. 26(1), 43–74 (2010)CrossRefGoogle Scholar
  67. 67.
    Vanderplank, R.: Resolving inherent conflicts: autonomous language learning from popular broadcast television. In: Barriers and Bridges: Media Technology in Language Learning, pp. 119–133 (1994)Google Scholar
  68. 68.
    Wise, R.J.S., Brownsett, S.L.E.: The contribution of the parietal lobes to speaking and writing. Cereb. Cortex 20(3), 517–523 (2009)Google Scholar
  69. 69.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791 (2002)CrossRefGoogle Scholar
  70. 70.
    Zani, A., Proverbio, A.M.: Cognitive electrophysiology of mind and brain. In: The Cognitive Electrophysiology of Mind and Brain, pp. 3–12. Elsevier (2003)Google Scholar
  71. 71.
    Zheng, W.L., Zhu, J.Y., Peng, Y., Lu, B.L.: EEG-based emotion classification using deep belief networks. In: 2014 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE (2014)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Christina Schneegass
    • 1
    Email author
  • Thomas Kosch
    • 1
  • Albrecht Schmidt
    • 1
  • Heinrich Hussmann
    • 1
  1. 1.LMU MunichMunichGermany

Personalised recommendations