Predicting Affect from Gaze Data during Interaction with an Intelligent Tutoring System

  • Natasha Jaques
  • Cristina Conati
  • Jason M. Harley
  • Roger Azevedo
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8474)


In this paper we investigate the usefulness of eye tracking data for predicting emotions relevant to learning, specifically boredom and curiosity. The data was collected during a study with MetaTutor, an intelligent tutoring system (ITS) designed to promote the use of self-regulated learning strategies. We used a variety of machine learning and feature selection techniques to predict students’ self-reported emotions from gaze data features. We examined the optimal amount of interaction time needed to make predictions, as well as which features are most predictive of each emotion. The findings provide insight into how to detect when students disengage from MetaTutor.


Random Forest Pupil Dilation Dynamic Bayesian Network Intelligent Tutoring System Pedagogical Agent 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    AlZoubi, O., D’Mello, S., Calvo, R.: Detecting naturalistic expressions of nonbasic affect using physiological signals (2012)Google Scholar
  2. 2.
    Anderson, J.R., Gluck, K.: What role do cognitive architectures play in intelligent tutoring systems. Cognition & Instruction, 227–262 (2001)Google Scholar
  3. 3.
    Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. In: AIED, July 6-10, pp. 17–24. IOS Press, Brighton (2009)Google Scholar
  4. 4.
    Azevedo, R., Harley, J., Trevors, G., et al.: Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems. In: IHMLT, pp. 427–449. Springer (2013)Google Scholar
  5. 5.
    Azevedo, R., Johnson, A., Chauncey, A., Burkett, C.: Self-regulated learning with MetaTutor: Advancing the science of learning with MetaCognitive tools. In: New Science of Learning, pp. 225–247. Springer (2010)Google Scholar
  6. 6.
    Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., Bouchet, F.: Inferring Learning from Gaze Data during Interaction with an Environment to Support Self-Regulated Learning. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 229–238. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  7. 7.
    Conati, C., Maclaren, H.: Empirically building and evaluating a probabilistic model of user affect. UMUAI 19(3), 267–303 (2009)Google Scholar
  8. 8.
    D’Mello, S., Graesser, A.: Mind and body: Dialogue and posture for affect detection in learning environments. FAIA 158, 161 (2007)Google Scholar
  9. 9.
    D’Mello, S., Graesser, A., Picard, R.W.: Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems 22(4), 53–61 (2007)CrossRefGoogle Scholar
  10. 10.
    D’Mello, S., Olney, A., Williams, C., Hays, P.: Gaze tutor: A gaze-reactive intelligent tutoring system. IJHCS 70(5), 377–398 (2012)Google Scholar
  11. 11.
    D’mello, S., Craig, S., Gholson, B., Franklin, S., Picard, R., Graesser, A.: Integrating affect sensors in an intelligent tutoring system. Affective Interactions, 7–13 (2005)Google Scholar
  12. 12.
    Field, A.: Discovering statistics using SPSS. Sage publications (2009)Google Scholar
  13. 13.
    Forbes-Riley, K., Litman, D., Friedberg, H., Drummond, J.: Intrinsic and extrinsic evaluation of an automatic user disengagement detector for an uncertainty-adaptive spoken dialogue system. In: NAACL: Human Language Technologies, pp. 91–102 (2012)Google Scholar
  14. 14.
    Gutica, Conati: Student Emotions with an Edu-Game: A Detailed Analysis (2013)Google Scholar
  15. 15.
    Harley, J.M., Bouchet, F., Azevedo, R.: Aligning and Comparing Data on Emotions Experienced during Learning with MetaTutor. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 61–70. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  16. 16.
    Hussain, M.S., Calvo, R.A.: Multimodal affect detection from physiological and facial features during ITS interaction. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 472–474. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Kardan, S., Conati, C.: Exploring gaze data for determining user learning with an interactive simulation. In: Masthoff, J., Mobasher, B., Desmarais, M.C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 126–138. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  18. 18.
    Klein, J., Moon, Y., Picard, R.W.: This computer responds to user frustration: Theory, design, and results. Interacting with Computers 14(2), 119–140 (2002)CrossRefGoogle Scholar
  19. 19.
    Kort, B., Reilly, R., Mostow, J., Picard, R.: Experimentally augmenting an intelligent tutoring system with human-supplied capabilities: Adding human-provided emotional scaf-folding to an automated reading tutor that listens. In: ICMI, p. 483 (2002)Google Scholar
  20. 20.
    Muldner, K., Christopherson, R., Atkinson, R., Burleson, W.: Investigating the utility of eye-tracking information on affect and reasoning for user modeling. In: Houben, G.-J., McCalla, G., Pianesi, F., Zancanaro, M. (eds.) UMAP 2009. LNCS, vol. 5535, pp. 138–149. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  21. 21.
    Pekrun, R., Goetz, T., Titz, W., Perry, R.P.: Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist 37(2), 91–105 (2002)CrossRefGoogle Scholar
  22. 22.
    Pekrun, R.: Emotions as drivers of learning and cognitive development. In: New Perspectives on Affect and Learning Technologies, pp. 23–39. Springer (2011)Google Scholar
  23. 23.
    Prendinger, H., Ishizuka, M.: The empathic companion: A character-based interface that addresses users’affective states. APAI 19(3-4), 267–285 (2005)Google Scholar
  24. 24.
    Sabourin, J., Mott, B., Lester, J.C.: Modeling learner affect with theoretically grounded dynamic bayesian networks. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part I. LNCS, vol. 6974, pp. 286–295. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  25. 25.
    Sabourin, J., Shores, L., Mott, B., Lester, J.: Predicting student self-regulation strategies in game-based learning environments, 141–150 (2012)Google Scholar
  26. 26.
    Sibert, J.L., Gokturk, M., Lavine, R.A.: The reading assistant: Eye gaze triggered auditory prompting for reading remediation, pp. 101–107. ACM Press (2000)Google Scholar
  27. 27.
    Silvia, P.J.: Interest—The curious emotion. Current Directions in Psychological Science 17(1), 57–60 (2008)CrossRefGoogle Scholar
  28. 28.
    Smilek, D., Carriere, J.S., Cheyne, J.A.: Out of Mind, Out of Sight Eye Blinking as Indicator and Embodiment of Mind Wandering. Psych. Sci. 21(6), 786–789 (2010)CrossRefGoogle Scholar
  29. 29.
    Wang, H., Chignell, M., Ishizuka, M.: Empathic tutoring software agents using real-time eye tracking. ETRA, 73–78 (2006)Google Scholar
  30. 30.
    Wang, N., Johnson, W.L., Mayer, R.E., Rizzo, P., Shaw, E., Collins, H.: The politeness effect: Pedagogical agents and learning outcomes. IJHCS 66(2), 98–112 (2008)Google Scholar
  31. 31.
    Wang, W., Li, Z., Wang, Y., Chen, F.: Indexing cognitive workload based on pupillary response under luminance and emotional changes. IUI, 247–256 (2013)Google Scholar
  32. 32.
    Zimmerman, B.J.: Self-efficacy: An essential motive to learn. Contemporary Educational Psychology 25(1), 82–91 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Natasha Jaques
    • 1
  • Cristina Conati
    • 1
  • Jason M. Harley
    • 2
  • Roger Azevedo
    • 3
  1. 1.University of British ColumbiaVancouverCanada
  2. 2.McGill UniversityMontrealCanada
  3. 3.North Carolina State UniversityRaleighUSA

Personalised recommendations