Detecting Collaborative Learning Through Emotions: An Investigation Using Facial Expression Recognition

  • Yugo HayashiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11528)


Providing adaptive feedback to learners engaging in collaborative learning activities is one research topic in the development of intelligent tutoring systems. However, there is a need to investigate how to systematically evaluate a learner’s activities and provide feedback on them. The present study investigates how emotional states, detected through facial recognition, can be utilized to capture the learning process in a simple jigsaw-type collaborative task. It was predicted that when learners argue with each other and reason deeply, they may experience several emotional states such as positive and negative states. The results show that when learners work harder on developing a mutual understanding through conflictive interaction, negative emotions can be used to predict this process. This study contributes to the knowledge of how emotional states detected by facial recognition technology can be applied to predict learning process in conflictive tasks. Moreover, these empirical results will impact the development of adaptive feedback mechanisms for intelligent tutoring systems for collaborative learning.


Collaborative learning Pedagogical conversational agents Emotion Learning assessment 



This work was supported by Grant-in-Aid for Scientific Research (C), Japan Society for the Promotion of Science (JSPS) KAKENHI Grant Number 16K00219.


  1. 1.
    Aronson, E., Patnoe, S.: The Jigsaw Classroom: Building Cooperation in the Classroom, 2nd edn. Addison Wesley Longman, New York (1997)Google Scholar
  2. 2.
    Azevedo, R., Cromley, J.: Does training on selfregulated learning facilitate students’ learning with hypermedia? J. Educ. Psychol. 96(3), 523–535 (2004)CrossRefGoogle Scholar
  3. 3.
    Biswas, G., Leelawong, K., Schwartz, D., Vye, N.: Learning by teaching: a new paradigm for educational software. Appl. Artif. Intell. 19(3), 363–392 (2005)CrossRefGoogle Scholar
  4. 4.
    Chi, M., Leeuw, N., Chiu, M., Lavancher, C.: Eliciting self-explanations improves understanding. Cogn. Sci. 18(3), 439–477 (1994)Google Scholar
  5. 5.
    von Davier, A.A., Hao, J., Liu, L., Kyllonen, P.: Interdisciplinary research agenda in support of assessment of collaborative problem solving: lessons learned from developing a collaborative science assessment prototype. Comput. Hum. Behav. 76, 631–640 (2017)CrossRefGoogle Scholar
  6. 6.
    D’Mello, S., Lehman, B., Pekrun, R., Graesser, A.: Confusion can be beneficial for learning. Learn. Instr. 29, 153–170 (2014)CrossRefGoogle Scholar
  7. 7.
    Fiore, S.M., Rosen, M.A., Smith-Jentsch, K.A., Salas, E., Letsky, M., Warner, N.: Toward an understanding of macrocognition in teams: predicting processes in complex collaborative contexts. Hum. Factors 52, 203–224 (2010)CrossRefGoogle Scholar
  8. 8.
    Graesser, A., Chipman, P., Haynes, B., Olney, A.: AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans. Educ. 48(4), 612–618 (2005)CrossRefGoogle Scholar
  9. 9.
    Graesser, A., McNamara, D.S.: Computational analyses of multilevel discourse comprehension. Top. Cogn. Sci. 3(2), 371–398 (2011). Scholar
  10. 10.
    Hayashi, Y.: On pedagogical effects of learner-support agents in collaborative interaction. In: Proceeding of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), pp. 22–32 (2012)CrossRefGoogle Scholar
  11. 11.
    Hayashi, Y.: Togetherness: Multiple pedagogical conversational agents as companions in collaborative learning. In: Proceeding of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), pp. 114–123 (2014)CrossRefGoogle Scholar
  12. 12.
    Hayashi, Y.: Coordinating knowledge integration with pedagogical agents: effects of agent gaze gestures and dyad synchronization. In: Proceeding of the 13th International Conference on Intelligent Tutoring Systems (ITS 2016), pp. 254–259 (2016)Google Scholar
  13. 13.
    Hayashi, Y.: Gaze feedback and pedagogical suggestions in collaborative learning. In: Nkambou, R., Azevedo, R., Vassileva, J. (eds.) ITS 2018. LNCS, vol. 10858, pp. 78–87. Springer, Cham (2018). Scholar
  14. 14.
    Hayashi, Y.: The power of a “maverick” in collaborative problem solving: an experimental investigation of individual perspective-taking within a group. Cogn. Sci. 42(S1), 69–104 (2018). Scholar
  15. 15.
    Hesse, F., Care, E., Buder, J., Sassenberg, K., Griffin, P.: A framework for teachable collaborative problem solving skills. In: Griffin, P., Care, E. (eds.) Assessment and Teaching of 21st Century Skills. EAIA, pp. 37–56. Springer, Dordrecht (2015). Scholar
  16. 16.
    Koedinger, K.R., Anderson, J.R., Hadley, W.H., Mark, M.A.: Intelligent tutoring goes to school in the big city. Int. J. Artif. Intell. Educ. (IJAIED) 8, 30–43 (1997)Google Scholar
  17. 17.
    Kuilenburg, v.H., Wiering, M., Uyl, d.M.: A model based method for automatic facial expression recognition. In: Proceedings of the European Conference on Machine Learning (ECML2005), pp. 194–205 (2005)Google Scholar
  18. 18.
    Kumar, R., Rose, C.: Architecture for building conversational architecture for building conversational agents that support collaborative learning. IEEE Trans. Learn. Technol. 4(1), 21–34 (2011)CrossRefGoogle Scholar
  19. 19.
    Lave, J., Wenger, E.: Situated Learning - Legitimate Peripheral Participation. Cambridge University Press, Cambridge (1991)CrossRefGoogle Scholar
  20. 20.
    Leelawong, K., Biswas, G.: Designing learning by teaching agents: the Betty’s brain system. Int. J. Artif. Intell. Educ. 18(3), 181–208 (2008)Google Scholar
  21. 21.
    Matsuda, N., Yarzebinski, E., Keiser, V., Raizada, R., Stylianides, G.J., Koedinger, K.R.: Studying the effect of a competitive game show in a learning by teaching environment. Int. J. Artif. Intell. Educ. 23(1), 1–21 (2013)CrossRefGoogle Scholar
  22. 22.
    McNamara, D., O’Reilly, T., Rowe, M.: iSTART: A Web-Based Tutor that Teaches Self-explanation and Metacognitive Reading Strategies. Lawrence Erlbaum Associates, Mahwah (2007)Google Scholar
  23. 23.
    Meier, A., Spada, H., Rummel, N.: A rating scheme for assessing the quality of computer-supported collaboration processes. Int. J. Comput.-Support. Collab. Learn. 2(1), 63–86 (2007). Scholar
  24. 24.
    Metcalfe, J., Wiebe, D.: Intuition in insight and noninsight problem solving. Mem. Cogn. 15(3), 238–246 (1987)CrossRefGoogle Scholar
  25. 25.
    Miyake, N.: Constructive interaction and the interactive process of understanding. Cogn. Sci. 10(2), 151–177 (1986)CrossRefGoogle Scholar
  26. 26.
    OECD: PISA 2015 Results (Volume V): Collaborative Problem Solving. OECD Publishing, Paris (2017).
  27. 27.
    Shirouzu, H., Miyake, N., Masukawa, H.: Cognitively active externalization for situated reflection. Cogn. Sci. 26(4), 469–501 (2002)CrossRefGoogle Scholar
  28. 28.
    VanLehn, K., Graesser, A.C., Jackson, G.T., Jordan, P., Olney, A., Rosé, C.P.: When are tutorial dialogues more effective than reading? Cogn. Sci. 31(1), 3–62 (2007). Scholar
  29. 29.
    Vygotsky, L.S.: The Development of Higher Psychological Processes. Harvard University Press, Cambridge (1980)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.College of Comprehensive PsychologyRitsumeikan UniversityOsakaJapan

Personalised recommendations