Multimodal Analytics for Real-Time Feedback in Co-located Collaboration

  • Sambit PraharajEmail author
  • Maren Scheffel
  • Hendrik Drachsler
  • Marcus Specht
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11082)


Collaboration is an important 21st century skill; it can take place in a remote or co-located setting. Co-located collaboration (CC) is a very complex process which involves subtle human interactions that can be described with multimodal indicators (MI) like gaze, speech and social skills. In this paper, we first give an overview of related work that has identified indicators during CC. Then, we look into the state-of-the-art studies on feedback during CC which also make use of MI. Finally, we describe a Wizard of Oz (WOz) study where we design a privacy-preserving research prototype with the aim to facilitate real-time collaboration in-the-wild during three co-located group PhD meetings (of 3–7 members). Here, human observers stationed in another room act as a substitute for sensors to track different speech-based cues (like speaking time and turn taking); this drives a real-time visualization dashboard on a public shared display. With this research prototype, we want to pave way for design-based research to track other multimodal indicators of CC by extending this prototype design using both humans and sensors.


Collaboration Feedback CSCL Intervention Multimodal indicators Multimodal learning analytics 


  1. 1.
    Anastasiou, D., Ras, E.: A questionnaire-based case study on feedback by a tangible interface. In: Proceedings of the 2017 ACM WS on Intelligent Interfaces for Ubiquitous and Smart Learning, pp. 39–42. ACM (2017)Google Scholar
  2. 2.
    Bachour, K., Kaplan, F., Dillenbourg, P.: An interactive table for supporting participation balance in face-to-face collaborative learning. IEEE Trans. Learn. Technol. 3(3), 203–213 (2010)CrossRefGoogle Scholar
  3. 3.
    Balaam, M., Fitzpatrick, G., Good, J., Harris, E.: Enhancing interactional synchrony with an ambient display. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 867–876. ACM (2011)Google Scholar
  4. 4.
    Bassiou, N., Tsiartas, A., Smith, J., Bratt, H., Richey, C., Shriberg, E., D’Angelo, C., Alozie, N.: Privacy-preserving speech analytics for automatic assessment of student collaboration. In: INTERSPEECH, pp. 888–892 (2016)Google Scholar
  5. 5.
    Bergstrom, T., Karahalios, K.: Conversation clock: visualizing audio patterns in co-located groups. In: 40th Annual Hawaii International Conference on System Sciences, p. 78. IEEE (2007)Google Scholar
  6. 6.
    Chikersal, P., Tomprou, M., Kim, Y.J., Woolley, A.W., Dabbish, L.: Deep structures of collaboration: physiological correlates of collective intelligence and group satisfaction. In: CSCW, pp. 873–888 (2017)Google Scholar
  7. 7.
    Cukurova, M., Luckin, R., Mavrikis, M., Millán, E.: Machine and human observable differences in groups’ collaborative problem-solving behaviours. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., Pérez-Sanagustín, M. (eds.) EC-TEL 2017. LNCS, vol. 10474, pp. 17–29. Springer, Cham (2017). Scholar
  8. 8.
    Cukurova, M., Luckin, R., Millán, E., Mavrikis, M.: The nispi framework: analysing collaborative problem-solving from students’ physical interactions. Comput. Educ. 116, 93–109 (2018)CrossRefGoogle Scholar
  9. 9.
    Davidsen, J., Ryberg, T.: This is the size of one meter: childrens bodily-material collaboration. Int. J. CSCL 12(1), 65–90 (2017)Google Scholar
  10. 10.
    Dede, C.: Comparing frameworks for 21st century skills. In: 21st Century Skills: Rethinking How Students Learn, vol. 20, pp. 51–76 (2010)Google Scholar
  11. 11.
    Di Mitri, D., Schneider, J., Drachsler, H., Specht, M.: From signals to knowledge. A conceptual model for multimodal learning analytics. J. Comput. Assist. Learn. 34(4), 338–349 (2018)CrossRefGoogle Scholar
  12. 12.
    Dillenbourg, P.: What do you mean by collaborative learning? (1999)Google Scholar
  13. 13.
    DiMicco, J.M., Pandolfo, A., Bender, W.: Influencing group participation with a shared display. In: Proceedings of the 2004 ACM Conference on CSCW, pp. 614–623. ACM (2004)Google Scholar
  14. 14.
    Dyckhoff, A.L.: Action research and learning analytics in higher education. eleed 10(1) (2014)Google Scholar
  15. 15.
    Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., Divakaran, A.: Multimodal analytics to study collaborative problem solving in pair programming. In: Proceedings of the 6th International Conference on LAK, pp. 516–517. ACM (2016)Google Scholar
  16. 16.
    Jermann, P., Mullins, D., Nüssli, M.A., Dillenbourg, P.: Collaborative gaze footprints: correlates of interaction quality. In: CSCL 2011 Conference Proceedings, vol. 1, pp. 184–191. International Society of the Learning Sciences (2011)Google Scholar
  17. 17.
    Kim, T., Chang, A., Holland, L., Pentland, A.S.: Meeting mediator: enhancing group collaboration using sociometric feedback. In: Proceedings of the 2008 ACM Conference on CSCW, pp. 457–466. ACM (2008)Google Scholar
  18. 18.
    Kulyk, O., Wang, J., Terken, J.: Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In: Renals, S., Bengio, S. (eds.) MLMI 2005. LNCS, vol. 3869, pp. 150–161. Springer, Heidelberg (2006). Scholar
  19. 19.
    Lubold, N., Pon-Barry, H.: Acoustic-prosodic entrainment and rapport in collaborative learning dialogues. In: Proceedings of the 2014 ACM WS on Multimodal Learning Analytics Workshop and Grand Challenge, pp. 5–12. ACM (2014)Google Scholar
  20. 20.
    Madan, A., Caneel, R., Pentland, A.S.: Groupmedia: distributed multi-modal interfaces. In: Proceedings of the 6th International Conference on Multimodal Interfaces, pp. 309–316. ACM (2004)Google Scholar
  21. 21.
    Martinez-Maldonado, R., Clayphan, A., Yacef, K., Kay, J.: Mtfeedback: providing notifications to enhance teacher awareness of small group work in the classroom. IEEE Trans. Learn. Technol. 8(2), 187–200 (2015)CrossRefGoogle Scholar
  22. 22.
    Meier, A., Spada, H., Rummel, N.: A rating scheme for assessing the quality of computer-supported collaboration processes. Int. J. CSCL 2(1), 63–86 (2007)Google Scholar
  23. 23.
    O’Donnell, A.M.: The role of peers and group learning (2006)Google Scholar
  24. 24.
    Pijeira-Daz, H.J., Drachsler, H., Kirschner, P.A., Järvelä, S.: Profiling sympathetic arousal in a physics course: how active are students? J. Comput. Assist. Learn. 34(4), 397–408 (2018)CrossRefGoogle Scholar
  25. 25.
    Richardson, D.C., Dale, R.: Looking to understand: the coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cogn. Sci. 29(6), 1045–1060 (2005)CrossRefGoogle Scholar
  26. 26.
    Scherr, R.E., Hammer, D.: Student behavior and epistemological framing: examples from collaborative active-learning activities in physics. Cogn. Instr. 27(2), 147–174 (2009)CrossRefGoogle Scholar
  27. 27.
    Schneider, B., Blikstein, P.: Unraveling students interaction around a tangible interface using multimodal learning analytics. J. Educ. Data Min. 7(3), 89–116 (2015)Google Scholar
  28. 28.
    Schneider, B., Pea, R.: Real-time mutual gaze perception enhances collaborative learning and collaboration quality. Int. J. CSCL 8(4), 375–397 (2013)Google Scholar
  29. 29.
    Schneider, B., Pea, R.: Toward collaboration sensing. Int. J. CSCL 9(4), 371–395 (2014)Google Scholar
  30. 30.
    Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., Pea, R.D.: 3d tangibles facilitate joint visual attention in dyads. International Society of the Learning Sciences, Inc. [ISLS] (2015)Google Scholar
  31. 31.
    Schneider, J., Di Mitri, D., Limbu, B., Drachsler, H.: Multimodal learning hub: a tool for capturing customizable multimodal learning experiences. In: Drachsler, H., et al. (eds.) EC-TEL 2018. LNCS, vol. 11082, pp. 45–58. Springer, AG (2018)CrossRefGoogle Scholar
  32. 32.
    Shih, P.C., Nguyen, D.H., Hirano, S.H., Redmiles, D.F., Hayes, G.R.: Groupmind: supporting idea generation through a collaborative mind-mapping tool. In: Proceedings of the ACM 2009 International Conference on Supporting Group Work, pp. 139–148. ACM (2009)Google Scholar
  33. 33.
    Spikol, D., Ruffaldi, E., Dabisias, G., Cukurova, M.: Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. J. Comput. Assist. Learn. 34(4), 366–377 (2018)CrossRefGoogle Scholar
  34. 34.
    Spikol, D., Ruffaldi, E., Landolfi, L., Cukurova, M.: Estimation of success in collaborative learning based on multimodal learning analytics features. In: Proceedings of the 17th ICALT, pp. 269–273. IEEE (2017)Google Scholar
  35. 35.
    Stahl, G., Law, N., Hesse, F.: Reigniting CSCL flash themes. Int. J. CSCL 8(4), 369–374 (2013)Google Scholar
  36. 36.
    Stiefelhagen, R., Zhu, J.: Head orientation and gaze direction in meetings. In: CHI 2002 Extended Abstracts on Human Factors in Computing Systems, pp. 858–859. ACM (2002)Google Scholar
  37. 37.
    Tausch, S., Hausen, D., Kosan, I., Raltchev, A., Hussmann, H.: Groupgarden: supporting brainstorming through a metaphorical group mirror on table or wall. In: Proceedings of the 8th Nordic Conference on Human-Computer Interaction, pp. 541–550. ACM (2014)Google Scholar
  38. 38.
    Terken, J., Sturm, J.: Multimodal support for social dynamics in co-located meetings. Pers. Ubiquitous Comput. 14(8), 703–714 (2010)CrossRefGoogle Scholar
  39. 39.
    Triglianos, V., Praharaj, S., Pautasso, C., Bozzon, A., Hauff, C.: Measuring student behaviour dynamics in a large interactive classroom setting. In: Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 212–220. ACM (2017)Google Scholar
  40. 40.
    Worsley, M., Blikstein, P.: Leveraging multimodal learning analytics to differentiate student learning strategies. In: Proceedings of the 5th International Conference on LAK, pp. 360–367. ACM (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Open UniversiteitHeerlenNetherlands
  2. 2.DIPFFrankfurt am MainGermany
  3. 3.Goethe UniversitätFrankfurt am MainGermany

Personalised recommendations