Abstract
In this paper, we describe a way of using multi-modal learning analytics to augment qualitative data. We extract facial expressions that may indicate particular emotions from videos of dyads playing an interactive table-top game built for a museum. From this data, we explore the correlation between students’ understanding of the biological and complex systems concepts showcased in the learning environment and their facial expressions. First, we show how information retrieval techniques can be used on facial expression features to investigate emotional variation during key moments of the interaction. Second, we connect these features to moments of learning identified by traditional qualitative methods. Finally, we present an initial pilot using these methods in concert to identify key moments in multiple modalities. We end with a discussion of our preliminary findings on interweaving machine and human analytical approaches.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Blikstein, P.: An atom is known by the company it keeps: a constructionist learning environment for materials science using multi-agent simulation (2006)
Chi, M.T.H., Roscoe, R.D., Slotta, J.D., Roy, M., Chase, C.C.: Misconceived causal explanations for emergent processes. Cogn. Sci. 36(1), 1–61 (2012). https://doi.org/10.1111/j.1551-6709.2011.01207.x. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1551-6709.2011.01207.x
Council, N.R.: Learning Science in Informal Environments: People, Places, and Pursuits. National Academies Press (2009)
D’Mello, S., Dieterle, E., Duckworth, A.: Advanced, analytic, automated (AAA) measurement of engagement during learning. Educ. Psychol. 52(2), 104–123 (2017). https://doi.org/10.1080/00461520.2017.1281747
D’Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learn. Instr. 22(2), 145–157 (2012). https://doi.org/10.1016/j.learninstruc.2011.10.001. http://www.sciencedirect.com/science/article/pii/S0959475211000806
Falk, J.H., Dierking, L.D.: The Museum Experience Revisited. Routledge (2016)
Gregory, J.: Game Engine Architecture. AK Peters/CRC Press (2017)
Martin, K., Wilensky, U.: Netlogo ant adapatation model (2019). http://ccl.northwestern.edu/netlogo/models/AntAdaptation
Martin, K.: Constructivist dialogue mapping: evaluating learning during play of ant adaptation, a complex interactive tabletop museum game. In: 31st Annual Visitor Studies Association, p. 20 (2018)
Martin, K.: Multitouch NetLogo for museum interactive game. In: Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW 2018, Jersey City, NJ, USA, pp. 5–8. ACM, New York (2018). https://doi.org/10.1145/3272973.3272989. http://doi.acm.org/10.1145/3272973.3272989
Martin, K., Horn, M., Wilensky, U.: Ant Adaptation: A Complex Interactive Multitouch Game about Ants Designed for Museums, August 2018
Martin, K., Horn, M., Wilensky, U.: Prevalence of direct and emergent schema and change after play. Inf. Educ. 18(1), 183 (2019)
McGaugh, J.L.: Memory and Emotion: The Making of Lasting Memories. Columbia University Press (2003)
McGaugh, J.L.: Make mild moments memorable: add a little arousal. Trends Cogn. Sci. 10(8), 345–347 (2006). https://doi.org/10.1016/j.tics.2006.06.001. http://www.sciencedirect.com/science/article/pii/S1364661306001355
Miles, M.B., Huberman, A.M., Saldaa, J.: Qualitative Data Analysis: A Methods Sourcebook, 3rd edn. Sage, Thousand Oaks (2014)
Ochoa, X., Worsley, M.: Editorial: augmenting learning analytics with multimodal sensory data. J. Learn. Anal. 3(2), 213–219 (2016). https://doi.org/10.18608/jla.2016.32.10. https://learning-analytics.info/journals/index.php/JLA/article/view/5081
Oviatt, S., Cohen, A., Weibel, N.: Multimodal learning analytics: description of math data corpus for ICMI grand challenge workshop. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI 2013, Sydney, Australia, pp. 563–568. ACM, New York (2013). https://doi.org/10.1145/2522848.2533790. http://doi.acm.org/10.1145/2522848.2533790
Schneider, B., Blikstein, P.: Unraveling students’ interaction around a tangible interface using multimodal learning analytics. J. Educ. Data Mining 7, 89–116 (2015)
Spikol, D., et al.: Exploring the interplay between human and machine annotated multimodal learning analytics in hands-on stem activities. In: Proceedings of the 6th International Learning Analytics & Knowledge Conference, vol. 6, pp. 522–523. Association for Computing Machinery (2016)
Taggart, R.W., Dressler, M., Kumar, P., Khan, S., Coppola, J.F.: Determining emotions via facial expression analysis software. In: Proceedings of Student-Faculty Research Day, CSIS, Pace University, May 2016
Wagner, J., Lingenfelser, F., Baur, T., Damian, I., Kistler, F., Andr, E.: The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM International Conference on Multimedia, MM 2013, Barcelona, Spain, pp. 831–834. ACM, New York (2013). https://doi.org/10.1145/2502081.2502223. http://doi.acm.org/10.1145/2502081.2502223
Wilensky, U.: Netlogo gaslab gas in a box model (1997). http://ccl.northwestern.edu/netlogo/models/GasLabGasinaBox
Wilensky, U.: NetLogo (and NetLogo user manual) (1999). https://ccl.northwestern.edu/netlogo/
Wilensky, U., Reisman, K.: Thinking like a wolf, a sheep, or a firefly: learning biology through constructing and testing computational theories–an embodied modeling approach. Cogn. Instr. 24(2), 171–209 (2006)
Worsley, M., Blikstein, P.: A multimodal analysis of making. Int. J. Artif. Intell. Educ. 28(3), 385–419 (2018). https://doi.org/10.1007/s40593-017-0160-1
Worsley, M., Scherer, S., Morency, L.P., Blikstein, P.: Exploring behavior representation for learning analytics. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, Seattle, Washington, USA, pp. 251–258. ACM, New York (2015). https://doi.org/10.1145/2818346.2820737. http://doi.acm.org/10.1145/2818346.2820737
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Martin, K., Wang, E.Q., Bain, C., Worsley, M. (2019). Computationally Augmented Ethnography: Emotion Tracking and Learning in Museum Games. In: Eagan, B., Misfeldt, M., Siebert-Evenstone, A. (eds) Advances in Quantitative Ethnography. ICQE 2019. Communications in Computer and Information Science, vol 1112. Springer, Cham. https://doi.org/10.1007/978-3-030-33232-7_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-33232-7_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33231-0
Online ISBN: 978-3-030-33232-7
eBook Packages: Computer ScienceComputer Science (R0)