Abstract
Recent advances in sensor technology allow us to investigate the physical, emotional, and cognitive state of a learner, providing more data for user-centered learning analytics, adaptive learning, and self-reflection. While the usage of physiological sensors for health applications has been thoroughly investigated, using sensors in a learning environment is very challenging, especially when sensors should be nonintrusive, not distracting from learning. In our project “Learning Analytics for Sensor-Based Adaptive Learning,” we address a wide range of challenges and research questions, from the development of a wearable sensor device for learners to the design of a sensor-based learning companion, from multimodal learning analytics to adaptive learning solutions in real-life environments. In this article, we show first steps toward emotion recognition, using a mixed-methods approach comprising qualitative and quantitative analysis, machine learning, and fuzzy logic. Finally, we discuss our interdisciplinary approach needed to resolve some of the complex questions related to sensor-based learning support.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
BITalino (r)evolution Plugged Kit BT 10.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
Förderkennzeichen 16SV7534K.
References
Ayata, D., Yaslan, Y., & Kamaşak, M. (2017). Emotion recognition via galvanic skin response: Comparison of machine learning algorithms and feature extraction methods. Istanbul University-Journal of Electrical & Electronics Engineering, 17(1), 3147–3156.
Bakharia, A., Kitto, K., Pardo, A., Gašević, D., & Dawson, S. (2016, April). Recipe for success: Lessons learnt from using xAPI within the connected learning analytics toolkit. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 378–382). ACM.
Bartram, L., Patra, A., & Stone, M. (2017, May). Affective color in visualization. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 1364–1374). ACM.
Boucsein, W. (2012). Electrodermal activity. Boston: Springer Science & Business Media.
Bradley, M. M., & Lang, P. J. (2007). The International Affective Picture System (IAPS) in the study of emotion and attention. In J. A. Coan & J. J. B. Allen (Eds.), Handbook of emotion elicitation and assessment (pp. 29–46). Oxford, UK: Oxford University Press.
Burleson, W., & Picard, R. W. (2004, August). Affective agents: Sustaining motivation to learn through failure and a state of stuck. In workshop on social and emotional intelligence in learning environments.
Camm, A. J., Malik, M., Bigger, J. T., Breithardt, G., Cerutti, S., Cohen, R. J., et al. (1996). Heart rate variability: Standards of measurement, physiological interpretation and clinical use. In task force of the European society of cardiology and the North American society of pacing and electrophysiology.
Cercone, K. (2008). Characteristics of adult learners with implications for online learning design. AACE Journal, 16(2), 137–159.
Chanel, G., & Mühl, C. (2015). Connecting brains and bodies: Applying physiological computing to support social interaction. Interacting with Computers, 27(5), 534–550.
Conati, C., Chabbal, R., & Maclaren, H. (2003, June). A study on using biometric sensors for monitoring user emotions in educational games. In Workshop on assessing and adapting to user attitudes and affect: Why, when and how.
Cook, M. (1979). Perceiving others: The psychology of interpersonal perception (Vol. 675). New York/London: Taylor & Francis.
D’Mello, S. (2013). A selective meta-analysis on the relative incidence of discrete affective states during learning with technology. Journal of Educational Psychology, 105(4), 1082.
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349.
Drachsler, H., & Greller, W. (2016, April). Privacy and analytics: It’s a DELICATE issue a checklist for trusted learning analytics. In Proceedings of the sixth international conference on learning analytics & knowledge (pp. 89–98). ACM.
Ekman, P., Levenson, R. W., & Friesen, W. V. (1983). Autonomic nervous system activity distinguishes among emotions. Science, 221(4616), 1208–1210.
Fairclough, S. H., Venables, L., & Tattersall, A. (2005). The influence of task demand and learning on the psychophysiological response. International Journal of Psychophysiology, 56(2), 171–184.
Ferdinando, H., Seppänen, T., & Alasaarela, E. (2018, June). Emotion recognition using neighborhood components analysis and ECG/HRV-based. In Pattern recognition applications and methods: 6th international conference, ICPRAM 2017, Porto, Portugal, February 24–26, 2017, Revised Selected Papers (Vol. 10857, p. 99). Springer.
Fortenbacher, A., Ninaus, M., Yun, H., Helbig, R., & Moeller, K. (2019). Sensor based adaptive learning-lessons learned. DELFI 2019.
Fortenbacher, A., Pinkwart, N., & Yun, H. (2017, March). [LISA] learning analytics for sensor-based adaptive learning. In Proceedings of the seventh international learning analytics & knowledge conference (pp. 592–593). ACM.
Freire, C., Ramos, R., Puertas, R., Lopez-Espinosa, M. J., Julvez, J., Aguilera, I., et al. (2010). Association of traffic-related air pollution with cognitive development in children. Journal of Epidemiology & Community Health, 64(3), 223–228.
Gruber, J., Mennin, D. S., Fields, A., Purcell, A., & Murray, G. (2015). Heart rate variability as a potential indicator of positive valence system disturbance: A proof of concept investigation. International Journal of Psychophysiology, 98(2), 240–248.
Gulz, A., & Haake, M. (2006). Virtual pedagogical agents–design guidelines regarding visual appearance and pedagogical roles. Current developments in technology-assisted education,© FORMATEX 2006.
Hwang, B., You, J., Vaessen, T., Myin-Germeys, I., Park, C., & Zhang, B. T. (2018). Deep ECGNet: An optimal deep learning framework for monitoring mental stress using ultra short-term ECG signals. Telemedicine Journal and e-Health, 24(10), 753–772.
Isbister, K. (2006). Better game characters by design: A psychological approach. Boca Raton, FL: CRC Press.
Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: A review. Biological Psychology, 84(3), 394–421.
Lang, P., & Bradley, M. M. (2007). The International Affective Picture System (IAPS) in the study of emotion and attention. Handbook of Emotion Elicitation and Assessment, 29.
Lang, P. J. (1995). The emotion probe: Studies of motivation and attention. American Psychologist, 50(5), 372.
Lanzetta, J. T., Cartwright-Smith, J., & Eleck, R. E. (1976). Effects of nonverbal dissimulation on emotional experience and autonomic arousal. Journal of Personality and Social Psychology, 33(3), 354.
Larsen, J. T., Berntson, G. G., Poehlmann, K. M., Ito, T. A., & Cacioppo, J. T. (2008). The psychophysiology of emotion. Handbook of Emotions, 3, 180–195.
Levenson, R. W., Ekman, P., & Friesen, W. V. (1990). Voluntary facial action generates emotion-specific autonomic nervous system activity. Psychophysiology, 27(4), 363–384.
Libby Jr., W. L., Lacey, B. C., & Lacey, J. I. (1973). Pupillary and cardiac activity during visual attention. Psychophysiology, 10(3), 270–294.
Malkawi, M., & Murad, O. (2013). Artificial neuro fuzzy logic system for detecting human emotions. Human-centric Computing and Information Sciences, 3(1), 3.
Mandryk, R. L., & Atkins, M. S. (2007). A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies. International Journal of Human-Computer Studies, 65(4), 329–347.
McCloud, S., & Manning, A. D. (1998). Understanding comics: The invisible art. IEEE Transactions on Professional Communications, 41(1), 66–69.
Pecchinenda, A. (1996). The affective significance of skin conductance activity during a difficult problem-solving task. Cognition & Emotion, 10(5), 481–504.
Picard, R. W., Vyzas, E., & Healey, J. (2001). Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis & Machine Intelligence, 23(10), 1175–1191.
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013, April). Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions. In 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG) (pp. 1–8). IEEE, New York.
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161.
Scheibe, S., & Fortenbacher, A. (2019). Heart Rate Variability als Indikator für den emotionalen Zustand eines Lernenden. In Proceedings of DELFI Workshops 2019. Gesellschaft für Informatik eVz.
Schneider, J., Börner, D., Van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: A review on sensor-based learning support. Sensors, 15(2), 4097–4133.
Tomaka, J., Blascovich, J., Kelsey, R. M., & Leitten, C. L. (1993). Subjective, physiological, and behavioral effects of threat and challenge appraisal. Journal of Personality and Social Psychology, 65(2), 248.
Vrana, S. R., Cuthbert, B. N., & Lang, P. J. (1986). Fear imagery and text processing. Psychophysiology, 23(3), 247–253.
Winton, W. M., Putnam, L. E., & Krauss, R. M. (1984). Facial and autonomic manifestations of the dimensional structure of emotion. Journal of Experimental Social Psychology, 20(3), 195–216.
Witten, I. H., Frank, E., Hall, M. A., & Pal, C. J. (2016). Data mining: Practical machine learning tools and techniques. San Francisco: Morgan Kaufmann.
Woolf, B. P. (2010). Building intelligent interactive tutors: Student-centered strategies for revolutionizing e-learning. Amsterdam: Morgan Kaufmann.
Yun, H., Fortenbacher, A., Helbig, R., & Pinkwart, N. (2019). In search of learning indicators: A study on sensor data and IAPS emotional pictures.
Yun, H., Fortenbacher, A., & Pinkwart, N. (2017). Improving a mobile learning companion for self-regulated learning using sensors. In CSEDU (1) (pp. 531–536).
Yun, H., Fortenbacher, A., & Scaff, P. (2019). Visualization of learning process and learner’s emotions: Current state, limitations and future work. In Weizenbaum conference (p. 4). DEU Berlin.
Yun, H., Israel, J. H., Fortenbacher, A., Rott, H., & Metzler, D. (2017). User-centric approach to the design of a mobile learning companion.
Yun, H., Riazy, S., Fortenbacher, A., & Simbeck, K. (2019). Code of practice for sensor-based learning. DELFI 2019 Berlin.
Zhang, X., Chen, X., & Zhang, X. (2018). The impact of exposure to air pollution on cognitive performance. Proceedings of the National Academy of Sciences, 115(37), 9193–9197.
Acknowledgments
This work was funded by the German Ministry for Education and Research (BMBF)Footnote 18
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Fortenbacher, A., Yun, H. (2020). Can Sensors Effectively Support Learning?. In: Pinkwart, N., Liu, S. (eds) Artificial Intelligence Supported Educational Technologies. Advances in Analytics for Learning and Teaching. Springer, Cham. https://doi.org/10.1007/978-3-030-41099-5_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-41099-5_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41098-8
Online ISBN: 978-3-030-41099-5
eBook Packages: EducationEducation (R0)