Abstract
Emotions are crucial for healthy cognitive functioning and have direct relevance to learning and achievement. Not surprisingly then, affective diagnoses constitute a significant aspect of expert human mentoring. Consequently, computer-based learning environments seek to incorporate the social dynamics of such human teacher–learner interactions in order to make learning with computers more engaging and effective. Advances in the field of affective computing have opened the possibility of studying emotions from their nonverbal manifestations and have motivated several efforts towards realising automatic affect inference. However, development and validation of affect recognition systems requires representative data to serve as the ground-truth. For viable applications of affect-sensitive technology, and to ensure ecological validity, the use of context-relevant, naturalistic data is preferred. This chapter reports results from the collection and subsequent annotation of data obtained in a learning scenario. The conceptual and methodological issues encountered during data collection are discussed, and problems with labelling and annotation are identified. It provides an integrated account of the complexity and challenges associated with emotion assessment in naturalistic situations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abrilian, S., Devillers, L., Buisine, S., & Martin, J.-C. (2005). EmoTV1: Annotation of real-life emotions for the specification of multimodal affective interfaces. Proceedings of 11th International Conference on Human-Computer Interaction (HCI 2005), Las Vegas, USA.
Abrilian, S., Devillers, L., & Martin, J.-C. (2006). Annotation of emotions in real-life video interviews: Variability between coders. International Conference of Language Resources & Evaluation. Genoa, Italy.
Afzal, S., Morrison, C., & Robinson, P. (2009). Intentional affect: An alternative notion of affective interaction with a machine. Proceedings of British HCI. Cambridge, UK.
Afzal, S., & Robinson, P. (2010). Measuring affect in learning – Motivation and methods. 10th IEEE International Conference on Advanced Learning Technologies (ICALT). Tunisia.
Arroyo, I., Cooper, D. G., Burleson, W., Woolf, B. P., Muldner, K., & Christopherson, R. (2009). Emotion Sensors Go to School. In V. Dimitrova, R. Mizoguchi, B. du Boulay & A. Grasser (Eds.), Artificial Intelligence in Education. Building Learning Systems that Care: from Knowledge Representation to Affective Modelling (Vol. Frontiers in Artificial Intelligence and Applications 200, pp. 17–24). Brighton, UK: IOS Press.
Bakeman, R., & Gothman, J. M. (1997). Observing interaction: An introduction to sequential analysis. Cambridge, UK: Cambridge University Press.
Baker, R., Rodrigo, M., & Xolocotzin, U. (2007). The dynamics of affective transitions in simulation problem-solving environments. In: Paiva, A., Prada, R., Picard, R.W. (Eds.), ACII 2007. LNCS, vol. 4738, pp. 666–677. Springer, Heidelberg.
Baron-Cohen, S., Golan, O., Wheelwright, S., & Hill, J. (2004). Mind reading: The interactive guide to emotions. London: Jessica Kingsley.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience and school. Washington, DC: National Academy Press.
Camtasia Studio. (2006). Version 3.1. TechSmith Software.
Cohn, J. F., & Schmidt, K. L. (2004). The timing of facial motion in posed and spontaneous smiles. International Journal of Wavelets, Multiresolution and Information Processing, 2, 1–12.
Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267–303.
Cowie, R., Douglas-Cowie, E., & Cox, C. (2005). Beyond emotion archetypes: Databases for emotion modelling using neural networks. Neural Networks, 18, 371–388.
D’Mello, S., Picard, R. W., & Graesser, A. (2007). Towards an affect-sensitive auto-tutor. IEEE Intelligent Systems, 22(4), 53.
D’Mello, S., Taylor, R., Davidson, K., & Graesser, A. (2008). Self versus teacher judgements of learner emotions during a tutoring session with AutoTutor. In B. Woolf et al. (Eds.), ITS 2008, LNCS 5091, (pp 9–18). Springer-Verlag.
Ekman, P., & Rosenberg, E. L. (1997). What the face reveals: Basic and applied studies of spontaneous expression using facial action coding system (FACS). New York: Oxford University Press.
El Kaliouby, R., & Teeters, A. (2007). Eliciting, capturing and tagging spontaneous facial affect in autism spectrum disorder. International Conference on Multimodal Interfaces. Aichi, Japan.
Elfenbein, H. A., Marsh, A. A., & Ambady, N. (2002). Emotional intelligence and the recognition of emotion from facial expressions. In L. F. Barrett & P. Salovey (Eds.), The wisdom of feelings: Processes underlying emotional intelligence (pp. 37–59). New York: Guilford.
Fleiss, J. L., Levin, B., & Paik, M. C. (2003). The measurement of interrater agreement. In J. L. Fleiss, B. Levin, & M. C. Paik (Eds.), Statistical methods for rates & proportions (3rd ed., pp. 598–626). Hoboken, NJ: Wiley.
Frank, M. G., Juslin, P. N., & Harrigan, J. A. (2005). Technical issues in recording nonverbal behaviour. In J. A. Harrigan (Ed.), The new handbook of methods in nonverbal behaviour research. New York: Oxford University Press.
Graesser, A. C., McDaniel, B., Chipman, P., Witherspoon, A., D’Mello, S., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. In R. Son (Ed.), Proceedings of the 28th Annual Meetings of the Cognitive Science Society (pp. 285–290). Mahwah, NJ: Erlbaum.
Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.
Isomursu, M., Tahti, M., Vainamo, S., & Kuutti, K. (2007). Experimental evaluation of five methods for collecting emotions in field settings with mobile application. International Journal of Human-Computer Studies, 65, 404–418.
Kort B., Reilly, R. & Picard R. (2001) An affective model of interplay between emotions and learning: reengineering educational pedagogy—building a learning companion, In T. Okamoto, R. Hartley, Kinshuk & J. P. Klus (Eds.), IEEE International Conference on Advanced Learning Technology: Issues, Achievements and Challenges (Madison, WI, IEEE Computer Society), 43–48.
Lepper, M. R., Woolverton, M., Mumme, D. L., & Gurtner, J. (1993). Motivational techniques of expert human tutors: Lessons for the design of computer-based tutors. In S. P. Lajoie & S. J. Derry (Eds.), Computers as cognitive tools (pp. 75–105). Hillsdale, IN: Erlbaum.
Lisetti, C., & Schiano, D. (2000). Facial expression recognition: Where human computer interaction, artificial intelligence and cognitive science intersect. Pragmatics and Cognition, 8(1), 185–235.
Manusov, V. L. (2005). Sourcebook of nonverbal measures: Going beyond words. Mahwah, NJ: Lawrence Erlbaum.
Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. Journal of the Learning Sciences, 2, 277–305.
O’Regan, K. (2003). Emotion and e-learning. Journal of Asynchronous Learning Networks, 7(3), 78–92.
Pantic, M., & Patras, I. (2006). Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man, and Cybernetics, 36(2), 433–449.
Pantic, M., & Rothkrantz, L. J. M. (2003). Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9), 1370–1390.
Pekrun, R. (2005). Progress and open problems in educational emotion research. Learning and Instruction, 15, 497–506.
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37, 91–105.
Picard, R. W. (1997). Affective computing. Cambridge, MA: MIT.
Picard, R. W., Papert, S., Bender, W., Blumberg, B., Breazeal, C., Cavallo, D., et al. (2004). Affective learning – A manifesto. BT Technology Journal, 22, 253–269.
Riggio, R. E., & Riggio, H. R. (2005). Self-report measures of emotional and nonverbal expressiveness. In V. Manusov (Ed.), The sourcebook of nonverbal measures: Going beyond words (pp. 105–111). Mahwah, NJ: Erlbaum.
Russell, J. A., & Fernandez-Dols, J. M. (1997). The psychology of facial expression. Cambridge, UK: Cambridge University Press.
Scherer, K. (2005). What are emotions? And how can they be measured? Social Science Information, 44(4), 695–729.
Schutz, P. A., Hong, J. Y., Cross, D. I., & Obson, J. N. (2006). Reflections on investigating emotion in educational activity settings. Educational Psychology Review, 18, 343–360.
Skemp, R. R. (1971). The psychology of learning mathematics. Hillsdale, NJ: Erlbaum.
White, C. H., & Sargent, J. (2005). Researcher choices and practices in the study of nonverbal communication. In V. L. Manusov (Ed.), Sourcebook of nonverbal measures: Going beyond words. Mahwah, NJ: Lawrence Erlbaum.
Wosnitza, M., & Volet, S. (2005). Origin, direction and impact of emotions in social online learning. Learning and Instruction, 15(5), 440–464.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Afzal, S., Robinson, P. (2011). Natural Affect Data: Collection and Annotation. In: Calvo, R., D'Mello, S. (eds) New Perspectives on Affect and Learning Technologies. Explorations in the Learning Sciences, Instructional Systems and Performance Technologies, vol 3. Springer, New York, NY. https://doi.org/10.1007/978-1-4419-9625-1_5
Download citation
DOI: https://doi.org/10.1007/978-1-4419-9625-1_5
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4419-9624-4
Online ISBN: 978-1-4419-9625-1
eBook Packages: Humanities, Social Sciences and LawEducation (R0)