Abstract
Two unobtrusive modalities for automatic emotion recognition are discussed: speech and facial expressions. First, an overview is given of emotion recognition studies based on a combination of speech and facial expressions. We will identify difficulties concerning data collection, data fusion, system evaluation and emotion annotation that one is most likely to encounter in emotion recognition research. Further, we identify some of the possible applications for emotion recognition such as health monitoring or e-learning systems. Finally, we will discuss the growing need for developing agreed standards in automatic emotion recognition research.
Chapter PDF
Similar content being viewed by others
References
Cowie, R., Schröder, M.: Piecing together the emotion jigsaw. In: Machine Learning for Multimodal Interaction, pp. 305–317 (2005)
Kapoor, A., Picard, R.W.: Multimodal affect recognition in learning environments. In: Proceedings of the ACM International Conference on Multimedia, pp. 677–682 (2005)
Kim, J., André, E., Rehm, M., Vogt, T., Wagner, J.: Integrating information from speech and physiological signals to achieve emotional sensitivity. In: Proceedings of Interspeech, pp. 809–812 (2005)
Zhai, J., Barreto, A.: Stress Recognition Using Non-invasive Technology. In: Proceedings of the 19th International Florida Artificial Intelligence Research Society Conference FLAIRS, pp. 395–400 (2006)
Banse, R., Scherer, K.R.: Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology 70, 614–636 (1996)
Ang, J., Dhillon, R., Krupski, A., Shriberg, E., Stolcke, A.: Prosody-based automatic detection of annoyance and frustration in Human-Computer Dialog. In: Proceedings of the ICSLP International Conference on Spoken Language Processing, pp. 2037–2040 (2002)
Nwe, T.L., Foo, S.W, De Silva, L.C.: Speech emotion recognition using hidden Markov models. Speech Communication 41, 603–623 (2003)
Vidrascu, L., Devillers, L.: Detection of real-life emotions in call centers. In: Proceedings of Interspeech, pp. 1841–1844 (2005)
Batliner, A., Steidl, S., Hacker, C., Nöth, E., Niemann, H.: Tales of tuning - prototyping for automatic classification of emotional user states. In: Proceedings of Interspeech, pp. 489–492 (2005)
Pantic, M., Rothkrantz, L.J.M.: Towards an Affect-Sensitive Multimodal Human-Computer Interaction. Proceedings of the IEEE 91, 1370–1390 (2003)
Ekman, P., Friesen, W.V.: Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978)
Pantic, M., Rothkrantz, L.J.M.: Automatic Analysis of Facial Expressions: The State of the Art. IEEE Transaction on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)
Pantic, M., Rothkrantz, L.J.M.: Expert system for automatic analysis of facial expression. Image and Vision Computing Journal 18, 881–905 (2000)
Cohen, I., Sebe, N., Chen, L., Garg, A., Huang, T.S.: Facial Expression Recognition from Video Sequences: Temporal and Static Modeling. Computer Vision and Image Understanding 91, 160–187 (2003)
Sebe, N., Sun, Y., Bakker, E., Lew, M.S., Cohen, I., Huang, T.S.: Towards Authentic Emotion Recognition. In: IEEE SMC International Conference on Systems, Man, and Cybernetics, pp. 623–628 (2004)
Den Uyl, M., Van Kuilenburg, H.: FaceReader: an online facial expression recognition system. In: Proceedings of 5th International Conference on Methods and Techniques in Behavorial Research, pp. 589–590 (2005)
De Silva, L.C., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multi-modal information. In: Proceedings of the ICICS International Conference on Information, Communications and Signal Processing, pp. 397–401 (1997)
Chen, L.S., Tao, H., Huang, T.S., Miyasato, T., Nakatsu, R.: Emotion recognition from audiovisual information. In: Proceedings of the IEEE Workshop on Multimedia Signal Processing, pp. 83–88 (1998)
De Silva, L.C., Ng, P.C.: Bimodal Emotion Recognition. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pp. 332–335 (2000)
Sebe, N., Cohen, I., Gevers, T., Huang, T.: Emotion Recognition Based on Joint Visual and Audio Cues. In: Proceedings of the ICPR International Conference on Pattern Recognition, pp. 1136–1139 (2006)
Chen, L.S.: Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction. Phd thesis (2000)
Douglas-Cowie, E., Cowie, R., Schröder, M.: A new emotion database: Considerations, sources and scope. In: Proceedings of ISCA ITRW Workshop on Speech and Emotion, pp. 39–44 (2000)
Lang, P.J.: The emotion probe - studies of motivation and attention. American Psychologist 50, 371–385 (1995)
Wagner, J., Kim, J., André, E.: From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification. In: Proceedings of the IEEE ICME International Conference on Multimedia & Expo, pp. 940–943 (2005)
Cox, C.: SALAS Sensitive Artificial Listener induction techniques. Paper presented to HUMAINE Network of Excellence Summer School (2004) (retrieved Februay 5, 2007), from http://emotion-research.net/ws/summerschool1/SALAS.ppt#259
Lazarro, N.: Why we play games: 4 keys to more emotion (paper retrieved February 5, 2007), from http://www.xeodesign.com/xeodesign_whyweplaygames.pdf
Douglas-Cowie, E., Devillers, L., Martin, J., Cowie, R., Savvidou, S., Abrilian, S., Cox, C.: Multimodal databases of everyday emotion: facing up to complexity. In: Proceedings of Interspeech, pp. 813–816 (2005)
AMI project, http://www.amiproject.org
Grootjen, M., Neerincx, M.A., Weert, J.C.M., Truong, K.P.: Measuring Cognitive Task Load on a Naval Ship: Implications of a Real World Environment. In: Proceedings of ACI (this volume) (2007)
HUMAINE project, http://emotion-research.net/
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Truong, K.P., van Leeuwen, D.A., Neerincx, M.A. (2007). Unobtrusive Multimodal Emotion Detection in Adaptive Interfaces: Speech and Facial Expressions. In: Schmorrow, D.D., Reeves, L.M. (eds) Foundations of Augmented Cognition. FAC 2007. Lecture Notes in Computer Science(), vol 4565. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-73216-7_40
Download citation
DOI: https://doi.org/10.1007/978-3-540-73216-7_40
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-73215-0
Online ISBN: 978-3-540-73216-7
eBook Packages: Computer ScienceComputer Science (R0)