Abstract
Humanity’s desire to enable machines to “understand” us drives research that seeks to uncover the mysteries of human beings and of their reactions. That is because a computer’s ability to correctly classify our emotions will lead to an enhanced experience for a user. Making use of the eye of the computer, a webcam, we can acquire human reaction data through the acquisition of facial images in response to stimuli. The data of interest in this research are changes in pupil size and gaze patterns in conjunction with classification of facial expression. Although fusion of these measurements has been considered in the past by Xiang and Kankanhalli [14] as well as Valverde et al. [15], their approach was quite different from ours. Both groups used a multimodal set-up: an eye tracker alongside a webcam and the stimulus was visual. A novel approach is to avoid costly eye trackers and rely on images acquired only from a standard webcam to measure changes in pupil size, gaze patterns and facial expression in response to auditory stimuli. The auditory mode is often preferred since luminance does not need to be accounted for, unlike visual stimulation from a monitor. The fusion of the information from these features is then used to distinguish between negative, neutral and positive emotional states. In this paper we discuss an experiment (n = 15) where the stimuli from the auditory version of the international affective picture system (IAPS) are used to elicit these three main emotions in participants. Webcam data is recorded during the experiments and advanced signal processing and feature extraction techniques are used on the resulting image files to achieve a model capable of predicting neutral, positive, and negative emotional states.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Klingner, J., Tversky, B., Hanrahan, P.: Effects of visual and verbal presentation on cognitive load in vigilance, memory and arithmetic tasks. J. of Psychophysiology 48(3), 323–332 (2011)
Petridis, S., Giannakopoulos, T., Spyropoulos, C.: Unobtrusive Low Cost Pupil Size Measurements using Web cameras. In: Proceedings of the 2nd International Workshop on Artificial Intelligence and NetMedicine, pp. 9–20 (2013)
Schwarz, L., Gamba, H., Pacheco, F., Ramos, R., Sovierzoski, M.: Pupil and Iris Detection in Dynamic Pupillometry using the OpenCV Library. In: 2012 5th Annual Congress on Image and Signal Processing, pp. 211–215 (2012)
Parlata, T., Surakka, V.: Pupil Size Variation as an Indication of Affective Processing. International Journal of Human-Computer Studies, 185–198 (2003)
Canento, F., Fred, A., Gamboa, H., Lourenco, A.: Multimodal Biosignal Sensor Data Handling for Emotion Recognition. In: Proceedings of the IEEE Sensors Conference (2011)
Xu, G., Wang, Y., Li, J., Zhou, X.: Real Time Detection of Eye Corners and Iris Center from Images Acquired by Usual Camera. In: 2009 Second International Conference Proceedings on Intelligent Networks and Intelligent Systems, pp. 401–404 (2009)
Zhai, J., Barreto, A.: Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables. In: Proceedings of the 28th IEEE EMBS International Conference, pp. 1355–1358 (2006)
Baldaci, S., Gockay, D.: Negative Sentiment in Scenarios Elicit Pupil Dilation Response: An Auditory Study. In: 2012 International Conference on Multimodal Interaction, pp. 529–532. ACM (2012)
Wang, W., Li, Z., Wang, Y., Chen, F.: Indexing Cognitive Workload Based on Pupillary Response under Luminance and Emotional Changes. In: 2013 International Conference on Intelligent User Interfaces, pp. 247–256. ACM (2013)
Partala, T., Jokiniemi, M., Surakka, V.: Pupillary Responses to Emotionally Provocative Stimuli. In: 2000 Eye Tracking Research & Applications Symposium, pp. 123–129. ACM (2000)
Babiker, A., Faye, I., Malik, A.: Non-conscious Behavior in Emotion Recognition. In: 2013 IEEE 9th International Colloquium on Signal Processing and its Applications, pp. 258–262 (2013)
Delibasis, K.K., Asvestas, P., Matsopoulos, G.K., Economopoulos, T., Assimakis, N.: A Real Time Eye-Motion Monitoring System. In: 16th International Conference on Systems, Signals and Image Processing, pp. 1–5 (2009)
Viola, P., Jones, M.: Robust Real-Time Object Detection. International Journal of Computer Vision 57(2), 137154 (2004)
Xiang, X., Kankanhalli, M.S.: A Multimodal Approach for Online Estimation of Subtle Facial Expression. In: Lin, W., Xu, D., Ho, A., Wu, J., He, Y., Cai, J., Kankanhalli, M., Sun, M.-T. (eds.) PCM 2012. LNCS, vol. 7674, pp. 402–413. Springer, Heidelberg (2012)
Valverde, L., DeLera, E., Fernandez, C.: Inferencing Emotions Through the Triangulation of Pupil Size Data, Facial Heuristics and Self-Assessment Techniques. In: 2010 Second International Conference on Mobile, Hybrid, and On-Line Learning, pp. 147–150. IEEE Computer Society (2010)
Bousefsaf, F., Maaoui, C., Pruski, A.: Remote Assessment of the Heart Rate Variability to Detect Mental State. In: 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops, pp. 348–351. IEEE (2013)
Terzis, V., Moridis, C., Economides, A.: Measuring Instant Emotions Based on Facial Expressions During Computer-Based Assessment. Personal and Ubiquitous Computing 17(1), 43–54 (2013)
Adams, R., Kleck, R.: Effects of Direct and Averted Gaze on the Perception of Facially Communicated Emotion. American Psychological Association’s Emotion 5(1), 3–11 (2005)
Bradley, M., Lang, P.: Affective Ratings of Sounds and Instruction Manual. In: The International Affective Digitized Sounds (2nd edn., IADS-2). Technical Report B-3, University of Florida, Gainesville, FL
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Sommer, N., Hirshfield, L., Velipasalar, S. (2014). Our Emotions as Seen through a Webcam. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Foundations of Augmented Cognition. Advancing Human Performance and Decision-Making through Adaptive Systems. AC 2014. Lecture Notes in Computer Science(), vol 8534. Springer, Cham. https://doi.org/10.1007/978-3-319-07527-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-07527-3_8
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07526-6
Online ISBN: 978-3-319-07527-3
eBook Packages: Computer ScienceComputer Science (R0)