Abstract
Facial expressions are formed through complicated muscular actions and can be taxonomized using the Facial Action Coding System (FACS). FACS breaks down human facial expressions into discreet action units (AUs) and often combines them together to form more elaborate expressions. In this paper, we present a comparative analysis of performance of automated facial expression recognition from thermal facial videos, visual facial videos, and their fusion. The feature extraction process consists of first placing regions of interest (ROIs) at 13 fiducial regions on the face that are critical for evaluating all action units, then extracting mean value in each of the ROIs, and finally applying principal component analysis (PCA) to extract the deviation from neutral expression at each of the corresponding ROIs. To classify facial expressions, we train a feed-forward multilayer perceptron with the standard deviation expression profiles obtained from the feature extraction stage. Our experimental results depicts that the thermal imaging modality outperforms visual modality, and hence overcomes some of the shortcomings usually noticed in the visual domain due to illumination and skin complexion variations. We have also shown that the decision level fusion of thermal and visual expression classification algorithms gives better results than either of the individual modalities.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Fasel, B., Luettin, J.: Automatic facial expression analysis: A survey. Pattern Recognition 36, 259–275 (1999)
Pantic, M., Member, S., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)
Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: Proc. IEEE Int’l Conf. Systems, Man and Cybernetics, pp. 592–597 (2004)
Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: Machine learning and application to spontaneous behavior. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, San Diego, California, pp. 568–573. IEEE Computer Society (2005)
Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16, 172–187 (2007)
Whitehill, J., Littlewort, G., Pasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2106–2111 (2009)
Socolinsky, D.A., Wolff, L.B., Neuheisel, J.D., Eveland, C.K.: Illumination invariant face recognition using thermal infrared imagery. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, p. 527 (2001)
Kong, S.G., Heo, J., Abidi, B.R., Paik, J., Abidi, M.A.: Recent advances in visual and infrared face recognition - a review. Computer Vision and Image Understanding 97, 103–135 (2005)
Khan, M.M., Ingleby, M., Ward, R.D.: Automated facial expression classification and affect interpretation using infrared measurement of facial skin temperature variations. ACM Trans. Auton. Adapt. Syst. 1, 91–113 (2006)
Yoshitomi, Y., Miyawaki, N., Tomita, S., Kimura, S.: Facial expression recognition using thermal image processing and neural network. Robot and Human Communication, 380–385 (1997)
Trujillo, L., Olague, G., Hammoud, R., Hernandez, B.: Automatic feature localization in thermal images for facial expression recognition. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005) - Workshops, vol. 03, pp. 14–21. IEEE Computer Society, Washington, DC (2005)
Socolinsky, D.A., Selinger, A.: Thermal face recognition over time. In: Proceedings of 17th International Conference on of the Pattern Recognition, ICPR 2004, vol. 4, pp. 187–190. IEEE Computer Society, Washington, DC (2004)
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press (1978)
Ekman, P., Friesen, W., Hager, J.C.: Facial Action Coding System Investigator’s Guide (2002)
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explorations 11 (2009)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wesley, A., Buddharaju, P., Pienta, R., Pavlidis, I. (2012). A Comparative Analysis of Thermal and Visual Modalities for Automated Facial Expression Recognition. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2012. Lecture Notes in Computer Science, vol 7432. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33191-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-642-33191-6_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33190-9
Online ISBN: 978-3-642-33191-6
eBook Packages: Computer ScienceComputer Science (R0)