Abstract
A system that could automatically analyze the facial actions in real-time has applications in a wide range of different fields. The previous facial action unit (AU) recognition approaches often recognize AUs or certain AU combinations in dividually and statically, ignoring the semantic relationships among AUs and the dynamics of AUs. Hence, these approaches cannot always recognize AUs reliably, robustly, and consistently due to the richness, ambiguity, and the dynamic nature of facial actions. In this work, a novel AU recognition system is proposed to sys tematically account for the relationships among AUs and their temporal evolutions based on a dynamic Bayesian network (DBN). The DBN provides a coherent and unified hierarchical probabilistic framework to represent probabilistic relationships among various AUs and to account for the temporal changes in facial action devel opment. Within the proposed system, robust computer vision techniques are used to obtain AU measurements. And such AU measurements are then applied as evidence to the DBN for inferring various AUs. The experiments show that the integration of AU relationships and AU dynamics with AU measurements yields significant improvement of AU recognition, especially under realistic environments including illumination variation, face pose variation, and occlusion.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bartlett, M. S., Littlewort, G., Frank, M. G., Lainscsek, C., Fasel, I., & Movellan, J. R. (2005). Recognizing facial expression: Machine learning and application to spontaneous behavior.Pro ceedings of CVPR052, pp. 568–573
Bartlett, M. S., Littlewort, G., Movellan, J. R., Frank, M. G. (2007). Auto facs coding URLhttp://mplab.ucsd.edu/grants/project1/research/Fully-Auto-FACS-Coding.html
Bassili, J. N. (1979). Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face.Journal of Personality and Social Psychology 37(11), 2049–2058.
Bazzo, J. J., Lamar, M. V. (2004). Recognizing facial actions using gabor wavelets with neutral face average difference.Proceedings of FGR04, pp. 505–510.
Cohen, I., Cozman, F. G., Sebe, N., Cirelo, M. C., & Huang, T. S. (2004). Semisupervised learning of classifiers: Theory, algorithms, and their application to human-computer interaction.IEEE Transactions on PAMI26(12), 1553–1567.
Cohen, I., Sebe, N., Garg, A., Chen, L., & Huang, T. (2003). Facial expression recognition from video sequences: Temporal and static modeling.Computer Vision and Image Understanding 91(1–2), 160–187.
Cohn, J. F., Reed, L. I., Ambadar, Z., Xiao, J., & Moriyama, T. (2004) Automatic analysis and recognition of brow actions and head motion in spontaneous facial behavior.In Proceedings of IEEE Int'l Conf on Systems, Man, and Cybernetics1, pp. 610–616
Cohn, J. F., & Zlochower, A. (1995). A computerized analysis of facial expression: Feasibility of automated discrimination.American Psychological Society, New York, New York.
Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Classifying facial actions.IEEE Transactions, on PAMI 21(10), 974–989.
Ekman, P., & Friesen, W. V. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.
Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system: The manual. Re search Nexus, Div. Salt Lake City, UT: Network Information Research Corp.
El Kaliouby, R., & Robinson, P. K. (2004). Real-time inference of complex mental states from facial expressions and head gestures.CVPRW'04 10, 154.
Fasel, B., & Luettin, J. (2000) Recognition of asymmetric facial action unit activities and intensi ties.Proceedings of ICPR00 1, pp. 1100–1103.
Gu, H., & Ji, Q. (2004). Facial event classification with task oriented dynamic bayesian network.Proceedings of CVPR042, pp. 870–875.
Heckerman, D. (1995). A tutorial on learning with Bayesian networks. Tech Report MSR-TR-95-06, Microsoft Research.
Heckerman, D., Geiger, D., Chickering, D. M. (1995). Learning Bayesian networks: The combi nation of knowledge and statistical data.Machine Learning 20(3), 197–243.
Heller, M., & Haynal, V. (1997). Depression and suicide faces. In: P., Ekman Rosenberg & E (Eds.)What the face reveals, (pp. 339–407). Oxford University Press. New York.
Kanade, T., Cohn, J. F., Tian, Y. (2000). Comprehensive database for facial expression analysis.Proceedings of FGR00, pp. 46–53.
Kapoor, A., Qi, Y., & Picard, R. W. (2003). Fully automatic upper facial action recognition.IEEE Int'l Workshop on Analysis and Modeling of Faces and Gestures, pp. 195–202.
Kjaerulff, U. (1995) dhugin: A computational system for dynamic time-sliced bayesian networks.International Journal of Forecasting, Special issue on Probability Forecasting11, 89–111.
Korb, K. B., & Nicholson, A. E. (2004).Bayesian artificial intelligence. London; Chapman and Hall/CRC.
Lanitis, A., Taylor, C. J., Cootes, T. F. (1997). Automatic interpretation and coding of face images using flexible models.IEEE Transactions on PAMI 19(7), 743–756.
Lien, J. J., Kanade, T., Cohn, J. F., & Li, C. (2000). Detection, tracking, and classification of action units in facial expression.Journal of Robotics and Autonomous Systems 31, 131–146.
Pantic M, Bartlett, M (2007). Machine analysis of facial expressions. In K. Delac, Grgic M. (Eds.)Face recognition, (pp. 377–416), Vienna: I-Tech Education.
Pantic, M., & Rothkrantz, L. J. M. (2004). Facial action recognition for facial expression analysis from static face images.IEEE Transactions on Systems, Man, and Cybernetics-Part B: Cyber netics 34(3),1449–1461.
Pantic, M., Valstar, M., Rademaker, R., Maat, L. (2005). Web-based database for facial expression analysis.Proceedings of IEEE Int'l Conf on Multmedia and Expo (ICME05)pp. 317–321.
Phillips, P. J., Flynn, P. J., Scruggs, T, Bowyer, K. W., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005). Overview of the face recognition grand challenge.Proceedings of CVPR051, pp. 947–954
Schwarz, G. (1978). Estimating the dimension of a model.The Annals of Statistics 6, 461–464.
Smith, E., Bartlett, M. S., & Movellan, J. R. (2001). Computer recognition of facial actions: A study of co-articulation effects.Proceedings of the 8th Annual Joint Symposium on Neural Computation.
Spiegelhalter, D., & Lauritzen, S. (1990). Sequential updating of conditional probabilities on di rected graphical structures.Networks 20, 579–605.
Tian, Y., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis.IEEE Transact. on PAMI 23(2), 97–115.
Tian, Y., Kanade, T., & Cohn, J. F. (2002). Evaluation of Gabor-wavelet-based facial action unit recognition in image sequences of increasing complexity.Proceedings of FGR02(pp. 218– 223).
Tong, Y., Liao, W., & Ji, Q. (2007). Intelligent systems lab (isl) database URLhttp://www.ecse. rpi.edu/homepages/cvrl/database/database.html
Valstar, M. F., Patras, I., & Pantic, M. (2005). Facial action unit detection using probabilistic ac tively learned support vector machines on tracked facial point data.Proceedings of CVPRW'05 on Vision for Human-Computer Interactionpp. 76.
Wang, P., Green, M. B., Ji, Q., & Wayman, J. (2005). Automatic eye detection and its validation.IEEE Workshop on Face Recognition Grand Challenge Experiments (with CVPR)3.
Williams, A. C. (2002). Facial expression of pain: An evolutionary account.Behavioral & Brain Sciences 25(4), 439–488.
Zhang, Y., & Ji, Q. (2005). Active and dynamic information fusion for facial expression under standing from image sequences.IEEE Transactions on PAMI 27(5), 699–714.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag London Limited
About this chapter
Cite this chapter
Tong, Y., Liao, W., Ji, Q. (2009). Automatic Facial Action Unit Recognition by Modeling Their Semantic And Dynamic Relationships. In: Tao, J., Tan, T. (eds) Affective Information Processing. Springer, London. https://doi.org/10.1007/978-1-84800-306-4_10
Download citation
DOI: https://doi.org/10.1007/978-1-84800-306-4_10
Publisher Name: Springer, London
Print ISBN: 978-1-84800-305-7
Online ISBN: 978-1-84800-306-4
eBook Packages: Computer ScienceComputer Science (R0)