Skip to main content
SpringerLink
Log in

Improving Simulation of Continuous Emotional Facial Expressions by Analyzing Videos of Human Facial Activities

  • Conference paper
PRIMA 2014: Principles and Practice of Multi-Agent Systems (PRIMA 2014)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 8861))

Abstract

Conversational agents are receiving significant attention from multi-agent and human computer interaction research societies. In order to make conversational agents more believable and friendly, giving them the ability to express emotions is one of research fields which have drawn a lot of attention lately. In this paper, we propose a work on analysis of how emotional facial activities happen temporally. Our goal is to find the temporal patterns of facial activity of six basic emotions in order to improve the simulation of continuous emotional facial expressions on a 3D face of an embodied agent. Using facial expression recognition techniques, we first analyze a spontaneous video database in order to consider how facial activities are related to six basic emotions temporally. From there, we bring out the general temporal patterns for facial expressions of the six basic emotions. Then, based on the temporal patterns, we propose a scheme for displaying continuous emotional states of a conversational agent on a 3D face.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. http://mmifacedb.eu/

  2. Albrecht, I.: Faces and Hands- Modeling and animating anatomical and photorealistic models with regard to the communicative competence of virtual humans. PhD thesis, University at des Saarlandes (2005)

    Google Scholar 

  3. Albrecht, I., Haber, J., Kähler, K., Schröder, M., Seidel, H.-P.: May i talk to you?:-) facial animation from text. In: Proceedings Pacific Graphics 2002, pp. 77–86 (2002)

    Google Scholar 

  4. Arnold, M.B.: Emotion and personality. Psychological aspects. Columbia University Press, New York (1960)

    Google Scholar 

  5. Bui, T.D., Heylen, D., Nijholt, A.: Building embodied agents that experience and express emotions: A football supporter as an example. In: Proc. CASA2004. Computer Graphics Society (2004)

    Google Scholar 

  6. Bui, T.D., Heylen, D., Nijholt, A.: Combination of facial movements on a 3d talking head. In: Proc. CGI 2004. IEEE Computer Society (2004)

    Google Scholar 

  7. Bui, T.D., Heylen, D., Poel, M., Nijholt, A.: Generation of facial expressions from emotion using a fuzzy rule based system. In: Stumptner, M., Corbett, D.R., Brooks, M. (eds.) Canadian AI 2001. LNCS (LNAI), vol. 2256, pp. 83–95. Springer, Heidelberg (2001)

    Google Scholar 

  8. Bui, T.D., Heylen, D., Poel, M., Nijholt, A.: ParleE: An adaptive plan based event appraisal model of emotions. In: Jarke, M., Koehler, J., Lakemeyer, G. (eds.) KI 2002. LNCS (LNAI), vol. 2479, pp. 129–143. Springer, Heidelberg (2002)

    Google Scholar 

  9. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Computer Vision and Image Understanding 61(1), 38–59 (1995)

    Article  Google Scholar 

  10. Darwin, C.: The expression of the emotions in man and animals. Univerity of Chicago Press, Chicago (1872/1965)

    Google Scholar 

  11. DeCarlo, D.C., Revilla, M.S., Venditti, J.: Making discourse visible: Coding and animating conversational facial displays. In: Computer Animation (2002)

    Google Scholar 

  12. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. In: David, J. (ed.) Proceedings of the National Academy of Sciences. New York University, New York (2014)

    Google Scholar 

  13. Ekman, P.: Emotion in the human face. Cambridge University Press, Cambridge (1982)

    Google Scholar 

  14. Ekman, P., Friesen, W.V.: Unmasking the Face: A Guide To Recognizing Emotions From Facial Clues. Prentice-Hall, Englewood Cliffs (1975)

    Google Scholar 

  15. Ekman, P., Friesen, W.V.: Facial Action Coding System. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  16. Ekman, P., Hager, J.: Facial action coding system affect interpretation database (facsaid), http://face-and-emotion.com/dataface/facsaid/description.jsp (retrieved)

  17. Ekman, P., Rosenberg, E.L.: What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS), Illustrated Edition. Oxford University Press (1997)

    Google Scholar 

  18. El-Nasr, M.S., Yen, J., Ioerger, T.R.: FLAME-fuzzy logic adaptive model of emotions. Autonomous Agents and Multi-Agent Systems 3(3), 219–257 (2000)

    Article  Google Scholar 

  19. Essa, I.A., Pentland, A.: A vision system for observing and extracting facial action parameters. In: Proceedings of IEEE Computer Vision and Pattern Recognition Conference (1994)

    Google Scholar 

  20. Friesen, W., Ekman, P.: EMFACS-7: Emotional Facial Action Coding System. University of California, California (unpublished manual, 1983)

    Google Scholar 

  21. Hayes-Roth, B., van Gent, R.: Story-making with improvisational puppets. In: Johnson, W.L., Hayes-Roth, B. (eds.) Proceedings of the 1st International Conference on Autonomous Agents, pp. 1–7. ACM Press, New York (1997)

    Chapter  Google Scholar 

  22. Isbister, K., Doyle, P.: Design and evaluation of embodied conversational agents: a proposed taxonomy. In: Proceedings of AAMAS 2002 Workshop on Embodied Conversational Agents? Let?s Specify and Evaluate Them!, Bologna, Italy (2002)

    Google Scholar 

  23. Izard, C.E.: Emotions and facial expressions: A perspective from differential emotions theory. In: Russell, J.A., Fernandez-Dols, J.M. (eds.) The Psychology of Facial Expression. Maison des Sciences de l’Homme and Cambridge University Press (1997)

    Google Scholar 

  24. Kappas, A.: What facial activity can and cannot tell us about emotions. In: Katsikitis, M. (ed.) The Human Face: Measurement and Meaning, pp. 215–234. Kluwer Academic Publishers, Dordrecht (2003)

    Chapter  Google Scholar 

  25. King, S.A., Parent, R.E., Olsafsky, B.: An anatomically-based 3d parametric lip model to support facial animation and synchronized speech. In: Proceedings of Deform 2000, pp. 7–19 (2000)

    Google Scholar 

  26. Kshirsagar, S., Magnenat-Thalmann, N.: A multilayer personality model. In: Proceedings of 2nd International Symposium on Smart Graphics, pp. 107–115. ACM Press (2002)

    Google Scholar 

  27. Kurlander, D., Skelly, T., Salesin, D.: Comic chat. In: SIGGRAPH 1996: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 225–236 (1996)

    Google Scholar 

  28. Latta, C., Alvarado, N., Adams, S.S., Burbeck, S.: An expressive system for animating characters or endowing robots with affective displays. In: Society for Artificial Intelligence and Social Behavior (AISB), 2002 Annual Conference, Symposium on Animating Expressive Characters for Social Interactions (2002)

    Google Scholar 

  29. Mohammad Mavadati, S., Mahoor Mohammad, H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: A spontaneous facial action intensity database. IEEE Transactions on Affective Computing 4(2), 151–160 (2013)

    Article  Google Scholar 

  30. Ngo, T.D., Bui, T.D.: When and how to smile: Emotional expression for 3D conversational agents. In: Ghose, A., Governatori, G., Sadananda, R. (eds.) PRIMA 2007. LNCS, vol. 5044, pp. 349–358. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  31. Paiva, A., Dias, J., Sobral, D., Aylett, R., Sobreperez, P., Woods, S., Zoll, C., Hall, L.: Caring for agents and agents that care: Building empathic relations with synthetic agents. In: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 194–201. IEEE Computer Society (1996)

    Google Scholar 

  32. Pantic, M., Patras, I.: Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proc. IEEE Conf. Systems, Man and Cybernetics, vol. 4, pp. 3358–3363 (2005)

    Google Scholar 

  33. Pantic, M., Valstar, M.F., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: Proc. 13th ACM Int’l Conf. Multimedia and Expo, pp. 317–321 (2005)

    Google Scholar 

  34. Perlin, K., Goldberg, A.: Improv: A system for scripting interactive actors in virtual worlds. Computer Graphics 30(Annual Conference Series), 205–216 (1996)

    Google Scholar 

  35. Picard, R.: Affective Computing. MIT Press, Cambridge (1997)

    Google Scholar 

  36. Plutchik, R.: Emotions: A general psychoevolutionary theory. In: Scherer, K.R., Ekman, P. (eds.) Approaches to Emotion. Lawrence Erlbaum, London (1984)

    Google Scholar 

  37. Raouzaiou, A., Karpouzis, K., Kollias, S.D.: Online gaming and emotion representation. In: García, N., Salgado, L., Martínez, J.M. (eds.) VLBV 2003. LNCS, vol. 2849, pp. 298–305. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  38. Reilly, W.S.: Believable social and emotional agents. Technical Report Ph.D. Thesis. Technical Report CMU-CS-96-138, Carnegie Mellon University, Pittsburgh, PA, USA (1996)

    Google Scholar 

  39. Russell, J.A.: Reading emotions from and into faces: Resurrecting a dimensional-contextual perspective. In: Russell, J.A., Fernndez-Dols, J.M. (eds.) The Psychology of Facial Expression. Cambridge University Press, New York (1997)

    Google Scholar 

  40. Scherer, K.R.: What does facial expression express. In: Strongman, K. (ed.) International Review of Studies on Emotion, vol. 2. Wiley, Chichester (1992)

    Google Scholar 

  41. Stern, A., Frank, A., Resner, B.: Virtual petz: A hybrid approach to creating autonomous, lifelike dogz and catz. In: Sycara, K.P., Wooldridge, M. (eds.) Proceedings of the 2nd International Conference on Autonomous Agents (Agents 1998), pp. 334–335. ACM Press, New York (1998)

    Google Scholar 

  42. Tanguy, E.: Emotions: the Art of Communication Applied to Virtual Actors. PhD thesis, Universit of Bath (2006)

    Google Scholar 

  43. Tian, Y., Kanade, T., Cohn, J.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Analysis and Machine Intelligence. 23(2), 97–115 (2001)

    Article  Google Scholar 

  44. Tomkins, S.S.: Affect, Imagery, Consciousness (Volume 1): The Positive Affects. Springer, New York (1962)

    Google Scholar 

  45. Velásquez, J.D.: Modeling emotions and other motivations in synthetic agents. In: Proceedings of the 14th National Conference on Artificial Intelligence and 9th Innovative Applications of Artificial Intelligence Conference (AAAI-97/IAAI 1997), pp. 10–15. AAAI Press, Menlo Park (1997)

    Google Scholar 

  46. Viola, P., Jones, M.: Robust real-time object detection, Tech. rep., Cambridge Research Laboratory Technical report series. (2) (2001)

    Google Scholar 

  47. Wallhoff, F.: The facial expressions and emotions database homepage (feedtum), http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.html

Download references

Author information

Authors and Affiliations

  1. University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam

    Thi Duyen Ngo, Thi Hong Nhan Vu, Viet Ha Nguyen & The Duy Bui

Authors
  1. Thi Duyen Ngo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thi Hong Nhan Vu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Viet Ha Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. The Duy Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Computer Science and Software Engineering, University of Wollongong, Northfields Avenue, 2522, Wollongong, NSW, Australia

    Hoa Khanh Dam

  2. Imperial College, SW7 2BT, London, UK

    Jeremy Pitt

  3. University of Electronic Science and Technology of China, 610051, Chengdu, Sichuan, China

    Yang Xu

  4. NICTA Queensland Research Laboratory, GPO Box 2434, 4001, Brisbane, QLD, Australia

    Guido Governatori

  5. Nagoya Institute of Technology, Gokiso-cho, 466-8555, Showa-ku, Nagoya, Japan

    Takayuki Ito

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer International Publishing Switzerland

About this paper

Cite this paper

Ngo, T.D., Vu, T.H.N., Nguyen, V.H., Bui, T.D. (2014). Improving Simulation of Continuous Emotional Facial Expressions by Analyzing Videos of Human Facial Activities. In: Dam, H.K., Pitt, J., Xu, Y., Governatori, G., Ito, T. (eds) PRIMA 2014: Principles and Practice of Multi-Agent Systems. PRIMA 2014. Lecture Notes in Computer Science(), vol 8861. Springer, Cham. https://doi.org/10.1007/978-3-319-13191-7_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-13191-7_18

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-13190-0

  • Online ISBN: 978-3-319-13191-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation