Skip to main content
SpringerLink
Log in

Emotion and Reinforcement: Affective Facial Expressions Facilitate Robot Learning

  • Conference paper
Artifical Intelligence for Human Computing

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4451))

Abstract

Computer models can be used to investigate the role of emotion in learning. Here we present EARL, our framework for the systematic study of the relation between emotion, adaptation and reinforcement learning (RL). EARL enables the study of, among other things, communicated affect as reinforcement to the robot; the focus of this chapter. In humans, emotions are crucial to learning. For example, a parent—observing a child—uses emotional expression to encourage or discourage specific behaviors. Emotional expression can therefore be a reinforcement signal to a child. We hypothesize that affective facial expressions facilitate robot learning, and compare a social setting with a non-social one to test this. The non-social setting consists of a simulated robot that learns to solve a typical RL task in a continuous grid-world environment. The social setting additionally consists of a human (parent) observing the simulated robot (child). The human’s emotional expressions are analyzed in real time and converted to an additional reinforcement signal used by the robot; positive expressions result in reward, negative expressions in punishment. We quantitatively show that the “social robot” indeed learns to solve its task significantly faster than its “non-social sibling”. We conclude that this presents strong evidence for the potential benefit of affective communication with humans in the reinforcement learning loop.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ashby, F.G., Isen, A.M., Turken, U.: A Neuro-psychological theory of positive affect and its influence on cognition. Psychological Review 106(3), 529–550 (1999)

    Article  Google Scholar 

  2. Avila-Garcia, O., Cañamero, L.: Using hormonal feedback to modulate action selection in a competitive scenario. In: From Animals to Animats 8: Proc. 8th Intl. Conf. on Simulation of Adaptive Behavior, pp. 243–252. MIT Press, Cambridge (2004)

    Google Scholar 

  3. Belavkin, R.V.: On relation between emotion and entropy. In: Proc. of the AISB’04 Symposium on Emotion, Cognition and Affective Computing, pp. 1–8. AISB Press (2004)

    Google Scholar 

  4. Berridge, K.C.: Pleasures of the brain. Brain and Cognition 52, 106–128 (2003)

    Article  Google Scholar 

  5. Blanchard, A.J., Cañamero, L.: Modulation of exploratory behavior for adaptation to the context. In: Proc. of the AISB’06 Symposium on Biologically Inspired Robotics (Biro-net), pp. 131–137. AISB Press (2006)

    Google Scholar 

  6. Botelho, L.M., Coelho, H.: Information processing, motivation and decision making. In: Proc. 4th International Workshop on Artificial Intelligence in Economics and Management (1998)

    Google Scholar 

  7. Breazeal, C.: Affective interaction between humans and robots. In: Kelemen, J., Sosík, P. (eds.) ECAL 2001. LNCS (LNAI), vol. 2159, pp. 582–591. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  8. Breazeal, C., Brooks, R.: Robot emotion: A functional perspective. In: Fellous, J.-M., Arbib, M. (eds.) Who needs emotions: The brain meets the robot, pp. 271–310. Oxford University Press, Oxford (2004)

    Google Scholar 

  9. Breazeal, C., Scassellati, B.: Robots that imitate humans. Trends in Cognitive Sciences 6(11), 481–487 (2002)

    Article  Google Scholar 

  10. Breazeal, C., Velasquez, J.: Toward teaching a robot ‘infant’ using emotive communication acts. In: Edmonds, B., Dautenhahn, K. (eds.) Socially Situated Intelligence: a workshop held at SAB’98, Zürich. University of Zürich Technical Report, pp. 25–40 (1998)

    Google Scholar 

  11. Broekens, J., Kosters, W.A., Verbeek, F.J.: On emotion, anticipation and adaptation: Investigating the potential of affect-controlled selection of anticipatory simulation in artificial adaptive agents. In press (2007)

    Google Scholar 

  12. Cañamero, D.: Designing emotions for activity selection. Dept. of Computer Science Technical Report DAIMI PB 545. University of Aarhus, Denmark (2000)

    Google Scholar 

  13. Charman, T., Baird, G.: Practitioner review: Diagnosis of autism spectrum disorder in 2- and 3-year-old children. Journal of Child Psychology and Psychiatry 43(3), 289–305 (2002)

    Article  Google Scholar 

  14. Clore, G.L., Gasper, K.: Feeling is believing: Some affective influences on belief. In: Frijda, N., Manstead, A.S.R., Bem, S. (eds.) Emotions and Beliefs, pp. 10–44. Cambridge Univ. Press, Cambridge (2000)

    Google Scholar 

  15. Cos-Aguilera, I., et al.: Ecological integration of affordances and drives for behaviour selection. In: Proc. of the Workshop on Modeling Natural Action Selection, pp. 225–228. AISB Press (2005)

    Google Scholar 

  16. Custers, R., Aarts, H.: Positive affect as implicit motivator: On the nonconscious operation of behavioral goals. Journal of Personality and Social Psychology 89(2), 129–142 (2005)

    Article  Google Scholar 

  17. Damasio, A.R.: Descartes’ error. Penguin, New York (1994)

    Google Scholar 

  18. Doya, K.: Metalearning and neuromodulation. Neural Networks 15(4), 495–506 (2002)

    Article  Google Scholar 

  19. Dreisbach, G., Goschke, K.: How positive affect modulates cognitive control: Reduced perseveration at the cost of increased distractibility. Journal of Experimental Psychology: Learning, Memory, and Cognition 30(2), 343–353 (2004)

    Article  Google Scholar 

  20. Fong, T., Nourbakhsh, I., Dautenhahn, K.: A survey of socially interactive robots. Robots and Autonomous Systems 42, 143–166 (2003)

    Article  MATH  Google Scholar 

  21. Forgas, J.P.: Feeling is believing? The role of processing strategies in mediating affective influences in beliefs. In: Frijda, N., Manstead, A.S.R., Bem, S. (eds.) Emotions and Beliefs, pp. 108–143. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  22. Frijda, N.H., Mesquita, B.: Beliefs through Emotions. In: Frijda, N., Manstead, A.S.R., Bem, S. (eds.) Emotions and Beliefs, pp. 45–77. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  23. Frijda, N.H., Manstead, A.S.R., Bem, S.: The influence of emotions on beliefs. In: Frijda, N., Manstead, A.S.R., Bem, S. (eds.) Emotions and Beliefs, pp. 1–9. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  24. Gandanho, S.C.: Learning behavior-selection by emotions and cognition in a multi-goal robot task. Journal of Machine Learning Research 4, 385–412 (2003)

    Article  Google Scholar 

  25. Gasper, K., Clore, L.G.: Attending to the big picture: Mood and global versus local processing of visual information. Psychological Science 13(1), 34–40 (2002)

    Article  Google Scholar 

  26. Isbell Jr., C.L., et al.: A social reinforcement learning agent. In: Proceedings of the fifth international conference on Autonomous agents, pp. 377–384. ACM Press, New York (2001)

    Chapter  Google Scholar 

  27. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)

    Google Scholar 

  28. Lin, L.J.: Reinforcement learning for robots using neural networks. Doctoral dissertation. Carnegie Mellon University, Pittsburgh (1993)

    Google Scholar 

  29. Mehrabian, A.: Basic Dimensions for a General Psychological Theory. OG&H Publishers, Cambridge (1980)

    Google Scholar 

  30. Mitsunaga, N., et al.: Robot behavior adaptation for human-robot interaction based on policy gradient reinforcement learning. In: Proc. Of the International Conference on Intelligent Robots and Systems (IROS), pp. 218–225. IEEE Computer Society Press, Los Alamitos (2005)

    Google Scholar 

  31. Thrun, S., et al.: A tour-guide robot that learns. In: Burgard, W., Christaller, T., Cremers, A.B. (eds.) KI 1999. LNCS (LNAI), vol. 1701, pp. 14–26. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  32. Pantic, M., et al.: Human computing and machine understanding of human behavior: A Survey. In: Proc. ACM Int’l Conf. Multimodal Interfaces, pp. 239–248 (2006)

    Google Scholar 

  33. Pantic, M., Rothkranz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12), 1424–1445 (2000)

    Article  Google Scholar 

  34. Papudesi, V.N., Huber, M.: Learning from reinforcement and advice using composite reward functions. In: Proc. Of the 16th International FLAIR Conference, pp. 361–365. AAAI Press, Menlo Park (2003)

    Google Scholar 

  35. Papudesi, V.N., Huber, M.: Interactive refinement of control policies for autonomous robots. In: Proc. of the 10th IASTED International Conference on Robotics and Applications, Honolulu HI, IASTED (2004)

    Google Scholar 

  36. Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)

    Google Scholar 

  37. Picard, R.W., et al.: Affective learning — A manifesto. BT Technology Journal 22(4), 253–269 (2004)

    Article  Google Scholar 

  38. Rolls, E.T.: Précis of The brain and emotion. Behavioral and Brain Sciences 23, 177–191 (2000)

    Article  Google Scholar 

  39. Russell, J.A.: Core affect and the psychological construction of emotion. Psychological Review 110(1), 145–172 (2003)

    Article  Google Scholar 

  40. Scherer, K.R.: Appraisal considered as a process of multilevel sequential checking. In: Scherer, K.R., Schorr, A., Johnstone, T. (eds.) Appraisal processes in emotion: Theory, Methods, Research, pp. 92–120. Oxford Univ. Press, New York (2001)

    Google Scholar 

  41. Sutton, R., Barto, A.: Reinforcement learning: An introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  42. Ogata, T., Sugano, S., Tani, J.: Open-end human robot interaction from the dynamical systems perspective: Mutual adaptation and incremental learning. In: Orchard, B., Yang, C., Ali, M. (eds.) IEA/AIE 2004. LNCS (LNAI), vol. 3029, pp. 435–444. Springer, Heidelberg (2004)

    Google Scholar 

  43. Thomaz, A.L., Breazeal, C.: Teachable characters: User studies, design principles, and learning performance. In: Gratch, J., et al. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 395–406. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  44. Thomaz, A.L., Breazeal, C.: Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In: Proc. of the 21st National Conference on Artificial Intelligence, AAAI Press, Menlo Park (2006b)

    Google Scholar 

  45. Thomaz, A.L., Hoffman, G., Breazeal, C.: Real-time interactive reinforcement learning for robots. In: Proc. of AAAI Workshop on Human Comprehensible Machine Learning, Pittsburgh, PA (2005)

    Google Scholar 

  46. Velasquez, J.D.: A computational framework for emotion-based control. In: SAB’98 Workshop on Grounding Emotions in Adaptive Systems (1998)

    Google Scholar 

  47. Wright, I.: Reinforcement learning and animat emotions. In: From Animals to Animats 4: Proc. of the 4th International Conference on the Simulation of Adaptive Behavior (SAB), pp. 272–284. MIT Press, Cambridge (1996)

    Google Scholar 

  48. OpenCV: http://www.intel.com/technology/computing/opencv/index.htm

Download references

Author information

Authors and Affiliations

  1. Leiden Institute of Advanced Computer Science, Leiden University, Niels Bohrwerg 1, 2333CA Leiden, The Netherlands

    Joost Broekens

Authors
  1. Joost Broekens
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Thomas S. Huang Anton Nijholt Maja Pantic Alex Pentland

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Berlin Heidelberg

About this paper

Cite this paper

Broekens, J. (2007). Emotion and Reinforcement: Affective Facial Expressions Facilitate Robot Learning. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds) Artifical Intelligence for Human Computing. Lecture Notes in Computer Science(), vol 4451. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72348-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-72348-6_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-72346-2

  • Online ISBN: 978-3-540-72348-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation