Modelling Behaviour Cycles for Life-Long Learning in Motivated Agents

  • Kathryn Merrick
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5361)


Natural systems such as plants, animals and humans exhibit behaviour that forms distinct, rhythmic cycles. These cycles permit individuals and societies to learn, adapt and evolve in complex, dynamic environments. This paper introduces a model of behaviour cycles for artificial systems. This model provides a new way to conceptualise and evaluate life-long learning in artificial agents. The model is demonstrated for evaluating the sensitivity of motivated reinforcement learning agents. Results show that motivated reinforcement learning agents can learn behaviour cycles that are relatively robust to changes in motivation parameters.


Behaviour cycles motivation life-long learning reinforcement learning sensitivity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahlgren, A., Halberg, F.: Cycles of nature: an introduction to biological rhythms. National Teachers Association, Washington (1990)Google Scholar
  2. 2.
    Bondu, A., Lemaire, V.: Active learning using adaptive curiosity. In: Proceedings of the Seventh International Conference on Epigenetic Robotics, Lund University (2007)Google Scholar
  3. 3.
    Brown, F.A.: Persistent activity rhythms in the oyster. American Journal of Physiology 178(33), 510–514 (1954)Google Scholar
  4. 4.
    Chavanne, T.: Variation in risk taking behaviour among female college students as a function of the menstrual cycle. Evolution and Human Behaviour 19(1), 27–32 (2003)CrossRefGoogle Scholar
  5. 5.
    Dunlap, J., Loros, J., De Coursey, P.: Chronobiology: biological timekeeping. Sinauer Associates (2003)Google Scholar
  6. 6.
    Ehrlich, P., Raven, P.: Butterflies and plants: a study in co-evolution. Evolution 18, 584–608 (1964)CrossRefGoogle Scholar
  7. 7.
    Green, R.G., Beatty, W.W., Arkin, R.M.: Human motivation: physiological, behavioural and social approaches. Allyn and Bacon, Inc., Massachussets (1984)Google Scholar
  8. 8.
    Kolb, D.A., Rubin, I.M., McIntyre, J.M. (eds.): Organizational Psychology: Readings on Human Behaviour in Organizations. Prentice-Hall, Englewood Cliffs (1984)Google Scholar
  9. 9.
    Laird, J., van Lent, M.: Interactive computer games: human-level AI’s killer application. In: National Conference on Artificial Intelligence (AAAI), pp. 1171–1178 (2000)Google Scholar
  10. 10.
    Macindoe, O., Maher, M.L., Merrick, K.: Agent based intrinsically motivated intelligent environments, Handbook on Mobile and Ubiquitous Computing: Innovations and Perspectives. American Scientific Publishers (2008)Google Scholar
  11. 11.
    Mac Namee, B., Dobbyn, S., Cunningham, P., O’Sulivan, C.: Simulating virtual humans across diverse situations. In: Rist, T., Aylett, R.S., Ballin, D., Rickel, J. (eds.) IVA 2003. LNCS, vol. 2792, pp. 159–163. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  12. 12.
    Merrick, K.: Modelling motivation for adaptive non-player characters in dynamic computer game worlds. ACM Computers in Entertainment 5(4) (2007)Google Scholar
  13. 13.
    Merrick, K., Maher, M.L.: Motivated reinforcement learning for non-player characters in persistent computer game worlds. In: ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, Los Angeles, USA (2006)Google Scholar
  14. 14.
    Merrick, K., Maher, M.L.: Motivated reinforcement learning for adaptive characters in open-ended simulation games. In: ACM SIGCHI International Conference on Advances in Computer Entertainment Technology, Salzburg, Austria, pp. 127–134 (2007)Google Scholar
  15. 15.
    Nefedov, S.A.: A Model of Demographic Cycles in Traditional Societies: The Case of Ancient China. Evolution and History 3(1), 69–80 (2004)Google Scholar
  16. 16.
    Nilsson, N.: Introduction to machine learning (accessed January 2006),
  17. 17.
    Oudeyer, P.Y., Kaplan, F., Hafner, V.: Intrinsic motivation systems for autonomous mental development. IEEE Transactions on Evolutionary Computation 11(2), 265–286 (2007)CrossRefGoogle Scholar
  18. 18.
    Singh, S., Barto, A.G., Chentanex, N.: Intrinsically motivated reinforcement learning. In: Advances in Neural Information Processing Systems 17 (NIPS), pp. 1281–1288 (2005)Google Scholar
  19. 19.
    Stoytchev, S.: Five basic principles of developmental robotics. In: NIPS Workshop on Grounding, Perception, Knowledge, and Cognition (2006)Google Scholar
  20. 20.
    Thrun, S., Mitchell, T.: Lifelong robot learning. Robotics and Autonomous Systems (1993)Google Scholar
  21. 21.
    Usher, D.: The dynastic cycle and the stationary state. The American Economic Review 79, 1031–1044 (1989)Google Scholar
  22. 22.
    Wever, R.: Human circadian rhythms under the influence of weak electric fields and the different aspects of these studies. Intn’l Journal of Biometeorology 17(3), 227–232 (1973)CrossRefGoogle Scholar
  23. 23.
    Winberg, S., Balkenius, C.: Generalization and specialization in reinforcement learning. In: Proceedings of the 7th Itn’l Conference on Epigenetic Robotics, Lund University (2007)Google Scholar
  24. 24.
    Zimecki, M.: The lunar cycle: effects on human and animal behaviour and physiology. Postepy Hig. Med. Dosw. 60, 1–7 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Kathryn Merrick
    • 1
  1. 1.School of Information Technology and Electrical EngineeringUniversity of New South Wales, Australian Defence Force AcademyCanberraAustralia

Personalised recommendations