Motivated Reinforcement Learning for Improved Head Actuation of Humanoid Robots

  • Jake Fountain
  • Josiah Walker
  • David Budden
  • Alexandre Mendes
  • Stephan K. Chalup
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8371)


The ability of an autonomous agent to self-localise within its environment is critically dependent on its ability to make accurate observations of static, salient features. This notion has driven considerable research into the development and improvement of feature extraction and object recognition algorithms, both within RoboCup and the robotics community at large. Instead, this paper focuses on the rarely-considered issue imposed by the limited field of view of humanoid robots; namely, determining an optimal policy for actuating a robot’s head, to ensure it observes regions of the environment that will maximise the positional information provided. The complexity of this task is magnified by a number of common computational issues; specifically high dimensional state spaces and noisy environmental observations. This paper details the application of motivated reinforcement learning to partially overcome these issues, leading to an 11% improvement (relative to the null case of uniformly distributed actuation policies) in self-localisation and ball-localisation for an agent trained online for less than one hour. The method is demonstrated as a viable method for improving self-localisation in robotics, without the need for further optimisation of object recognition or tuning of probabilistic filters.


motivated reinforcement learning localisation Fourier basis head actuation simulated curiosity 


  1. 1.
    Wong, A.S.W., Chalup, S.K., Bhatia, S., Jalalian, A., Kulk, J., Nicklin, S., Ostwald, M.J.: Visual gaze analysis of robotic pedestrians moving in urban space. Architectural Science Review 55(3), 213–223 (2012)CrossRefGoogle Scholar
  2. 2.
    Merrick, E.K., Maher, M.L.: Motivated Reinforcement Learning: Curious Characters for Multuser Games. Springer, Dordrecht (2009)CrossRefGoogle Scholar
  3. 3.
    Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Matsubara, H.: Robocup: A challenge problem for ai. AI Magazine 18(1) (1991)Google Scholar
  4. 4.
    Budden, D., Fenn, S., Walker, J., Mendes, A.: A novel approach to ball detection for humanoid robot soccer. In: Thielscher, M., Zhang, D. (eds.) AI 2012. LNCS, vol. 7691, pp. 827–838. Springer, Heidelberg (2012)Google Scholar
  5. 5.
    Wan, E., van der Merwe, R.: The unscented kalman filter for nonlinear estimation. In: Adaptive Systems for Signal Processing, Communications, and Control Symposium, AS-SPCC 2000, pp. 153–158. The IEEE (2000)Google Scholar
  6. 6.
    Watkins, C.: Learning from Delayed Rewards. PhD thesis, Cambridge University (1989)Google Scholar
  7. 7.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  8. 8.
    Wundt, W.: Principles of Physiology and Psychology. Macmillan, New York (1910)Google Scholar
  9. 9.
    Saunders, R., Gero, J.S.: Designing for interest and novelty - motivating design agents. In: de Vries, B., van Leeuwen, J., Achten, H. (eds.) Proceedings of the Ninth International Conference on Computer Aided Architectural Design Futures, pp. 725–738. Kluwer Academic Publishers (2001)Google Scholar
  10. 10.
    Merrick, K.E., Isaacs, A., Barlow, M., Gu, N.: A shape grammar approach to computational creativity and procedural content generation in massively multiplayer online role playing games. Entertainment Computing 4(2), 115–130 (2013)CrossRefGoogle Scholar
  11. 11.
    Merrick, K.: Intrinsic motivation and introspection in reinforcement learning. IEEE Transactions on Autonomous Mental Development 4(4), 315–329 (2012)CrossRefGoogle Scholar
  12. 12.
    Konidaris, G., Osentoski, S., Thomas, P.S.: Value function approximation in reinforcement learning using the Fourier basis. In: Burgard, W., Roth, D. (eds.) Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2011, pp. 380–385. AAAI Press, San Francisco (2011)Google Scholar
  13. 13.
    The RoboCup Institution: RoboCup Soccer Humanoid League Rules and Setup for the 2013 Competition in Eindhoven, DRAFT (2012),
  14. 14.
    Majdik, A., Popa, M., Tamas, L., Szoke, I., Lazea, G.: New approach in solving the kidnapped robot problem. In: Robotics (ISR), 2010 41st International Symposium on and 2010 6th German Conference on Robotics (ROBOTIK), pp. 1–6 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Jake Fountain
    • 1
  • Josiah Walker
    • 1
  • David Budden
    • 1
  • Alexandre Mendes
    • 1
  • Stephan K. Chalup
    • 1
  1. 1.The University of NewcastleAustralia

Personalised recommendations