Decentralized Reinforcement Learning Applied to Mobile Robots

  • David L. LeottauEmail author
  • Aashish Vatsyayan
  • Javier Ruiz-del-Solar
  • Robert Babuška
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9776)


In this paper, decentralized reinforcement learning is applied to a control problem with a multidimensional action space. We propose a decentralized reinforcement learning architecture for a mobile robot, where the individual components of the commanded velocity vector are learned in parallel by separate agents. We empirically demonstrate that the decentralized architecture outperforms its centralized counterpart in terms of the learning time, while using less computational resources. The method is validated on two problems: an extended version of the 3-dimensional mountain car, and a ball-pushing behavior performed with a differential-drive robot, which is also tested on a physical setup.


Multiagent learning Decentralized control Reinforcement learning Robot soccer 



This work was partially funded by FONDECYT under Project Number 1161500. David Leonardo Leottau was funded under grant CONICYT-PCHA/Doctorado Nacional/2013-63130183. The authors would like to thank Technical University of Delft for providing the resources to test the learnt policies on an experimental setup.


  1. 1.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  2. 2.
    Riedmiller, M., Gabel, T., Hafner, R., Lange, S.: Reinforcement learning for robot soccer. Auton. Robots 27(1), 55–73 (2009)CrossRefGoogle Scholar
  3. 3.
    Martin, J., Lope, H.D.: A distributed reinforcement learning architecture for multi-link robots. In: 4th International Conference on Informatics in Control, Automation and Robotics, ICINCO 2007. Number 3, Angers, Francia, pp. 192–197 (2007)Google Scholar
  4. 4.
    Busoniu, L., Babuska, R., De-Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 38(2), 156–172 (2008)CrossRefGoogle Scholar
  5. 5.
    Busoniu, L., Schutter, B.D., Babuska, R.: Decentralized reinforcement learning control of a robotic manipulator. In: Ninth International Conference on Control, Automation, Robotics and Vision, ICARCV 2006, 5–8 December, Singapore, pp. 1–6. IEEE (2006)Google Scholar
  6. 6.
    Stone, P., Veloso, M.: Multiagent systems: a survey from a machine learning perspective. Auton. Robot. 8(3), 1–57 (2000)CrossRefGoogle Scholar
  7. 7.
    Taylor, M.E., Kuhlmann, G., Stone, P.: Autonomous transfer for reinforcement learning. In: The Autonomous Agents and Multi-agent Systems Conference (AAMAS), Number May, Estoril, Portugal, pp. 283–290 (2008)Google Scholar
  8. 8.
    Systems, M.: Miabotpro manual (2016)Google Scholar
  9. 9.
    Leottau, D.L., Ruiz-del-Solar, J., MacAlpine, P., Stone, P.: A study of layered learning strategies applied to individual behaviors in robot soccer. In: Almeida, L., Ji, J., Steinbauer, G., Luke, S. (eds.) RoboCup 2015. LNCS (LNAI), vol. 9513, pp. 290–302. Springer, Cham (2015). Scholar
  10. 10.
    Takahashi, Y., Asada, M.: Multi-layered learning system for real robot behavior acquisition. In: Kordic, V., (ed.) Cutting Edge Robotics, Number pp. 357–375, July 2005 (2004)Google Scholar
  11. 11.
    Emery, R., Balch, T.: Behavior-based control of a non-holonomic robot in pushing tasks. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164), vol. 3, pp. 2381–2388. IEEE (2001)Google Scholar
  12. 12.
    Taylor, M., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Vatsyayan, A.: Video: centralized and decentralized reinforcement learning of the ball-pushing behavior (2016).
  14. 14.
    Dziomin, U., Kabysh, A., Golovko, V., Stetter, R.: A multi-agent reinforcement learning approach for the efficient control of mobile robot. In: 2013 IEEE 7th International Conference on Intelligent Data Acquisition and Advanced Computing Systems (IDAACS), vol. 2, pp. 867–873. IEEE (2013)Google Scholar
  15. 15.
    Leottau, D.L., Ruiz-del-Solar, J.: An accelerated approach to decentralized reinforcement learning of the Ball-Dribbling behavior. In: AAAI Workshops, Austin, Texas USA, pp. 23–29 (2015)Google Scholar
  16. 16.
    Kabysh, A., Golovko, V., Lipnickas, A.: Influence learning for multi-agent system based on reinforcement learning. Int. J. Comput. 11(1), 39–44 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • David L. Leottau
    • 1
    Email author
  • Aashish Vatsyayan
    • 2
  • Javier Ruiz-del-Solar
    • 1
  • Robert Babuška
    • 2
  1. 1.Advanced Mining Technology Center, Department of Electrical EngineeringUniversidad de ChileSantiagoChile
  2. 2.Delft Center for Systems and ControlDelft University of TechnologyDelftThe Netherlands

Personalised recommendations