Abstract
We report on an investigation of reinforcement learning techniques for the learning of coordination in cooperative multi-agent systems. Specifically, we focus on two novel approaches: one is based on a new action selection strategy for Q-learning [10], and the other is based on model estimation with a shared action-selection protocol. The new techniques are applicable to scenarios where mutual observation of actions is not possible.
To date, reinforcement learning approaches for such independent agents did not guarantee convergence to the optimal joint action in scenarios with high miscoordination costs. We improve on previous results [2] by demonstrating empirically that our extension causes the agents to converge almost always to the optimal joint action even in these difficult cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
C. Boutilier. Sequential optimality and coordination in multiagent systems. In Proceedings of the Sixteenth International Joint Conference on Articial Intelligence (IJCAI-99), pages 478–485, 1999.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the Fifteenth National Conference on Articial Intelligence, pages 746–752, 1998.
Drew Fudenberg and David K. Levine. The Theory of Learning in Games. MIT Press, Cambridge, MA, 1998.
Leslie Pack Kaelbling, Michael Littman, and Andrew W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 1996.
Martin Lauer and Martin Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In Proceedings of the Seventeenth International Conference in Machine Learning, 2000.
Sandip Sen and Mahendra Sekaran. Individual learning of coordination knowledge. JETAI, 10(3): 333–356, 1998.
Sandip Sen, Mahendra Sekaran, and John Hale. Learning to coordinate without sharing information. In Proceedings of the Twelfth National Conference on Artificial Intelligence, pages 426–431, Seattle, WA, 1994.
S. Singh, T. Jaakkola, M. L. Littman, and C Szpesvari. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning Journal, 38(3):287–308, 2000.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine Learning, pages 330–337, 1993.
C. J. C. H. Watkins. Learning from Delayed Rewards. PhD thesis, Cambridge University, Cambridge, England, 1989.
Gerhard Weiss. Learning to coordinate actions in multi-agent systems. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, volume 1, pages 311–316. Morgan Kaufmann Publ., 1993.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kapetanakis, S., Kudenko, D., Strens, M.J.A. (2003). Reinforcement Learning Approaches to Coordination in Cooperative Multi-agent Systems. In: Alonso, E., Kudenko, D., Kazakov, D. (eds) Adaptive Agents and Multi-Agent Systems. AAMAS AAMAS 2002 2001. Lecture Notes in Computer Science(), vol 2636. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-44826-8_2
Download citation
DOI: https://doi.org/10.1007/3-540-44826-8_2
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40068-4
Online ISBN: 978-3-540-44826-6
eBook Packages: Springer Book Archive