Abstract
We compare the coordination structures of agents using different types of inputs for their deep Q-networks (DQNs) by having agents play a distributed task execution game. The efficiency and performance of many multi-agent systems can be significantly affected by the coordination structures formed by agents. One important factor that may affect these structures is the information provided to an agent’s DQN. In this study, we analyze the differences in coordination structures in an environment involving walls to obstruct visibility and movement. Additionally, we introduce a new DQN input, which performs better than past inputs in a dynamic setting. Experimental results show that agents with their absolute locations in their DQN input indicate a granular level of labor division in some settings, and that the consistency of the starting locations of agents significantly affects the coordination structures and performances of agents.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Foerster, J.N., Assael, Y.M., de Freitas, N., Whiteson, S.: Learning to communicate to solve riddles with deep distributed recurrent Q-networks. CoRR, vol. abs/1602.02672 (2016)
Gupta, J.K., Egorov, M., Kochenderfer, M.: Cooperative multi-agent control using deep reinforcement learning. In: Sukthankar, G., Rodriguez-Aguilar, J.A. (eds.) AAMAS 2017. LNCS (LNAI), vol. 10642, pp. 66–83. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71682-4_5
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. CoRR, vol. abs/1706.02275 (2017)
Miyashita, Y., Sugawara, T.: Cooperation and coordination regimes by deep Q-learning in multi-agent task executions. In: Tetko, I.V., Kurková, V., Karpov, P., Theis, F. (eds.) ICANN 2019. LNCS, vol. 11727, pp. 541–554. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30487-4_42
Leibo, J.Z., Zambaldi, V., Lanctot, M., Marecki, J., Graepel, T.: Multi-agent reinforcement learning in sequential social dilemmas. In: International Foundation for Autonomous Agents and Multiagent Systems, ser. AAMAS 2017, Richland, SC, pp. 464–473 (2017)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. CoRR, vol. abs/1509.02971 (2015)
van Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. CoRR, vol. abs/1509.06461 (2015)
Lample, G., Chaplot, D.S.: Playing FPS games with deep reinforcement learning. CoRR, vol. abs/1609.05521 (2016)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Mnih, V., et al.: Playing Atari with deep reinforcement learning. CoRR, vol. abs/1312.5602 (2013)
Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., De Freitas, N.: Dueling network architectures for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ser. ICML 2016. JMLR.org, pp. 1995–2003 (2016)
Hausknecht, M., Stone, P.: Deep recurrent Q-learning for partially observable MDPs. CoRR, vol. abs/1507.06527 (2015)
Hessel, M., et al.: Rainbow: combining improvements in deep reinforcement learning. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI 2018), New Orleans, Louisiana, USA, 2–7 February 2018, pp. 3215–3222. AAAI Press (2018)
Diallo, E.A.O., Sugawara, T.: Coordination in adversarial multi-agent with deep reinforcement learning under partial observability. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI) (2019)
Chollet, F., et al.: Keras (2015). https://github.com/fchollet/keras
Tieleman, T., Hinton, G.: Neural Networks for Machine Learning - Lecture 6a - Overview of mini-batch gradient descent (2012)
Acknowlegement
This work is partly supported by JSPS KAKENHI Grant number 17KT0044.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Smith, K., Miyashita, Y., Sugawara, T. (2021). Analysis of Coordination Structures of Partially Observing Cooperative Agents by Multi-agent Deep Q-Learning. In: Uchiya, T., Bai, Q., Marsá Maestre, I. (eds) PRIMA 2020: Principles and Practice of Multi-Agent Systems. PRIMA 2020. Lecture Notes in Computer Science(), vol 12568. Springer, Cham. https://doi.org/10.1007/978-3-030-69322-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-69322-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69321-3
Online ISBN: 978-3-030-69322-0
eBook Packages: Computer ScienceComputer Science (R0)