Measuring Collaborative Emergent Behavior in Multi-agent Reinforcement Learning

  • Sean L. BartonEmail author
  • Nicholas R. Waytowich
  • Erin Zaroukian
  • Derrik E. Asher
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 876)


Multi-agent reinforcement learning (RL) has important implications for the future of human-agent teaming. We show that improved performance with multi-agent RL is not a guarantee of the collaborative behavior thought to be important for solving multi-agent tasks. To address this, we present a novel approach for quantitatively assessing collaboration in continuous spatial tasks with multi-agent RL. Such a metric is useful for measuring collaboration between computational agents and may serve as a training signal for collaboration in future RL paradigms involving humans.


Multi-agent reinforcement learning Deep reinforcement learning Human-agent teaming Collaboration 


Acknowledgements and Disclosure

This research was sponsored by the Army Research Laboratory and was accomplished under Cooperative Agreement Number W911NF-18-2-0058. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.


  1. 1.
    Matignon, L., Laurent, G.J., Le Fort-Piat, N.: Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. Knowl. Eng. Rev. 27, 1–31 (2012)CrossRefGoogle Scholar
  2. 2.
    Sen, S., Sekaran, M., Hale, J., et al.: Learning to coordinate without sharing information. In: AAAI, pp. 426–431 (1994)Google Scholar
  3. 3.
    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)CrossRefGoogle Scholar
  4. 4.
    Lowe, R., Wu, Y., Tamar, A., Harb, J., Pieter Abbeel, O., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 6382–6393. Curran Associates, Inc. (2017)Google Scholar
  5. 5.
    Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., Whiteson, S.: Counterfactual Multi-Agent Policy Gradients. arXiv:1705.08926 [cs] (2017)
  6. 6.
    Matignon, L., Laurent, G., Le Fort-Piat, N.: Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2007, pp. 64–69 (2007)Google Scholar
  7. 7.
    Lauer, M., Riedmiller, M.: An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In: Proceedings of the Seventeenth International Conference on Machine Learning. Citeseer (2000)Google Scholar
  8. 8.
    Claus, C., Boutilier, C.: The dynamics of reinforcement learning in cooperative multiagent systems. In: AAAI/IAAI 1998, pp. 746–752 (1998)Google Scholar
  9. 9.
    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym. arXiv:1606.01540 [cs] (2016)
  10. 10.
    Sugihara, G., May, R., Ye, H., Hsieh, C.-H., Deyle, E., Fogarty, M., Munch, S.: Detecting causality in complex ecosystems. Science 1227079 (2012)Google Scholar
  11. 11.
    Parasuraman, R., Sheriden, T.B., Wickens, C.D.: A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 30, 286–297 (2000)CrossRefGoogle Scholar
  12. 12.
    Rovira, E., McGarry, K., Parasuraman, R.: Effects of imperfect automation on decision making in a simulated command and control task. Hum. Factors 49, 76–87 (2007)CrossRefGoogle Scholar
  13. 13.
    Klein, G., Woods, D.D., Bradshaw, J.M., Hoffman, R.R., Feltovich, P.J.: Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intell. Syst. 19, 91–95 (2004)CrossRefGoogle Scholar

Copyright information

© This is a U.S. government work and not under copyright protection in the U.S.; foreign copyright protection may apply 2019

Authors and Affiliations

  • Sean L. Barton
    • 1
    Email author
  • Nicholas R. Waytowich
    • 2
  • Erin Zaroukian
    • 1
  • Derrik E. Asher
    • 1
  1. 1.Computational and Information Sciences DirectorateU.S. Army Research LaboratoryAdelphiUSA
  2. 2.Human Research and Engineering DirectorateU.S. Army Research LaboratoryAdelphiUSA

Personalised recommendations