Abstract
Iterated negotiations are a well-established method for coordinating distributed activities in multiagent systems. However, if several of these take place concurrently, the participants’ activities can mutually influence each other. In order to cope with the problem of interrelated interaction outcomes in partially observable environments, we apply distributed reinforcement learning to concurrent many-object negotiations. To this end, we discuss iterated negotiations from the perspective of repeated games, specify the agents’ learning behavior, and introduce decentral decision-making criteria for terminating a negotiation. Furthermore, we empirically evaluate the approach in a multiagent resource allocation scenario. The results show that our method enables the agents to successfully learn mutual best response behaviors which approximate Nash equilibrium allocations. Additionally, the learning constrains the required interaction effort for attaining these results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Berndt, J.O., Herzog, O.: Distributed Reinforcement Learning for Optimizing Resource Allocation in Autonomous Logistics Processes. In: Kreowski, H.-J., Scholz-Reiter, B., Thoben, K.-D. (eds.) LDIC 2012, Bremen (2012)
Buşoniu, L., Babuška, R., De Schutter, B.: Multi-agent Reinforcement Learning: An Overview. In: Srinivasan, D., Jain, L.C. (eds.) Innovations in Multi-Agent Systems and Applications - 1. SCI, vol. 310, pp. 183–221. Springer, Heidelberg (2010)
Claus, C., Boutilier, C.: The Dynamics of Reinforcement Learning in Cooperative Multiagent Systems. In: AAAI 1998, Madison, pp. 746–752 (1998)
Cramton, P., Shoham, Y., Steinberg, R. (eds.): Combinatorial Auctions. The MIT Press, Cambridge (2006)
Faratin, P., Sierra, C., Jennings, N.R.: Negotiation decision functions for autonomous agents. Robot. Auton. Syst. 24(3-4), 159–182 (1998)
Foundation for Intelligent Physical Agents: FIPA Iterated Contract Net Interaction Protocol Specification, Standard (2002); document No. SC00030H
Gjerstad, S., Dickhaut, J.: Price Formation in Double Auctions. Game. Econ. Behav. 22(1), 1–29 (1998)
Jennings, N.R., Faratin, P., Lomuscio, A.R., Parsons, S., Wooldridge, M.J., Sierra, C.: Automated Negotiation: Prospects, Methods and Challenges. Group Decis. Negot. 10, 199–215 (2001)
Kaisers, M., Tuyls, K.: Frequency Adjusted Multiagent Q-learning. In: van der Hoek, W., Kaminka, G.A., Lespérance, Y., Luck, M., Sen, S. (eds.) AAMAS 2010, pp. 309–315. IFAAMAS, Toronto (2010)
Luckhart, C., Irani, K.B.: An Algorithmic Solution of N-Person Games. In: AAAI 1986, vol. 1, pp. 158–162. Morgan Kaufmann, Philadelphia (1986)
Nash, J.: Non-cooperative Games. Ann. Math. 54(2), 286–295 (1950)
Porter, R., Nudelman, E., Shoham, Y.: Simple search methods for finding a Nash equilibrium. Game. Econ. Behav. 63(2), 642–662 (2008)
Ramezani, S., Endriss, U.: Nash Social Welfare in Multiagent Resource Allocation. In: David, E., Gerding, E., Sarne, D., Shehory, O. (eds.) Agent-Mediated Electronic Commerce, pp. 117–131. Springer, Heidelberg (2010)
Richter, J., Klusch, M., Kowalczyk, R.: Monotonic Mixing of Decision Strategies for Agent-Based Bargaining. In: Klügl, F., Ossowski, S. (eds.) MATES 2011. LNCS, vol. 6973, pp. 113–124. Springer, Heidelberg (2011)
Schuldt, A., Berndt, J.O., Herzog, O.: The Interaction Effort in Autonomous Logistics Processes: Potential and Limitations for Cooperation. In: Hülsmann, M., Scholz-Reiter, B., Windt, K. (eds.) Autonomous Cooperation and Control in Logistics, pp. 77–90. Springer, Berlin (2011)
Schuldt, A., Gehrke, J.D., Werner, S.: Designing a Simulation Middleware for FIPA Multiagent Systems. In: Jain, L., Gini, M., Faltings, B.B., Terano, T., Zhang, C., Cercone, N., Cao, L. (eds.) WI-IAT 2008, pp. 109–113. IEEE Computer Society Press, Sydney (2008)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)
v. Neumann, J.: Zur Theorie der Gesellschaftsspiele. Math. Ann. 100, 295–320 (1928)
v. Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton University Press, Princeton (1944)
Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3-4), 279–292 (1992)
Winoto, P., McCalla, G.I., Vassileva, J.: Non-Monotonic-Offers Bargaining Protocol. Auton. Agent. Multi-Ag. 11, 45–67 (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Berndt, J.O., Herzog, O. (2012). Distributed Learning of Best Response Behaviors in Concurrent Iterated Many-Object Negotiations. In: Timm, I.J., Guttmann, C. (eds) Multiagent System Technologies. MATES 2012. Lecture Notes in Computer Science(), vol 7598. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33690-4_4
Download citation
DOI: https://doi.org/10.1007/978-3-642-33690-4_4
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33689-8
Online ISBN: 978-3-642-33690-4
eBook Packages: Computer ScienceComputer Science (R0)