Abstract
Humans appraise the environment in daily life. We are implementing appraisal mechanisms into reinforcement learning agents. One of such mechanisms we proposed is the utility-based Q-learning, which learns behaviors from subjective utilities derived from payoffs the agent gains and a utility-derivation function the agent has. In the previous work, we know that payoff-based evolution brings utility-derivation functions that facilitate mutual cooperation in iterated prisoner’s dilemma games. However, the evolution process itself has not yet been known well. In this work, we investigate the process in terms of what determines the evolution direction. We introduce two metrics showing preference of actions based on the evolved subjective utilities, which divide the evolution space into four regions. In each region, the metrics will explain the evolution directions.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Here we ignore the border areas.
References
Rilling, J.K., Gutman, D.A., Zeh, T.R., Pagnoni, G., Berns, G.S., Kilts, C.D.: A neural basis for social cooperation. Neuron 35, 395–405 (2002)
Sandholm, T.W., Crites, R.H.: Multiagent reinforcement learning in the Iterated Prisoner’s Dilemma. BioSystems 37, 147–166 (1996)
Moriyama, K.: Utility based Q-learning to facilitate cooperation in Prisoner’s Dilemma games. Web Intell. Agent Syst. 7(3), 233–242 (2009)
Moriyama, K., Kurihara, S., Numao, M.: Evolving subjective utilities: Prisoner’s Dilemma game examples. In: Proceedings of the 10th International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 233–240 (2011)
Axelrod, R.: The Evolution of Cooperation. Basic Books, New York (1984)
Watkins, C.J.C.H., Dayan, P.: Technical note: Q-learning. Mach. Learn. 8, 279–292 (1992)
Goldberg, D.E.: Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, Reading (1989)
Axelrod, R.: The evolution of strategies in the Iterated Prisoner’s Dilemma. In: Davis, L. (ed.) Genetic Algorithms and Simulated Annealing, pp. 32–41. Pitman/Morgan Kaufmann, London/Los Altos (1987)
Nowak, M.A.: Five rules for the evolution of cooperation. Science 314, 1560–1563 (2006)
Singh, S., Lewis, R.L., Barto, A.G., Sorg, J.: Intrinsically motivated reinforcement learning: an evolutionary perspective. IEEE Trans. Auton. Ment. Dev. 2, 70–82 (2010)
Vassiliades, V., Christodoulou, C.: Multiagent reinforcement learning in the Iterated Prisoner’s Dilemma: fast cooperation through evolved payoffs. In: Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN) (2010)
Acknowledgments
This work was partly supported by JSPS KAKENHI Grant Number JP16K00302, Kayamori Foundation of Informational Science Advancement, and the Hori Sciences & Arts Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Miyawaki, M., Moriyama, K., Mutoh, A., Matsui, T., Inuzuka, N. (2019). Evolution Direction of Reward Appraisal in Reinforcement Learning Agents. In: Jezic, G., Chen-Burger, YH., Howlett, R., Jain, L., Vlacic, L., Šperka, R. (eds) Agents and Multi-Agent Systems: Technologies and Applications 2018. KES-AMSTA-18 2018. Smart Innovation, Systems and Technologies, vol 96. Springer, Cham. https://doi.org/10.1007/978-3-319-92031-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-92031-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92030-6
Online ISBN: 978-3-319-92031-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)