Abstract
A method to optimize stepsize parameters in exponential moving average (EMA) based on Newton’s method to minimize square errors is proposed. The stepsize parameters used in reinforcement learning methods should be selected and adjusted carefully for dynamic and non-stationary environments. To find the suitable values for the stepsize parameters through learning, a framework to acquire higher-order derivatives of learning values by the stepsize parameters has been proposed. Based on this framework, the authors extend a method to determine the best stepsize using Newton’s method to minimize EMA of square error of learning. The method is confirmed by mathematical theories and by results of experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bonarini, A., Lazaric, A., de Cote, E.M., Restelli, M.: Improving cooperation among self-interested reinforcement learning agents. In: Proc. of Workshop on Reinforcement Learning in Non-Stationary Environments. ECML-PKDD 2005 (October 2005)
Bowling, M., Veloso, M.: Multiagent learning using a variable learning rate. Artificial Intelligence 136, 215–250 (2002)
Even-dar, E., Mansour, Y.: Learning rates for q-learning. Journal of Machine Learning Research 5, 2003 (December 2003)
George, A.P., Powell, W.B.: Adaptive stepsizes for recursive estimation with applications in approximate dynamic programming. Machine learning 65(1), 167–198 (2006)
Noda, I.: Adaptation of stepsize parameter for non-stationary environments by recursive exponential moving average. In: Prof. of ECML 2009 LNIID Workshop, ECML, pp. 24–31 (September 2009)
Noda, I.: Recursive Adaptation of Stepsize Parameter for Non-stationary Environments. In: Taylor, M.E., Tuyls, K. (eds.) ALA 2009. LNCS, vol. 5924, pp. 74–90. Springer, Heidelberg (2010)
Sato, M., Kimura, H., Kobayashi, S.: TD algorithm for the variance of return and mean-variance reinforcement learning (in japanese). Transactions of the Japanese Society for Artificial Intelligence 16(3F), 353–362 (2001)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Noda, I. (2011). Adaption of Stepsize Parameter Using Newton’s Method. In: Kinny, D., Hsu, J.Yj., Governatori, G., Ghose, A.K. (eds) Agents in Principle, Agents in Practice. PRIMA 2011. Lecture Notes in Computer Science(), vol 7047. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-25044-6_28
Download citation
DOI: https://doi.org/10.1007/978-3-642-25044-6_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-25043-9
Online ISBN: 978-3-642-25044-6
eBook Packages: Computer ScienceComputer Science (R0)