Abstract
Shaping functions can be used in multi-task reinforcement learning (RL) to incorporate knowledge from previously experienced source tasks to speed up learning on a new target task. Earlier work has not clearly motivated choices for the shaping function. This paper discusses and empirically compares several alternatives, and demonstrates that the most intuive one may not always be the best option. In addition, we extend previous work on identifying good representations for the value and shaping functions, and show that selecting the right representation results in improved generalization over tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Asmuth, J., Littman, M.L., Zinkov, R.: Potential-based shaping in model-based reinforcement learning. In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pp. 604–609. The AAAI Press (2008)
Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)
Elfwing, S., Uchibe, E., Doya, K., Christensen, H.I.: Co-evolution of shaping: Rewards and meta-parameters in reinforcement learning. Adaptive Behavior 16(6), 400–412 (2008)
Hachiya, H., Sugiyama, M.: Feature Selection for Reinforcement Learning: Evaluating Implicit State-Reward Dependency via Conditional Mutual Information. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS, vol. 6321, pp. 474–489. Springer, Heidelberg (2010)
Jong, N.K., Stone, P.: State abstraction discovery from irrelevant state variables. In: IJCAI 2005 (2005)
Konidaris, G.D., Barto, A.G.: Autonomous shaping: Knowledge transfer in reinforcement learning. In: Proc. 23rd International Conference on Machine Learning, pp. 489–496 (2006)
Lazaric, A., Ghavamzadeh, M.: Bayesian multi-task reinforcement learning. In: ICML, pp. 599–606 (2010)
Li, L., Walsh, T.J., Littman, M.L.: Towards a unified theory of state abstraction for MDPs. In: Aritificial Intelligence and Mathematics (2006)
Mahadevan, S.: Representation discovery in sequential decision making. In: AAAI (2010)
Ng, A.Y., Harada, D., Russell, S.: Policy invariance under reward transformations: Theory and application to reward shaping. In: Proc. 16th International Conference on Machine Learning (1999)
Parr, R., Li, L., Taylor, G., Painter-Wakefield, C., Littman, M.L.: An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In: ICML, pp. 752–759 (2008)
Petrik, M., Taylor, G., Parr, R., Zilberstein, S.: Feature selection using regularization in approximate linear programs for Markov decision processes. In: ICML, pp. 871–878 (2010)
Singh, S., Lewis, R.L., Barto, A.G.: Where do rewards come from? In: Proc. 31st Annual Conference of the Cognitive Science Society, pp. 2601–2606 (2009)
Snel, M., Whiteson, S.: Multi-task evolutionary shaping without pre-specified representations. In: Genetic and Evolutionary Computation Conference, GECCO 2010 (2010)
Sorg, J., Singh, S.: Transfer via soft homomorphisms. In: Proc. 8th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2009), pp. 741–748 (2009)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press (1998)
Tanaka, F., Yamamura, M.: Multitask reinforcement learning on the distribution of MDPs. In: Proc. 2003 IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA 2003), pp. 1108–1113 (2003)
Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research 10(1), 1633–1685 (2009)
Walsh, T.J., Li, L., Littman, M.L.: Transferring state abstractions between MDPs. In: ICML 2006 Workshop on Structural Knowledge Transfer for Machine Learning (2006)
Wiewiora, E., Cottrell, G., Elkan, C.: Principled methods for advising reinforcement learning agents. In: Proc. 20th International Conference on Machine Learning, pp. 792–799 (2003)
Wilson, A., Fern, A., Ray, S., Tadepalli, P.: Multi-task reinforcement learning: a hierarchical Bayesian approach. In: ICML, pp. 1015–1022 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Snel, M., Whiteson, S. (2012). Multi-Task Reinforcement Learning: Shaping and Feature Selection. In: Sanner, S., Hutter, M. (eds) Recent Advances in Reinforcement Learning. EWRL 2011. Lecture Notes in Computer Science(), vol 7188. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29946-9_24
Download citation
DOI: https://doi.org/10.1007/978-3-642-29946-9_24
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29945-2
Online ISBN: 978-3-642-29946-9
eBook Packages: Computer ScienceComputer Science (R0)