Abstract
We propose using a statistical-relational model, the Markov Logic Network, for knowledge transfer in reinforcement learning. Our goal is to extract relational knowledge from a source task and use it to speed up learning in a related target task. We show that Markov Logic Networks are effective models for capturing both source-task Q-functions and source-task policies. We apply them via demonstration, which involves using them for decision making in an initial stage of the target task before continuing to learn. Through experiments in the RoboCup simulated-soccer domain, we show that transfer via Markov Logic Networks can significantly improve early performance in complex tasks, and that transferring policies is more effective than transferring Q-functions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Croonenborghs, T., Driessens, K., Bruynooghe, M.: Learning relational skills for inductive transfer in relational reinforcement learning. In: Blockeel, H., Ramon, J., Shavlik, J., Tadepalli, P. (eds.) ILP 2007. LNCS (LNAI), vol. 4894, pp. 88–97. Springer, Heidelberg (2007)
Fernandez, F., Veloso, M.: Probabilistic policy reuse in a reinforcement learning agent. In: AAMAS (2006)
Kok, S., Singla, P., Richardson, M., Domingos, P.: The Alchemy system for statistical relational AI. Technical report, University of Washington (2005)
Lowd, D., Domingos, P.: Efficient Weight Learning for Markov Logic Networks. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) KDD 2007. LNCS (LNAI), vol. 4702, pp. 200–211. Springer, Heidelberg (2007)
Maclin, R., Shavlik, J., Torrey, L., Walker, T.: Knowledge-based support vector regression for reinforcement learning. In: IJCAI Workshop on Reasoning, Representation, and Learning in Computer Games (2005)
Madden, M., Howley, T.: Transfer of experience between reinforcement learning environments with progressive difficulty. AI Review 21, 375–398 (2004)
Noda, I., Matsubara, H., Hiraki, K., Frank, I.: Soccer server: A tool for research on multiagent systems. Applied Artificial Intelligence 12, 233–250 (1998)
De Raedt, L.: Logical and Relational Learning. Springer, Heidelberg (2008)
Richardson, M., Domingos, P.: Markov logic networks. Machine Learning 62, 107–136 (2006)
Srinivasan, A.: The Aleph manual (2001)
Stone, P., Sutton, R.: Scaling reinforcement learning toward RoboCup soccer. In: ICML (2001)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Torrey, L., Shavlik, J.: Transfer learning. In: Soria, E., Martin, J., Magdalena, R., Martinez, M., Serrano, A. (eds.) Handbook of Research on Machine Learning Applications. IGI Global (2009)
Torrey, L., Shavlik, J., Natarajan, S., Kuppili, P., Walker, T.: Transfer in reinforcement learning via Markov Logic Networks. In: AAAI Workshop on Transfer Learning for Complex Tasks (2008)
Torrey, L., Shavlik, J., Walker, T., Maclin, R.: Relational macros for transfer in reinforcement learning. In: ICML (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Torrey, L., Shavlik, J. (2010). Policy Transfer via Markov Logic Networks. In: De Raedt, L. (eds) Inductive Logic Programming. ILP 2009. Lecture Notes in Computer Science(), vol 5989. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13840-9_23
Download citation
DOI: https://doi.org/10.1007/978-3-642-13840-9_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-13839-3
Online ISBN: 978-3-642-13840-9
eBook Packages: Computer ScienceComputer Science (R0)