Abstract
Relational reinforcement learning is a Q-learning technique for relational state-action spaces. It aims to enable agents to learn how to act in an environment that has no natural representation as a tuple of constants. In this case, the learning algorithm used to approximate the mapping between state-action pairs and their so called Q(uality)-value has to be not only very reliable, but it also has to be able to handle the relational representation of state-action pairs.
In this paper we investigate the use of Gaussian processes to approximate the quality of state-action pairs. In order to employ Gaussian processes in a relational setting we use graph kernels as the covariance function between state-action pairs. Experiments conducted in the blocks world show that Gaussian processes with graph kernels can compete with, and often improve on, regression trees and instance based regression as a generalisation algorithm for relational reinforcement learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68 (1950)
Barnett, S.: Matrix Methods for Engineers and Scientists. McGraw-Hill, New York (1979)
Collins, M., Duffy, N.: Convolution kernels for natural language. In: Dietterich, T.G., Becker, S., Ghahramani, Z. (eds.) Advances in Neural Information Processing Systems, vol. 14, MIT Press, Cambridge (2002)
Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge (2000)
Diestel, R.: Graph Theory. Springer, Heidelberg (2000)
Driessens, K., Džeroski, S.: Integrating experimentation and guidance in relational reinforcement learning. In: Sammut, C., Hoffmann, A. (eds.) Proceedings of the Nineteenth International Conference on Machine Learning, pp. 115–122. Morgan Kaufmann Publishers, Inc., San Francisco (2002)
Driessens, K., Ramon, J.: Relational instance based regression for relational reinforcement learning. In: Proceedings of the 20th International Conference on Machine Learning (2003) (to be published)
Driessens, K., Ramon, J., Blockeel, H.: Speeding up relational reinforcement learning through the use of an incremental first order decision tree learner. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 97–108. Springer, Heidelberg (2001)
Džeroski, S., De Raedt, L., Blockeel, H.: Relational reinforcement learning. In: Proceedings of the 15th International Conference on Machine Learning, pp. 136–143. Morgan Kaufmann, San Francisco (1998)
Forbes, J., Andre, D.: Representations for learning control policies. In: de Jong, E., Oates, T. (eds.) Proceedings of the ICML 2002 Workshop on Development of Representations, pp. 7–14. The University of New South Wales, Sydney (2002)
Gärtner, T.: Exponential and geometric kernels for graphs. In: NIPS Workshop on Unreal Data: Principles of Modeling Nonvectorial Data (2002)
Gärtner, T.: Kernel-based multi-relational data mining. SIGKDD Explorations (2003)
Gärtner, T., Flach, P.A., Wrobel, S.: On graph kernels: Hardness results and efficient alternatives. In: Proceedings of the 16th Annual Conference on Computational Learning Theory and the 7th Kernel Workshop (2003)
Gärtner, T., Lloyd, J.W., Flach, P.A.: Kernels for structured data. In: Matwin, S., Sammut, C. (eds.) ILP 2002. LNCS (LNAI), vol. 2583, pp. 66–83. Springer, Heidelberg (2003)
Haussler, D.: Convolution kernels on discrete structures. Technical report, Department of Computer Science, University of California at Santa Cruz (1999)
Imrich, W., KlavĹľar, S.: Product Graphs: Structure and Recognition. John Wiley, Chichester (2000)
Kaelbling, L., Littman, M., Moore, A.: Reinforcement learning: A survey. Journal of Artificial Intelligence Research 4, 237–285 (1996)
Kashima, H., Inokuchi, A.: Kernels for graph classification. In: ICDM Workshop on Active Mining (2002)
Korte, B., Vygen, J.: Combinatorial Optimization: Theory and Algorithms. Springer, Heidelberg (2002)
Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., Watkins, C.: Text classification using string kernels. Journal of Machine Learning Research 2 (2002)
MacKay, D.J.C.: Introduction to Gaussian processes (1997), available at http://wol.ra.phy.cam.ac.uk/mackay
Ormoneit, D., Sen, S.: Kernel-based reinforcement learning. Machine Learning 49, 161–178 (2002)
Schaal, S., Atkeson, C.G., Vijayakumar, S.: Real-time robot learning with locally weighted statistical learning. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 288–293. IEEE Press, Piscataway (2000)
Schölkopf, B., Smola, A.J.: Learning with Kernels. MIT Press, Cambridge (2002)
Smart, W.D., Kaelbling, L.P.: Practical reinforcement learning in continuous spaces. In: Proceedings of the 17th International Conference on Machine Learning, pp. 903–910. Morgan Kaufmann, San Francisco (2000)
Sutton, R., Barto, A.: Reinforcement Learning: an introduction. The MIT Press, Cambridge (1998)
Watkins, C.: Learning from Delayed Rewards. PhD thesis, King’s College, Cambridge (1989)
Zien, A., Ratsch, G., Mika, S., Schölkopf, B., Lengauer, T., Muller, K.-R.: Engineering support vector machine kernels that recognize translation initiation sites. Bioinformatics 16(9), 799–807 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gärtner, T., Driessens, K., Ramon, J. (2003). Graph Kernels and Gaussian Processes for Relational Reinforcement Learning. In: Horváth, T., Yamamoto, A. (eds) Inductive Logic Programming. ILP 2003. Lecture Notes in Computer Science(), vol 2835. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39917-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-540-39917-9_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20144-1
Online ISBN: 978-3-540-39917-9
eBook Packages: Springer Book Archive