Advertisement

Efficient Uncertainty Propagation for Reinforcement Learning with Limited Data

  • Alexander Hans
  • Steffen Udluft
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5768)

Abstract

In a typical reinforcement learning (RL) setting details of the environment are not given explicitly but have to be estimated from observations. Most RL approaches only optimize the expected value. However, if the number of observations is limited considering expected values only can lead to false conclusions. Instead, it is crucial to also account for the estimator’s uncertainties. In this paper, we present a method to incorporate those uncertainties and propagate them to the conclusions. By being only approximate, the method is computationally feasible. Furthermore, we describe a Bayesian approach to design the estimators. Our experiments show that the method considerably increases the robustness of the derived policies compared to the standard approach.

Keywords

Reinforcement learning model-based uncertainty Bayesian modeling 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Calafiore, G., El Ghaoui, L.: On distributionally robust chance-constrained linear programs. In: Optimization Theory and Applications (2006)Google Scholar
  2. 2.
    D’Agostini, G.: Bayesian Reasoning in Data Analysis: A Critical Introduction. World Scientific Publishing, Singapore (2003)CrossRefzbMATHGoogle Scholar
  3. 3.
    Delage, E., Mannor, S.: Percentile optimization in uncertain Markov decision processes with application to efficient exploration. In: Proc. of the Int. Conf. on Machine Learning (2007)Google Scholar
  4. 4.
    Engel, Y., Mannor, S., Meir, R.: Bayes meets Bellman: the Gaussian process approach to temporal difference learning. In: Proc. of the Int. Conf. on Machine Learning (2003)Google Scholar
  5. 5.
    Engel, Y., Mannor, S., Meir, R.: Reinforcement learning with Gaussian processes. In: Proc. of the Int. Conf. on Machine Learning (2005)Google Scholar
  6. 6.
    Ghavamzadeh, M., Engel, Y.: Bayesian policy gradient algorithms. In: Advances in Neural Information Processing Systems (2006)Google Scholar
  7. 7.
    Ghavamzadeh, M., Engel, Y.: Bayesian actor-critic algorithms. In: Proc. of the Int. Conf. on Machine Learning (2007)Google Scholar
  8. 8.
    Nilim, A., El Ghaoui, L.: Robustness in Markov decision problems with uncertain transition matrices. In: Advances in Neural Information Processing Systems (2003)Google Scholar
  9. 9.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons Canada, Ltd., Chichester (1994)CrossRefzbMATHGoogle Scholar
  10. 10.
    Schneegass, D., Udluft, S., Martinetz, T.: Uncertainty propagation for quality assurance in reinforcement learning. In: Proc. of the Int. Joint Conf. on Neural Networks (2008)Google Scholar
  11. 11.
    Strehl, A.L., Littman, M.L.: An empirical evaluation of interval estimation for markov decision processes. In: 16th IEEE Int. Conf. on Tools with Artificial Intelligence, pp. 128–135 (2004)Google Scholar
  12. 12.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  13. 13.
    Tresp, V.: The wet game of chicken. Siemens AG, CT IC 4, Technical Report (1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Alexander Hans
    • 1
    • 2
  • Steffen Udluft
    • 1
  1. 1.Siemens AG, Corporate Technology, Information & Communications, Learning SystemsMunichGermany
  2. 2.Neuroinformatics and Cognitive Robotics LabIlmenau Technical UniversityIlmenauGermany

Personalised recommendations