Abstract
Predictive state representations (PSRs) have been proposed recently as an alternative representation for environments with partial observability. The representation is rooted in actions and observations, so it holds the promise of being easier to learn than Partially Observable Markov Decision Processes (POMDPs). However, comparatively little work has explored planning algorithms using PSRs. Exact methods developed to date are no faster than existing exact planning approaches for POMDPs, and only memory-based PSRs have been shown so far to have an advantage in terms of planning speed. In this paper, we present an algorithm for approximate planning in PSRs, based on an approach similar to point-based value iteration in POMDPs. The point-based approach turns out to be a natural match for the PSR state representation. We present empirical results showing that our approach is either comparable or better than POMDP point-based planning.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bonet, B.: An epsilon-optimal grid-based algorithm for partially observable Markov decision processes. In: Proceedings of ICML, pp. 51–58 (2002)
Cassandra, A.R., Littman, M.L., Kaelbling, L.P.: A simple, fast, exact methods for partially observable Markov decision processes. In: Proceedings of UAI, pp. 54–61 (1997)
Even-Dar, E., Kakade, S.M., Mansour, Y.: Planning in POMDPS using Multiplicity Automata. In: Proceedings of UAI (2005)
Izadi, M.T., Precup, D.: A planning algorithm for predictive state representation. In: Proceedings of IJCAI, pp. 1520–1521 (2003)
Izadi, M.T., Precup, D.: Model minimization by linear PSR. In: Proceedings of IJCAI, pp. 1749–1750 (2005a)
Izadi, M.T., Precup, D.: Using rewards in POMDP belief update. In: ECML 2005 (2005b)
James, M., Singh, S., Littman, M.: Planning with predictive state representation. In: Proceedings of International Conference on Machine Learning and Applications (ICMLA) (2004)
James, M.R.: Using Predictions for Planning and Modeling in Stochastic Environments. PhD thesis, The University of Michigan (2005)
James, M.R., Singh, S.: Planning in models that combine memory with predictive representations of state. In: Proceedings of AAAI (2005)
James, M.R., Wessling, T., Vlassis, N.: Improving approximate value iteration using memories and predictive state representations. In: Proceedings of AAAI (2006)
Littman, M., Sutton, R., Singh, S.: Predictive representations of state. In: Proceedings of NIPS 2001 (2002)
Pineau, J., Gordon, G., Thrun, S.: Point-based value iteration: An anytime algorithms for POMDPs. In: Proceedings of IJCAI, pp. 1025–1032 (2003)
Pineau, J., Gordon, G.: POMDP Planning for Robust Robot Control. In: Proceedings of International Symposium on Robotics Research (ISRR)
Poupart, P., Boutilier, C.: Value-directed Compression of POMDPs. In: Proceedings of NIPS 2002, pp. 1547–1554 (2003)
Poupart, P., Boutilier, C.: VDCBPI: an Approximate Scalable Algorithm for Large Scale POMDPs. In: Proceedings of NIPS 2003, pp. 1081–1088 (2004)
Singh, S., James, M.R., Rudary, M.R.: Predictive state representations: a new theory for modeling dynamical systems. In: Proceedings of UAI, pp. 512–519 (2004)
Smith, T., Simmons, R.: Heuristic search value iteration for POMDPs. In: Proceedings of UAI, pp. 520–527 (2004)
Spaan, M.T.J., Vlassis, N.A.: PERSEUS: Randomized point-base value iteration for POMDPs. Journal of Artificial Intelligence Research, 195–220 (2005)
Rafols, E., Ring, M., Sutton, R.S., Tanner, B.: Using Predictive Representations to Improve Generalization in Reinforcement Learning. In: Proceedings of IJCAI (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Izadi, M.T., Precup, D. (2008). Point-Based Planning for Predictive State Representations. In: Bergler, S. (eds) Advances in Artificial Intelligence. Canadian AI 2008. Lecture Notes in Computer Science(), vol 5032. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68825-9_13
Download citation
DOI: https://doi.org/10.1007/978-3-540-68825-9_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-68821-1
Online ISBN: 978-3-540-68825-9
eBook Packages: Computer ScienceComputer Science (R0)