Abstract
This chapter is concerned with the Linear Programming (LP) approach to MDPs in general Borel spaces, valid for several criteria, including the finite horizon and long run expected average cost, as well as the infinite horizon expected discounted cost.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
E. Altman, Constrained Markov Decision Processes, Chapman & Hall/CRC, Boca Raton, 1999.
E.J. Anderson and P. Nash, Linear Programming in Infinite-Dimensional Spaces, Wiley, Chichester, U.K., 1987.
R.B. Ash, Real Analysis and Probability, Academic Press, New York, 1972.
P. Billingsley, Convergence of Probability Measures, Wiley, New York, 1968.
N. Bourbaki, Integration, Chap. IX. Hermann, Paris, 1969.
H. Brézis, Analyse Fonctionnelle: Théorie and Applications, 4th Ed., Masson, Paris, 1993.
A. Bhatt and V. Borkar, “Occupation measures for controlled Markov processes: Characterization and optimality,” Ann. Probab. 24, 531–1562, 1996.
V. Borkar, “A convex analytic approach to Markov decision processes,” Probab. Theory Relat. Fields 78, 583–602, 1988.
V. Borkar, “Ergodic control of Markov chains with constraints — the general case,” SIAM J. Control Optimization 32, 176–186, 1994.
G.T. De Ghellinck, “Les problèmes de décisions séquentielles,” Cahiers du Centre d’Etudes de Recherche Opérationnelle 2, 161–179, 1960.
F. D’Epenoux, “Sur un problème de production et de stockage dans l’aléatoire,” Revue Française de Recherche Opérationnelle 14, 3–16, 1960.
E.V. Denardo, “On linear programming in a Markov decision problem,” Management Science 16, 281–288, 1970.
W.K. Haneveld, Duality in Stochastic Linear and Dynamic Programming, Lecture Notes in Economics and Mathematical Systems, Springer-Verlag, 1986.
W.R. Heilmann, “Solving stochastic dynamic programming problems by linear programming - an annotated bibliography,” Z. Oper. Res. Ser. A 22, 43–53, 1978.
W.R. Heilmann, “Solving a general discounted dynamic program by linear programming,” Z. Wahrsch. Gebiete 48, 339–346, 1979.
O. Hernández-Lerma and J. González-Hernández, “Infinite linear programming and multichain Markov control processes in uncountable spaces,” SIAM J. Contr. Optim. 36, 313–335, 1998.
O. Hernández-Lerma, Adaptive Markov Control Processes, Springer-Verlag, New York, 1989.
O. Hernandez-Lerma and J. González-Hernández, “Constrained Markov control processes in Borel spaces: the discounted case”, Math. Meth. Oper. Res. 52, 271–285, 2000.
O. Hernández-Lerma and J.B. Lasserre, Discrete-Time Markov Control Processes: Basic Optimality Criteria, Springer-Verlag, New York, 1996.
O. Hernández-Lerma and J.B. Lasserre, “Approximation schemes for infinite linear programs,” SIAM J. Optim. 8, 973–988, 1998.
O. Hernández-Lerma and J.B. Lasserre, ‘Linear programming approximations for Markov control processes in metric spaces,” Acta Appl. Math. 51, 123–139, 1998.
O. Hernández-Lerma and J.B. Lasserre, Further Topics on Discrete-Time Markov Control processes, Springer-Verlag, New York, 1999.
O. Hernández-Lerma and R. Romera, “Pareto optimality in multiobjective Markov control processes”, Reporte interno #278, Depto. de Matematicas, CINVESTAV-IPN, Mexico, (submitted).
K. Hinderer, Foundations of Non-stationary Dynamic Programming with Discrete-Time Parameter, Lecture Notes in Oper. Res. and Math. Syst. 33, Springer-Verlag, Berlin, 1970.
A. Hordijk and L.C.M. Kallenberg, “Linear programming and Markov decision chains,” Management Science 25, 352–362, 1979.
A. Hordijk and J.B. Lasserre, “Linear programming formulation of MDPs in countable state space: the multichain case,” Zeitschrift fur Oper. Res. 40, 91–108, 1994.
Y. Huang and M. Kurano, “The LP approach in average rewards MDPs with multiple cost constraints: the countable state case”, J. Infor. Optim. Sci. 18, 33–47, 1997.
L.C.M. Kallenberg, Linear Programming and Finite Markovian Control Problems, Mathematical Centre Tracts 148, Amsterdam, 1983.
J.B. Lasserre, “Average optimal stationary policies and linear programming in countable space Markov decision processes,” J. Math. Anal. Appl. 183, 233–249, 1994.
A.S. Manne, “Linear programming and sequential decisions,” Management Science 6, 259–267, 1960.
M.S. Mendiondo and R. Stockbridge, “Approximation of infinite-dimensional linear programming problems which arise in stochastic control,” SIAM J. Contr. Optim. 36, 1448–1472, 1998.
K.R. Parthasarathy, Probability Measures on Metric Spaces, Academic Press, New York, 1967.
A.B. Piunovskiy, Optimal Control of Random Sequences in Problems with Constraints, Kluwer Academic Publishers, Dordrecht, 1997.
U. Rieder, “Measurable selection theorems for optimization problems,” Manuscripta Math. 24, 115–131, 1978.
A.P. Robertson and W. Robertson, Topological Vector Spaces, Cambridge University Press, Cambridge, U.K., 1964.
W. Rudin, Real and Complex Analysis, 3rd ed. McGraw-Hill, New York, 1986.
L. Sennott, “On computing average optimal policies with application to routing to parallel queues”, Math. Meth. Oper. Res. 45, 45–62, 1997.
L. Sennott, “Stochastic dynamic programming and the control of queueing systems”, Wiley, New York, 1999.
R. Stockbridge, “Time-average control of martingale problems: A linear programming formulation,” Ann. Prob. 18, 190–205, 1990.
A.M. Vershik, “Some remarks on the infinite-dimensional problems of linear programming,” Russian Math. Surveys 29, 117–124, 1970.
A.M. Vershik and V. Temel’t, “Some questions concerning the approximation of the optimal value of infinite-dimensional problems in linear programming,” Siberian Math. J. 9, 591–601, 1968.
K. Yamada, “Duality theorem in Markovian decision problems,” J. Math. Anal. Appl. 50, 579–595, 1975.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer Science+Business Media New York
About this chapter
Cite this chapter
Hernández-Lerma, O., Lasserre, J.B. (2002). The Linear Programming Approach. In: Feinberg, E.A., Shwartz, A. (eds) Handbook of Markov Decision Processes. International Series in Operations Research & Management Science, vol 40. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-0805-2_12
Download citation
DOI: https://doi.org/10.1007/978-1-4615-0805-2_12
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-5248-8
Online ISBN: 978-1-4615-0805-2
eBook Packages: Springer Book Archive