Abstract
Desirable properties of the infinite histories of a finite state Markov Decision Process are specified in terms of a finite number of events represented as ω-regular sets. An infinite history of the process produces a reward which depends on the properties it satisfies. We investigate the existence of optimal policies and provide algorithms for the construction of such policies.
Extended Abstract
Supported in part by ESPRIT BRA project SPEC.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
References
S. Aggarwal, C. Courcoubetis and P. Wolper, "Adding Liveness Properties to Coupled Finite-State Machines", AT&T Bell Laboratories Technical Memorandum, to appear in ACM TOPLAS.
F. Beutler and K. Ross, "Optimal Policies for Controlled Markov Chains with a constraint", J. Math. Analysis and Appl., 112, pp. 236–252, 1985.
F. Beutler and K. Ross, "Time-Average Optimal Constrained Semi-Markov Decision Processes", Adv. Applied Prob., 18(2), pp. 341–359, 1986.
R. Cieslak, C. Desclaux, A. Fawaz and P. Varaiya, "Supervisory Control of Discrete-Event Processes with Partial Information", IEEE Trans. on Automatic Control, 33(3), pp. 249–260, March 1988.
C. Courcoubetis and M. Yannakakis, "Verifying Temporal Properties of Finite-State Probabilistic Programs", Proc. of 29th FOCS, 1988, pp. 338–345, Oct. 1988.
C. Derman, Finite-State Markovian Decision Processes, Academic Press, New York, 1970.
L.C.M. Kallenberg, Linear Programming and Finite Markovian Control Problems, Mathematical Center Tracts, Amsterdam, 1983.
R. McNaughton, "Testing and Generating Infinite Sequences by a Finite Automaton", Information and Control, 9(1966), pp. 521–530.
A. Pnueli, "The Temporal Logic of Concurrent Programs", Theoretical Computer Science 13(1981), pp. 45–60.
J. P. Queille, J. Sifakis, "Fairness and Related Properties in Transition Systems", Research Report #292, IMAG, Grenoble, 1982.
M. O. Rabin, "Automata on Infinite Objects and Church's Problem", Proc. Regional AMS Conf. Series in Math. 13(1972), pp. 1–22.
S. Ross, Introduction to Stochastic Dynamic Programming, Academic Press, New York, 1983.
K. Ross and R. Varadarajan, "Markov Decision Processes with Sample Path Constraints; the Communicating Case", to appear in Operations Research, 1990.
P. Ramadge and W.M. Wonham, "Supervisory Control of a Class of Discrete-Event Processes", SIAM J. on Contr. and Optimization, 25(1), pp. 206–230, January 1987.
S. Safra, "On the Complexity of ω-automata", Proc. of 29th FOCS, 1988, pp. 319–327, Oct. 1988.
M. Vardi, "Automatic Verification of Probabilistic Concurrent Finite-State Programs", Proc. of 26th STOC, 1985.
M. Vardi and P. Wolper, "An automata-Theoretic Approach to Automatic Program Verification", Proc. 1st Symp. on Logic in Computer Science, 1986.
W. M. Wonham and P. Ramadge, "On the Supremal Controllable Sublanguage of a Given Language", SIAM J. on Contr. and Optimization, 25(3), pp. 637–659, May 1987.
P. Wolper, M. Y. Vardi, A. P. Sistla, "Reasoning about Infinite Computation Paths", Proc. of 24th IEEE Symp. on Foundations of Computer Science, 1983, pp. 185–194.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1990 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Courcoubetis, C., Yannakakis, M. (1990). Markov decision processes and regular events. In: Paterson, M.S. (eds) Automata, Languages and Programming. ICALP 1990. Lecture Notes in Computer Science, vol 443. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0032043
Download citation
DOI: https://doi.org/10.1007/BFb0032043
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-52826-5
Online ISBN: 978-3-540-47159-2
eBook Packages: Springer Book Archive