pp 1–16 | Cite as

Finite horizon risk-sensitive continuous-time Markov decision processes with unbounded transition and cost rates

  • Xin Guo
  • Qiuli Liu
  • Yi ZhangEmail author
Research Paper


We consider a risk-sensitive continuous-time Markov decision process over a finite time duration. Under the conditions that can be satisfied by unbounded transition and cost rates, we show the existence of an optimal policy, and the existence and uniqueness of the solution to the optimality equation out of a class of possibly unbounded functions, to which the Feynman–Kac formula was also justified to hold.


Continuous-time Markov decision processes Risk-sensitive criterion Optimality equation 

Mathematics Subject Classification

Primary 90C40 Secondary 60J75 



This work is partially supported by Natural Science Foundation of Guangdong Province (Grant No. 2014A030313438), Zhujiang New Star (Grant No. 201506010056), Guangdong Province outstanding young teacher training plan (Grant No. YQ2015050).

Compliance with ethical standards

Conflict of interest

There is no potential conflicts of interest.

Ethical standard

Research do not have human participants and/or animals.


  1. Bäuerle N, Rieder U (2014) More risk-sensitive Markov decision processes. Math Oper Res 39:105–120CrossRefGoogle Scholar
  2. Bäuerle N, Popp A (2018) Risk-sensitive stopping problems for continuous-time Markov chains. Stochastics 90:411–431CrossRefGoogle Scholar
  3. Cavazos-Cadena R, Montes-de-Oca R (2000) Optimal stationary policies in risk-sensitive dynamic programs with finite state space and nonnegative rewards. Appl Math 27:167–185Google Scholar
  4. Cavazos-Cadena R, Montes-de-Oca R (2000) Nearly optimal policies in risk-sensitive positive dynamic programming on discrete spaces. Math Methods Oper Res 52:133–167CrossRefGoogle Scholar
  5. Ghosh M, Saha S (2014) Risk-sensitive control of continuous time Markov chains. Stochastics 86:655–675CrossRefGoogle Scholar
  6. Guo X, Zhang Y (2018) On risk-sensitive piecewise deterministic Markov decision processes. Appl. Math Optim. in press.
  7. Guo XP, Huang X, Huang Y (2015) Finite-horizon optimality for continuous-time Markov decision processes with unbounded transition rates. Adv Appl Probab 47:1064–1087CrossRefGoogle Scholar
  8. Guo XP, Piunovskiy A (2011) Discounted continuous-time Markov decision processes with constraints: unbounded transition and loss rates. Math Oper Res 36:105–132CrossRefGoogle Scholar
  9. Hernández-Lerma O, Lasserre J (1996) Discrete-time Markov control processes. Springer, New YorkCrossRefGoogle Scholar
  10. Hernández-Lerma O, Lasserre J (1999) Further topics on discrete-time Markov control processes. Springer, New YorkCrossRefGoogle Scholar
  11. Howard R, Matheson J (1972) Risk-sensitive Markov decision proceses. Manag Sci 18:356–369CrossRefGoogle Scholar
  12. Jacod J (1975) Multivariate point processes: predictable projection, Radon–Nicodym derivatives, representation of martingales. Z. Wahrscheinlichkeitstheorie und verwandte Gebiete 31:235–253CrossRefGoogle Scholar
  13. Jaśkiewicz A (2008) A note on negative dynamic programming for risk-sensitive control. Oper Res Lett 36:531–534CrossRefGoogle Scholar
  14. Kitaev M (1986) Semi-Markov and jump Markov controlled models: average cost criterion. Theory Probab Appl 30:272–288CrossRefGoogle Scholar
  15. Kitaev M, Rykov V (1995) Controlled queueing systems. CRC Press, New YorkGoogle Scholar
  16. Kumar KS, Chandan P (2013) Risk-sensitive control of jump process on denumerable state space with near monotone cost. Appl Math Optim 68:311–331CrossRefGoogle Scholar
  17. Patek S (2001) On terminating Markov decision processes with a risk-averse objective function. Automatica 37:1379–1386CrossRefGoogle Scholar
  18. Piunovski A, Khametov V (1985) New effective solutions of optimality equations for the controlled Markov chains with continuous parameter (the unbounded price-function). Problems Control Inform Theory 14:303–318Google Scholar
  19. Piunovskiy A, Zhang Y (2011) Discounted continuous-time Markov decision processes with unbounded rates: the convex analytic approach. SIAM J Control Optim 49:2032–2061CrossRefGoogle Scholar
  20. Piunovskiy A, Zhang Y (2014) Discounted continuous-time Markov decision processes with unbounded rates and randomized history-dependent policies: the dynamic programming approach. 4OR-Q J Oper Res 12, 4975Google Scholar
  21. Wei Q (2016) Continuous-time Markov decision processes with risk-sensitive finite-horizon cost criterion. Math Methods Oper Res 84:461–487CrossRefGoogle Scholar
  22. Wei Q, Chen X (2016) Continuous-time Markov decision processes under the risk-sensitive average cost criterion. Oper Res Lett 44:457–462CrossRefGoogle Scholar
  23. Zhang Y (2017) Continuous-time Markov decision processes with exponential utility. SIAM J Control Optim 55:2636–2660CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Mathematical SciencesUniversity of LiverpoolLiverpoolUK
  2. 2.School of Mathematical SciencesSouth China Normal UniversityGuangzhouChina

Personalised recommendations