Mathematical Methods of Operations Research

, Volume 49, Issue 2, pp 255–266 | Cite as

The finiteness of the reward function and the optimal value function in Markov decision processes

  • Qiying Hu
  • Chen Xu


This paper studies the discrete time Markov decision processes (MDP) with expected discounted total reward, where the state space is countable, the action space is measurable, the reward function is extended real-valued, and the discount rate may be any real number. Two conditions (GC) and (C) are presented, which are weaker than that presented in literature. By eliminating some worst actions, the state space S can be partitioned into sets S , S −∞, S 0, on which the optimal value function equals +∞ , −∞ or is finite, respectively. Furthermore, the validity of the optimality equation is shown when its right hand side is well defined, especially, when it is restricted to the subset S 0. The reward function r (i, a) is finite and bounded above in a for each iS 0. Finally, some sufficient conditions for (GC) and (C) are given.

Key words: Markov decision processes expected discounted total rewards optimality equation decomposing the state space eliminating actions. 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer-Verlag Berlin Heidelberg 1999

Authors and Affiliations

  • Qiying Hu
    • 1
  • Chen Xu
    • 2
  1. 1.School of Economy and Management, Xidian University, Xian, P.R. China (e-mail:
  2. 2.Department of Soft Science, Shenzhen University, Shenzhen, Guang Dong, P.R. ChinaCN

Personalised recommendations