Zero-Sum Ergodic Semi-Markov Games with Weakly Continuous Transition Probabilities

  • A. Jaśkiewicz


Zero-sum ergodic semi-Markov games with weakly continuous transition probabilities and lower semicontinuous, possibly unbounded, payoff functions are studied. Two payoff criteria are considered: the ratio average and the time average. The main result concerns the existence of a lower semicontinuous solution to the optimality equation and its proof is based on a fixed-point argument. Moreover, it is shown that the ratio average as well as the time average payoff stochastic games have the same value. In addition, one player possesses an ε-optimal stationary strategy (ε>0), whereas the other has an optimal stationary strategy.


Zero-sum semi-Markov games Optimality equations ε-optimal strategies 


  1. 1.
    Jaśkiewicz, A., Nowak, A.S.: Zero-sum ergodic stochastic games with Feller transition probabilities. SIAM J. Control Optim. 45, 773–789 (2006) MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Küenle, H.-U.: On Markov games with average reward criterion and weakly continuous transition probabilities. SIAM J. Control Optim. 45, 2156–2168 (2007) MATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    González-Trejo, J.I., Hernández-Lerma, O., Hoyos-Reyes, L.F.: Minimax control of discrete-time stochastic systems. SIAM J. Control Optim. 41, 1626–1659 (2003) MATHCrossRefGoogle Scholar
  4. 4.
    Jaśkiewicz, A.: Zero-sum semi-Markov games. SIAM J. Control Optim. 41, 723–739 (2002) MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Vega-Amaya, O.: Zero-sum semi-Markov games: fixed-point solutions of the Shapley equation. SIAM J. Control Optim. 42, 1876–1894 (2003) MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Jaśkiewicz, A.: A fixed point approach to solve the average cost optimality equation for semi-Markov decision processes with Feller transition probabilities. Commun. Stat.—Theory Methods 36(14), 2559–2575 (2007) MATHCrossRefGoogle Scholar
  7. 7.
    Jaśkiewicz, A.: On the equivalence of two expected average cost criteria for semi-Markov control processes. Math. Oper. Res. 29(2), 326–338 (2004) MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Jaśkiewicz, A., Nowak, A.S.: Optimality in Feller semi-Markov control processes. Oper. Res. Lett. 34, 713–718 (2006) MATHCrossRefMathSciNetGoogle Scholar
  9. 9.
    Maitra, A., Sudderth, W.: Borel stochastic games with limsup payoffs. Ann. Probab. 21, 861–885 (1993) MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Kushner, H.J.: Numerical approximations for stochastic differential games: the ergodic case. SIAM J. Control Optim. 42, 1911–1933 (2004) MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability. Springer, New York (1993) MATHGoogle Scholar
  12. 12.
    Nishimura, K., Stachurski, J.: Stochastic optimal policies when the discount rate vanishes. J. Econ. Dyn. Control 31, 1416–1430 (2007) CrossRefMathSciNetMATHGoogle Scholar
  13. 13.
    Stockey, N.L., Lucas, R.E., Prescott, E.C.: Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge (1989) Google Scholar
  14. 14.
    Bertsekas, D.P., Shreve, S.E.: Stochastic Optimal Control: The Discrete Time Case. Academic Press, New York (1978) MATHGoogle Scholar
  15. 15.
    Klein, E., Thompson, A.C.: Theory of Correspondences. Wiley, New York (1984) MATHGoogle Scholar
  16. 16.
    Himmelberg, C.J., Van Vleck, F.S.: Multifunctions with values in a space of probability measures. J. Math. Anal. Appl. 50, 108–112 (1975) MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Meyn, S.P., Tweedie, R.L.: Computable bounds for geometric convergence rates of Markov chains. Ann. Appl. Probab. 4, 981–1011 (1994) MATHCrossRefMathSciNetGoogle Scholar
  18. 18.
    Ross, S.M.: Applied Probability Models with Optimization Applications. Holden-Day, San Francisco (1970) MATHGoogle Scholar
  19. 19.
    Sennott, L.I.: Average cost semi-Markov decision processes and the control of queueing systems. Probab. Eng. Inf. Sci. 3, 247–272 (1989) MATHCrossRefGoogle Scholar
  20. 20.
    Yushkevich, A.: On semi-Markov controlled models with an average reward criterion. Theory Probab. Appl. 26, 796–803 (1981) CrossRefGoogle Scholar
  21. 21.
    Nowak, A.S.: Measurable selection theorems for minimax stochastic optimization problems. SIAM J. Control Optim. 23, 466–476 (1985) MATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Hernández-Lerma, O., Lasserre, J.B.: Further Topics on Discrete-Time Markov Control Process. Springer, New York (1999) Google Scholar
  23. 23.
    Nowak, A.S.: Semicontinuous nonstationary stochastic games. J. Math. Anal. Appl. 117, 84–99 (1986) MATHCrossRefMathSciNetGoogle Scholar
  24. 24.
    Fan, K.: Minimax theorems. Proc. Natl. Acad. Sci. USA 39, 42–47 (1953) MATHCrossRefGoogle Scholar
  25. 25.
    Feinberg, E.A.: Constrained semi-Markov decision processes with average rewards. Math. Methods Oper. Res. 39, 257–288 (1994) MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Resnick, S.I.: Adventures in Stochastic Processes. Birkhauser, Boston (1992) MATHGoogle Scholar
  27. 27.
    Feinberg, E.A., Lewis, M.E.: Optimality of four-threshold policies in inventory systems with customer returns and borrowing/storage options. Probab. Eng. Inf. Sci. 19, 45–71 (2005) MATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Institute of MathematicsPolish Academy of SciencesWarszawaPoland

Personalised recommendations