Numerical Methods for Stochastic Differential Games: The Ergodic Cost Criterion

  • Harold J. Kushner
Part of the Annals of the International Society of Dynamic Games book series (AISDG, volume 9)


The Markov chain approximation method is a widely used, relatively easy to apply, and efficient family of methods for the bulk of stochastic control problems in continuous time, for reflected jump-diffusion type models. It is the most general method available, it has been shown to converge under broad conditions, via probabilistic methods, and there are good algorithms for solving the numerical problems, if the dimension is not too high (see the basic reference [16]). We consider a class of stochastic differential games with a reflected diffusion system model and ergodic cost criterion, where the controls for the two players are separated in the dynamics and cost function. The value of the game exists and the numerical method converges to this value as the discretization parameter goes to zero. The actual numerical method solves a stochastic game for a finite state Markov chain and ergodic cost criterion. The essential conditions are nondegeneracy of the diffusion and that a weak local consistency condition hold “almost everywhere” for the numerical approximations, just as for the control problem. The latter is close to a minimal condition. Such ergodic and “separated” game models occur in risk-sensitive and robust control.


Differential Game Dynamic Game Stochastic Game Stochastic Control Problem Stochastic Differential Game 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    M. Bardi, S. Bottacini, and M. Falcone. Convergence of discrete schemes for discontinuous value functions of pursuit-evasion games. In G.I. Olsder, editor, New Trends in Dynamic Games and Applications. Birkhäuser, Boston, 1995.Google Scholar
  2. [2]
    M. Bardi, M. Falcone, and P. Soravia. Fully discrete schemes for the value function of pursuit-evasion games. In T. Basar and A. Haurie, editors, Advances in Stochastic Games and Applications. Birkhäuser, Boston, 199Google Scholar
  3. [3]
    M. Bardi, M. Falcone, and P. Soravia. Numerical methods for pursuitevasion games via viscosity solutions. In M. Bardi, T.E.S. Raghavan, and T. Parthasarathy, editors, Stochastic and Differential Games: Theory and Numerical Methods. Birkh„auser, Boston, 1998.Google Scholar
  4. [4]
    M. Bardi and P. Soravia. Approximation of differential games of pursuitevasion by discrete time games. In R.P. Hämäläinen and H.K. Ehtamo, editors, Differential Games: Developments in Modeling and Computation. Springer-Verlag, Berlin and New York, 1991.Google Scholar
  5. [5]
    P. Billingsley. Convergence of Probability Measures; second edition. Wiley, New York, 1999.Google Scholar
  6. [6]
    P. Dupuis and H. Ishii. On Lipschitz continuity of the solution mapping to the Skorokhod problem, with applications. Stochastics Stochastic Rep., 35:31–62, 1991.MATHMathSciNetGoogle Scholar
  7. [7]
    S.N. Ethier and T.G. Kurtz. Markov Processes: Characterization and Convergence. Wiley, New York, 1986.MATHGoogle Scholar
  8. [8]
    W.H. Fleming and W. McEneaney. Risk-sensitive control on an infinite time horizon. SIAM J. Control Optim., 33:1881–1815, 1995.MATHCrossRefMathSciNetGoogle Scholar
  9. [9]
    I. Karatzas and S.E. Shreve. Brownian Motion and Stochastic Calculus. Springer-Verlag, New York, 198Google Scholar
  10. [10]
    H. J. Kushner. The Gauss-Seidel numerical procedure for Markov stochastic games. IEEE Trans in Automatic Control, 49:1779–1782, 2004.CrossRefMathSciNetGoogle Scholar
  11. [11]
    H.J. Kushner. Probability Methods for Approximations in Stochastic Control and for Elliptic Equations. Academic Press, New York, 1977.MATHGoogle Scholar
  12. [12]
    H.J. Kushner. Numerical methods for stochastic control problems in continuous time. SIAMJ. Control Optim., 28:999–1048, 1990.MATHCrossRefMathSciNetGoogle Scholar
  13. [13]
    H.J. Kushner. Heavy Traffic Analysis of Controlled Queueing and Communication Networks. Springer-Verlag, Berlin and New York, 2001.Google Scholar
  14. [14]
    H J. Kushner. Numerical methods for stochastic differential games. SIAM J. Control Optim., 41:457–486, 2002.MATHCrossRefMathSciNetGoogle Scholar
  15. [15]
    H.J. Kushner. Numerical approximations for stochastic differential games: The ergodic case. SIAMJ. Control Optim., 42:1911–1933, 2004.MATHCrossRefMathSciNetGoogle Scholar
  16. [16]
    H.J. Kushner and P. Dupuis. Numerical Methods for Stochastic Control Problems in Continuous Time. Springer-Verlag, Berlin and New York, 1992. Second edition, 2001.MATHGoogle Scholar
  17. [17]
    O. Pourtallier and M. Tidball. Approximation of the value function for a class of differential games with target. Research report 2942, INRIA, France, 1996.Google Scholar
  18. [18]
    O. Pourtallier and B. Tolwinski. Discretization of Isaac’s equation. Report: INRIA, France, 1992.Google Scholar
  19. [19]
    M. Tidball. Undiscounted zero-sum differential games with stopping times. In G.J. Olsder, editor, New Trends in Dynamic Games and Applications. Birkhäuser, Boston, 1995.Google Scholar
  20. [20]
    M. Tidball and R.L.V. González. Zero-sum differential games with stopping times: Some results and about its numerical resolution. In T. Basar and A. Haurie, editors, Advances in Dynamic Games and Applications. Birkhäuser, Boston, 1994.Google Scholar

Copyright information

© Birkhäuser Boston 2007

Authors and Affiliations

  • Harold J. Kushner
    • 1
  1. 1.Division of Applied MathematicsBrown UniversityProvidenceUSA

Personalised recommendations