Optimal Control of Perturbed Markov Chains: The Multitime Scale Case

  • J. P. Quadrat
Part of the International Centre for Mechanical Sciences book series (CISM, volume 280)


Given a controled perturbed Markov chain of transition matrix mu(ε), where ε is the perturbation scale and u the control, we study the solution expansion in ε, wε, of the dynamic programming equation:
mu(ε), cu(ε), λ(ε) are polynomials in ε. The case λ(ε) = ε leads to study Markov chains on a time scale of order 1/ε. The state space and the control set are finite.


Markov Chain Dynamic Programming Equation Implicit System Aggregate Chain Recurrent Class 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    P. BERNHARD. Sur les systèmes dynamiques linéaires implicites singuliers, à paraître SIAM J. on control and optimization et rapport INRIA 69, 1981.Google Scholar
  2. [2]
    M. CODERCH, A.S. WILLSKY, S.S. SASTRY, D.A. CASTANON. Hierarchical aggregation of linear systems with multiple time scales, MIT Report LIDS-P-1187, mars 1982.Google Scholar
  3. [3]
    M. CODERCH, A.S. WILLSKY, S.S. SASTRY. Hierarchical aggregation of singularly perturbed finite state Markov chains submitted to stochastics.Google Scholar
  4. [4]
    P.J. COURTOIS. Decomposability, ACM Monograph Series, Academic Press, 1977.Google Scholar
  5. [5]
    F. DELEBECQUE. A reduction process for perturbed Markov chains, à paraître SIAM J. of applied math. to appear.Google Scholar
  6. [6]
    F. DELEBECQUE, J.P. QUADRAT. Optimal control of Markov chains admitting strong and weak interactions, Automatica, Vol. 17, no 2, pp. 281–296, 1981.CrossRefMATHMathSciNetGoogle Scholar
  7. [7]
    F. DELEBECQUE,- J.P. QUADRAT. The optimal cost expansion of finite controls finite states Markov chains with weak and stong interactions. Analysis and optimization of systems, Lecture Notes an control and Inf. Science 28 Springer Verlag, 1980.Google Scholar
  8. [8]
    A.A. PERVOZVANSKII, A.V. GAITSGORI. Decomposition aggregation and approximate optimization en Russe, Nauka, Moscou, 1979.Google Scholar
  9. [9]
    T. KATO. Perturbation theory for linear operator, Springer Verlag, 1976.Google Scholar
  10. [10]
    B.L. MILLER, A.F. VEINOTT. Discrete dynamic programming with small interest rate. An. math. stat. 40, 1969, pp. 366–370.CrossRefMATHMathSciNetGoogle Scholar
  11. [11]
    R. PHILIPS, P.KOKOTOVIC. A singular perturbation approach to modelling and control of Markov chains IEEE A.C. Bellman issue, 1981.Google Scholar
  12. [12]
    H. SIMON, A. ANDO. Aggregation of variables in dynamic systems, Econometrica, 29, 111–139, 1961.CrossRefMATHGoogle Scholar
  13. [13]
    J. KEMENY, L. SNELL. Finite Markov chains, Van Nostrand, 1960.Google Scholar
  14. [14]
    O. MURON. Evaluation de politiques de maintenance pour un système complexe, RIRO, vol. 14, n° 3, pp. 265–282, 1980.MATHMathSciNetGoogle Scholar
  15. [15]
    S.L. CAMPBELL, C.D. MEYER Jr. Generalized inverses of linear transformations. Pitman, London, 1979.MATHGoogle Scholar

Copyright information

© Springer-Verlag Wien 1983

Authors and Affiliations

  • J. P. Quadrat
    • 1
  1. 1.INRIALe Chesnay CédexFrance

Personalised recommendations