Adaptive control of Markov chains

  • V. Borkar
  • P. Varaiya
Adaptive Control
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 14)


Consider a controlled Markov chain whose transition probabilities depend upon an unknown parameter α taking values in a finite set A. To each α is associated a prespecified stationary control law ø(α). The adaptive control law selects at each time t the control action indicated by ø(αt) where αt is the maximum likelihood estimate of α. It is shown that αt converges to a parameter α* such that the transition probabilities corresponding to α* and ø(α*) are the same as those corresponding to α0 and ø(α*) where α0 is the true parameter.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    P. Mandl, Estimation and control in Markov chains, Adv. Appl. Prob. 6, 40–60, 1974.Google Scholar
  2. [2]
    K. Åström and B. Wittenmark, On self-tuning regulators, Automatic 9, 185–199, 1973.CrossRefGoogle Scholar
  3. [3]
    L. Ljung and B. Wittenmark, Asymptotic properties of self-tuning regulators, TFRT-3071, Dept. of Auto. Contr., Lund Institute of Technology, 1974Google Scholar
  4. [4]
    Y. Baram and N. Sandell, Jr., Consistent situation of finite parameter sets with application to linear system identification, IEEE Trans. Auto. Contr., vol. AC-23, no. 3, 451–454, June 1978.CrossRefGoogle Scholar
  5. [5]
    M. Loève, Probability Theory, Princeton: Van Nostrand, 1960.Google Scholar

Copyright information

© Springer-Verlag 1979

Authors and Affiliations

  • V. Borkar
    • 1
  • P. Varaiya
    • 1
  1. 1.Department of Electrical Engineering and Computer Sciences and the Electronics Research LaboratoryUniversity of CaliforniaBerkeleyUSA

Personalised recommendations