Skip to main content
SpringerLink
Log in

The ODE Method and Spectral Theory of Markov Operators

  • Conference paper
  • First Online:
Stochastic Theory and Control

Part of the book series: Lecture Notes in Control and Information Sciences ((LNCIS,volume 280))

Abstract

We give a development of the ODE method for the analysis of recursive algorithms described by a stochastic recursion. With variability modeled via an underlying Markov process, and under general assumptions, the following results are obtained:

  1. (i)

    Stability of an associated ODE implies that the stochastic recursion is stable in a strong sense when a gain parameter is small.

  2. (ii)

    The range of gain-values is quantified through a spectral analysis of an associated linear operator, providing a non-local theory, even for nonlinear systems.

  3. (iii)

    A second-order analysis shows precisely how variability leads to sensitivity of the algorithm with respect to the gain parameter.

All results are obtained within the natural operator-theoretic framework of geometrically ergodic Markov processes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. L. Arnold. Random dynamical systems. Springer-Verlag, Berlin, 1998.

    MATH  Google Scholar 

  2. S. Balaji and S.P. Meyn. Multiplicative ergodicity and large deviations for an irreducible Markov chain. Stochastic Process. Appl., 90(1):123–144, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  3. R. Bellman. Limit theorems for non-commutative operations. I. Duke Math. J., 21, 1954.

    Google Scholar 

  4. Michel Benaïm. Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII, pages 1–68. Springer, Berlin, 1999.

    Chapter  Google Scholar 

  5. Albert Benveniste, Michel Métivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations. Springer-Verlag, Berlin, 1990. Translated from the French by Stephen S. Wilson.

    MATH  Google Scholar 

  6. D.P. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Atena Scientific, Cambridge, Mass, 1996.

    MATH  Google Scholar 

  7. B. Bharath and V. S. Borkar. Stochastic approximation algorithms: overview and recent trends. Sāadhanā, 24(4–5):425–452, 1999. Chance as necessity.

    Article  MATH  MathSciNet  Google Scholar 

  8. V.S. Borkar and S.P. Meyn. The O.D.E. Method for Convergence of Stochastic Approximation and Reinforcement Learning. SIAM J. Control Optim., 38:447–69, 2000.

    Article  MATH  MathSciNet  Google Scholar 

  9. P. Bougerol. Limit theorem for products of random matrices with Markovian dependence. In Proceedings of the 1st World Congress of the Bernoulli Society, Vol. 1 (Tashkent, 1986), pages 767–770, Utrecht, 1987. VNU Sci. Press.

    Google Scholar 

  10. A. Crisanti, G. Paladin, and A. Vulpiani. Products of random matrices in statistical physics. Springer-Verlag, Berlin, 1993.

    MATH  Google Scholar 

  11. O. Dabeer and E. Masry. The LMS adaptive algorithm: Asymptotic error analysis. In Proceedings of the 34th Annual Conference on Information Sciences and Systems, CISS 2000, pages WP1-6–WP1-7, Princeton, NJ, March 2000.

    Google Scholar 

  12. Paul Fischer and Hans Ulrich Simon, editors. Computational learning theory, Berlin, 1999. Springer-Verlag. Lecture Notes in Artificial Intelligence.

    Google Scholar 

  13. H. Furstenberg and H. Kesten. Products of random matrices. Ann. Math. Statist, 31:457–469, 1960.

    Article  MATH  MathSciNet  Google Scholar 

  14. László Gerencsér. Almost sure exponential stability of random linear differential equations. Stochastics Stochastics Rep., 36(2):91–107, 1991.

    MATH  MathSciNet  Google Scholar 

  15. R. Gharavi and V. Anantharam. Structure theorems for partially asynchronous iterations of a nonnegative matrix with random delays. Sādhanā, 24(4–5):369–423, 1999. Chance as necessity.

    MATH  MathSciNet  Google Scholar 

  16. S.V. Hanly and D. Tse. Multiaccess fading channels. II. Delay-limited capacities. IEEE Trans. Inform. Theory, 44(7):2816–2831, 1998.

    Article  MATH  MathSciNet  Google Scholar 

  17. J. A. Joslin and A. J. Heunis. Law of the iterated logarithm for a constant-gain linear stochastic gradient algorithm. SIAM J. Control Optim., 39(2):533–570 (electronic), 2000.

    Article  MATH  MathSciNet  Google Scholar 

  18. I. Kontoyiannis and S.P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Submitted, 2001. Also presented at the 2001 INFORMS Applied Probability Conference, NY, July, 2001.

    Google Scholar 

  19. I. Kontoyiannis and S.P. Meyn. Spectral theory and limit theorems for geometrically ergodic Markov processes. Part II: Empirical measures & unbounded functionals. Preprint, 2001.

    Google Scholar 

  20. H. J. Kushner. Approximation and weak convergence methods for random processes, with applications to stochastic systems theory. MIT Press, Cambridge, MA, 1984.

    MATH  Google Scholar 

  21. H.J. Kushner and G. Yin. Stochastic approximation algorithms and applications. Springer-Verlag, New York, 1997.

    MATH  Google Scholar 

  22. L. Ljung. On positive real transfer functions and the convergence of some recursive schemes. IEEE Trans. Automatic Control, AC-22(4):539–551, 1977.

    Article  MathSciNet  Google Scholar 

  23. Jean Mairesse. Products of irreducible random matrices in the (max, +) algebra. Adv. in Appl. Probab., 29(2):444–477, 1997.

    Article  MATH  MathSciNet  Google Scholar 

  24. M. Medard, S.P. Meyn, and J. Huang. Capacity benefits from channel sounding in Rayleigh fading channels. INFOCOM (submitted), 2001.

    Google Scholar 

  25. S. P. Meyn and R. L. Tweedie. Markov Chains and Stochastic Stability. Springer-Verlag, London, 1993.

    MATH  Google Scholar 

  26. S.P. Meyn and R.L. Tweedie. Computable bounds for geometric convergence rates of Markov chains. Ann. Appl. Probab., 4(4):981–1011, 1994.

    Article  MATH  MathSciNet  Google Scholar 

  27. G.V. Moustakides. Exponential convergence of products of random matrices, application to the study of adaptive algorithms. International Journal of Adaptive Control and Signal Processing, 2(12):579–597, 1998.

    Article  MathSciNet  Google Scholar 

  28. V. I. Oseledec. Markov chains, skew products and ergodic theorems for “general” dynamic systems. Teor. Verojatnost. i Primenen., 10:551–557, 1965.

    MathSciNet  Google Scholar 

  29. V. I. Oseledec. A multiplicative ergodic theorem. Characteristic Ljapunov, exponents of dynamical systems. Trudy Moskov. Mat. Obšč., 19:179–210, 1968.

    MathSciNet  Google Scholar 

  30. J. B. T. M Roerdink. Products of random matrices or ”why do biennials live longer than two years? CWI Quarterly, 2:37–44, 1989.

    MATH  MathSciNet  Google Scholar 

  31. E. Seneta. Non-negative Matrices and Markov Chains. Springer-Verlag, New York, Second edition, 1980.

    Google Scholar 

  32. H. Thorisson. Coupling, stationarity, and regeneration. Springer-Verlag, New York, 2000.

    MATH  Google Scholar 

  33. D. Tse and S.V. Hanly. Multiaccess fading channels. I. Polymatroid structure, optimal resource allocation and throughput capacities. IEEE Trans. Inform. Theory, 44(7):2796–2815, 1998.

    Article  MATH  MathSciNet  Google Scholar 

  34. Divakar Viswanath. Random Fibonacci sequences and the number 1.13198824.... Math. Comp., 69(231):1131–1155, 2000.

    Article  MATH  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Illinois at Urbana-Champaign, Urbana, IL, 61801, USA

    Jianyi Huang & Sean P. Meyn

  2. Brown University, Box F, 182 George St., Providence, RI, 02912, USA

    Ioannis Kontoyiannis

Authors
  1. Jianyi Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ioannis Kontoyiannis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sean P. Meyn
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Dep. of Mathematics, University of Kansas, 405 Snow Hall, Lawrence, KS, 66045, USA

    Bozenna Pasik-Duncan

Rights and permissions

Reprints and permissions

Copyright information

© 2002 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Huang, J., Kontoyiannis, I., Meyn, S.P. (2002). The ODE Method and Spectral Theory of Markov Operators. In: Pasik-Duncan, B. (eds) Stochastic Theory and Control. Lecture Notes in Control and Information Sciences, vol 280. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-48022-6_15

Download citation

  • DOI: https://doi.org/10.1007/3-540-48022-6_15

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-43777-2

  • Online ISBN: 978-3-540-48022-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation