Skip to main content
SpringerLink
Log in

Stochastic Stability and Drift Criteria for Markov Chains in Networked Control

  • Chapter
  • First Online:
Stochastic Networked Control Systems

Part of the book series: Systems & Control: Foundations & Applications ((SCFA))

  • 2378 Accesses

Abstract

This chapter presents an overview of the theory of Markov chains and drift criteria to establish stochastic stability of Markov chains. Random-time state-dependent stochastic drift criteria are presented together with a class of application areas in networked control systems. Criteria for transience and other forms of stochastic stability are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Aström, K.J., Bernhardsson, B.: Comparison of Riemann and Lebesgue sampling for first order stochastic systems. In: Proceedings of the IEEE Conference on Decision and Control, pp. 2011–2016, Dec 2002

    Google Scholar 

  2. Borkar, V.S.: Convex analytic methods in Markov decision processes. In: Feinberg, E.A., Shwartz, A. (eds.) Handbook of Markov Decision Processes, pp. 347–375. Kluwer, Boston (2001)

    Google Scholar 

  3. Borkar, V.S., Meyn, S.P.: The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim. 38, 447–469 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  4. Burnashev, M.V.: Data transmission over a discrete channel with feedback. Random transmission time. Prob. Inform. Transm. 12, 10–30 (1976)

    MATH  Google Scholar 

  5. Cloosterman, M.B.G., van de Wouw, N., Heemels, W.P.M.H., Nijmeijer, H.: Stability of networked control systems with uncertain time-varying delays. IEEE Trans. Autom. Control 54, 1575–1580 (2009)

    Article  Google Scholar 

  6. Connor, S.B., Fort, G.: State-dependent foster-lyapunov criteria for subgeometric convergence of Markov chains. Stoch. Process. Appl. 119, 176–4193 (2009)

    Article  MathSciNet  Google Scholar 

  7. Dai, J.G.: On positive harris recurrence of multiclass queueing networks: a unified approach via fluid limit models. Ann. Appl. Probab. 5, 49–77 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  8. Dai, J.G., Meyn, S.P.: Stability and convergence of moments for multiclass queueing networks via fluid limit models. IEEE Trans. Autom. Control 40, 1889–1904 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  9. Douc, R., Fort, G., Moulines, E., Soulier, P.: Practical drift conditions for subgeometric rates of convergence. Ann. Appl. Probab. 14, 1353–1377 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  10. Fort, G., Meyn, S.P., Moulines, E., Priouret, P.: ODE methods for skip-free Markov chain stability with applications to MCMC. Ann. Appl. Probab. 18, 664–707 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  11. Fralix, B.H.: Foster-type criteria for Markov chains on general spaces. J. Appl. Probab. 43, 1194–1200 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Gupta, V., Chung, T.H., Hassibi, B., Murray, R.M.: On a stochastic sensor selection algorithm with applications in sensor scheduling and sensor coverage. Automatica 42, 251–260 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  13. Hairer, M.: Convergence of Markov Processes. Lecture Notes, University of Warwick, 2010

    Google Scholar 

  14. Hernandez-Lerma, O., Lasserre, J.B.: Markov Chains and Invariant Probabilities. Birkhäuser, Basel (2003)

    Book  MATH  Google Scholar 

  15. Imer, O.C., Başar, T.: Optimal control with limited controls. In: Proceedings of the American Control Conference, pp. 298–303, Minneapolis, MN, 2006

    Google Scholar 

  16. Lasserre, J.B.: Invariant probabilities for Markov chains on a metric space. Stat. Probab. Lett. 34, 259–265 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  17. Lemmon, M.: Event-triggered feedback in control, estimation, and optimization. In: Bemporad, A., Heemels, M., Johansson, M. (eds.) Networked Control Systems, ser. Lecture Notes in Control and Information Sciences, pp. 293–358. Springer, New York, (2010)

    Google Scholar 

  18. Lipsa, G.M., Martins, N.C.: Remote state estimation with communication costs for first-order LTI systems. IEEE Trans. Autom. Control 56, 2013–2025 (2011)

    Article  MathSciNet  Google Scholar 

  19. Johansson, K.H., Rabi, M., Johansson, M.: Optimal stopping for event-triggered sensing and actuation. In: Proceedings of the IEEE Conference on Decision and Control, pp. 3607–3612, Dec 2008

    Google Scholar 

  20. Malyšev, V.A., Menšikov, M.V.: Ergodicity, continuity and analyticity of countable Markov chains. Trans. Moscow Math. Soc. 39, 1–48 (1981)

    Google Scholar 

  21. Meyn, S.P.: Control Techniques for Complex Networks. Cambridge University Press, Cambridge (2007)

    Book  Google Scholar 

  22. Meyn, S.P., Tweedie, R.: Stability of Markovian processes I: criteria for discrete-time chains. Adv. Appl. Probab. 24, 542–574 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  23. Meyn, S.P., Tweedie, R.: Markov Chains and Stochastic Stability. Springer, London (1993)

    Book  MATH  Google Scholar 

  24. Meyn, S.P., Tweedie, R.: State-dependent criteria for convergence of Markov chains. Ann. Appl. Probab. 4, 149–168 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  25. Polyanskiy, Y., Poor, H.V., Verdú, S.: Variable-length coding with feedback in the non-asymptotic regime. In: Proceedings of the IEEE International Symposium on Information Theory, pp. 231–235, 2010

    Google Scholar 

  26. Quevedo, D., Nešić, D.: Robust stability of packetized predictive control of nonlinear systems with disturbances and markovian packet losses. Automatica 48(8), 1803–1811 (2012)

    Article  MathSciNet  Google Scholar 

  27. Quevedo, D., Østergaard, J., Nešić, D.: Packetized predictive control of stochastic systems over bit-rate limited channels with packet loss. IEEE Trans. Autom. Control 56(12), 2854–2868 (2011)

    Article  Google Scholar 

  28. Roberts, G.O., Rosenthal, J.S.: General state space Markov chains and MCMC algorithms. Prob. Surv. 1, 20–71 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  29. Sahai, A.: Anytime information theory. Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA (2001)

    Google Scholar 

  30. Sahai, A., Mitter, S.: The necessity and sufficiency of anytime capacity for stabilization of a linear system over a noisy communication link part I: Scalar systems. IEEE Trans. Inf. Theor. 52(8), 3369–3395 (2006)

    Article  MathSciNet  Google Scholar 

  31. Tabuada, P.: Event-triggered real-time scheduling of stabilizing control tasks. IEEE Trans. Autom. Control 52, 1680–1685 (2007)

    Article  MathSciNet  Google Scholar 

  32. Wu, W., Arapostathis, A.: Optimal sensor querying: General markovian and lqg models with controlled observations. IEEE Trans. Autom. Control 53, 1392–1405 (2008)

    Article  MathSciNet  Google Scholar 

  33. You, K., Xie, L.: Minimum data rate for mean square stabilization of discrete LTI systems over lossy channels. IEEE Trans. Autom. Control. 55, 2373–2378, Oct 2010

    Article  MathSciNet  Google Scholar 

  34. Yüksel, S.: A random time stochastic drift result and application to stochastic stabilization over noisy channels. In: Proceedings of the Annual Allerton Conference on Communications, Control and Computing, Monticello, IL, 2009

    Google Scholar 

  35. Yüksel, S.: Stochastic stabilization of noisy linear systems with fixed-rate limited feedback. IEEE Trans. Autom. Control 55, 2847–2853 (2010)

    Article  Google Scholar 

  36. Yüksel, S., Başar, T.: Control over noisy forward and reverse channels. IEEE Trans. Autom. Control 56, 1014–1029 (2011)

    Article  Google Scholar 

  37. Yüksel, S., Meyn, S.P.: Random-time, state-dependent stochastic drift for Markov chains and application to stochastic stabilization over erasure channels. IEEE Trans. Autom. Control 58, 47–59 (2013)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Statistics, Queen’s University, Kingston, Ontario, Canada

    Serdar Yüksel

  2. Department of Electrical and Computer Engineering, Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, Urbana, Illinois, USA

    Tamer Başar

Authors
  1. Serdar Yüksel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tamer Başar
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

Yüksel, S., Başar, T. (2013). Stochastic Stability and Drift Criteria for Markov Chains in Networked Control. In: Stochastic Networked Control Systems. Systems & Control: Foundations & Applications. Birkhäuser, New York, NY. https://doi.org/10.1007/978-1-4614-7085-4_6

Download citation

Publish with us

Policies and ethics

Search

Navigation