Encyclopedia of Systems and Control

Living Edition
| Editors: John Baillieul, Tariq Samad

Perturbation Analysis of Steady-State Performance and Sensitivity-Based Optimization

  • Chair ProfessorXi-Ren  Cao
Living reference work entry
DOI: https://doi.org/10.1007/978-1-4471-5102-9_57-1


We introduce the theories and methodologies that utilize the special features of discrete event dynamic systems (DEDSs) for perturbation analysis (PA) and optimization of steady-state performance. Such theories and methodologies usually take different perspectives from the traditional optimization approaches and therefore may lead to new insights and efficient algorithms. The topic discussed includes the gradient-based optimization for systems with continuous parameters and the direct-comparison-based optimization for systems with discrete policies, which is an alternative to dynamic programming and may apply when the latter fails. Furthermore, these new insights can also be applied to continuous-time and continuous-state dynamic systems, leading to a new paradigm of optimal control.

This is a preview of subscription content, log in to check access.


  1. Baxter J, Bartlett PL (2001) Infinite-horizon policy-gradient estimation. J Artif Intell Res 15: 319–350MATHMathSciNetGoogle Scholar
  2. Cao XR (1985) Convergence of parameter sensitivity estimates in a stochastic experiment. IEEE Trans Autom Control 30:834–843CrossRefGoogle Scholar
  3. Cao XR (2005) A basic formula for online policy gradient algorithms. IEEE Trans Autom Control 50(5):696–699CrossRefGoogle Scholar
  4. Cao XR (2007) Stochastic learning and optimization – a sensitivity-based approach. Springer, New YorkCrossRefMATHGoogle Scholar
  5. Cao XR, Wan YW (1998) Algorithms for sensitivity analysis of Markov systems through potentials and perturbation realization. IEEE Trans Control Syst Technol 6:482–494CrossRefGoogle Scholar
  6. Cao XR, Wan XW (2013) Analysis of non-linear behavior – a sensitivity-based approach. submittedGoogle Scholar
  7. Cao XR, Wang DX, Lu T, Xu YF (2011) Stochastic control via direct comparison. Discret Event Dyn Syst Theory Appl 21:11–38CrossRefMATHMathSciNetGoogle Scholar
  8. Cassandras CG, Lafortune S (1999) Introduction to discrete event systems. Kluwer Academic Publishers, BostonCrossRefMATHGoogle Scholar
  9. Fang HT, Cao XR (2004) Potential-based on-line policy iteration algorithms for Markov decision processes. IEEE Trans Autom Control 49:493–505CrossRefMathSciNetGoogle Scholar
  10. Fu MC, Hu JQ (1997) Conditional Monte Carlo: gradient estimation and optimization applications. Kluwer Academic Publishers, BostonCrossRefMATHGoogle Scholar
  11. Glasserman P (1991) Gradient estimation via perturbation analysis. Kluwer Academic Publishers, BostonMATHGoogle Scholar
  12. Heidelberger P, Cao XR, Zazanis M, Suri R (1988) Convergence properties of infinitesimal perturbation analysis estimates. Manag Sci 34:1281-1302CrossRefMATHMathSciNetGoogle Scholar
  13. Ho YC, Cao XR (1983) Perturbation analysis and optimization of queueing networks. J Optim Theory Appl 40:559–582CrossRefMATHMathSciNetGoogle Scholar
  14. Ho YC, Cao XR (1991) Perturbation analysis of discrete-event dynamic systems. Kluwer Academic Publisher, BostonCrossRefMATHGoogle Scholar
  15. Marbach P, Tsitsiklis TN (2001) Simulation-based optimization of Markov reward processes. IEEE Trans Autom Control 46:191–209CrossRefMATHMathSciNetGoogle Scholar
  16. Puterman ML (1994) Markov decision processes: discrete stochastic dynamic programming. Wiley, New YorkCrossRefMATHGoogle Scholar
  17. Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat 22:400–407CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Chair ProfessorXi-Ren  Cao
    • 1
  1. 1.Department of Finance and Department of AutomationShanghai Jiao Tong UniversityShanghaiChina