Advertisement

Abstract

Perturbation analysis (PA) is the core of the gradient-based (or policy gradient) learning and optimization approach. The basic principle of PA is that the derivative of a system’s performance with respect to a parameter of the system can be decomposed into the sum of many small building blocks, each of which measures the effect of a single perturbation on the system’s performance, and this effect can be estimated on a sample path of the system.

Keywords

Markov Chain Service Time Sample Path Busy Period Perturbation Analysis 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 51.
    X. R. Cao, Realization Probabilities: The Dynamics of Queueing Systems, Springer-Verlag, New York, 1994.MATHCrossRefGoogle Scholar
  2. 72.
    C. G. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Kluwer Academic Publishers, Boston, 1999.MATHGoogle Scholar
  3. 107.
    M. C. Fu and J. Q. Hu, Conditional Monte Carlo: Gradient Estimation and Optimization Applications, Kluwer Academic Publishers, Boston, 1997.MATHGoogle Scholar
  4. 142.
    Y. C. Ho and X. R. Cao, Perturbation Analysis of Discrete-Event Dynamic Systems, Kluwer Academic Publisher, Boston, 1991.MATHGoogle Scholar
  5. 62.
    X. R. Cao and H. F. Chen, “Perturbation Realization, Potentials and Sensitivity Analysis of Markov Processes,” IEEE Transactions on Automatic Control, Vol. 42, 1382-1393, 1997.MATHCrossRefMathSciNetGoogle Scholar
  6. 70.
    X. R. Cao, X. M. Yuan, and L. Qiu, “A Single Sample Path-Based Performance Sensitivity Formula for Markov Chains,” IEEE Transactions on Automatic Control, Vol. 41, 1814-1817, 1996.MATHCrossRefMathSciNetGoogle Scholar
  7. 87.
    E. Çinlar, Introduction to Stochastic Processes, Prentice Hall, Englewood Cliffs, New Jersey, 1975.Google Scholar
  8. 202.
    C. D. Meyer, “The Role of the Group Generalized Inverse in the Theory of Finite Markov Chains,” SIAM Review, Vol. 17, 443-464, 1975.MATHCrossRefMathSciNetGoogle Scholar
  9. 153.
    G. Hooghiemstra, M. Keane, and S. Van De Ree, “Power Series for Stationary Distribution of Coupled Processor Models,” SIAM Journal of Applied Mathematics, Vol. 48, 1159-1166, 1988.MATHCrossRefGoogle Scholar
  10. 267.
    Y. Zhu and H. Li, “The MacLaurin Expansion for a G/G/1 Queue with Markov-Modulated Arrivals and Services,” Queueing Systems: Theory and Applications, Vol. 14, 125-134, 1993.MATHCrossRefMathSciNetGoogle Scholar
  11. 120.
    W. B. Gong and J. Q. Hu, “The Maclaurin Series for the GI/G/1 Queue,” Journal of Applied Probability, Vol. 29, 176-184, 1992.MATHCrossRefMathSciNetGoogle Scholar
  12. 29.
    J. P. C. Blanc, “A Numerical Approach to Cyclic-Server Queueing Models,” Queueing Systems: Theory and Applications, Vol. 6, 173-188, 1990.MATHCrossRefMathSciNetGoogle Scholar
  13. 158.
    J. Q. Hu, S. Nananukul, and W. B. Gong, “A New Approach to (s, S) Inventory Systems,” Journal of Applied Probability, Vol. 30, 898-912, 1993.MATHCrossRefMathSciNetGoogle Scholar
  14. 162.
    T. Kailath, Linear Systems, Prentice Hall, Englewood Cliffs, New Jersey, 1980.MATHGoogle Scholar
  15. 174.
    P. Lancaster and M. Tismenetsky, The Theory of Matrices with Applications, Second Edition, Academic Press, Orlando, 1985.MATHGoogle Scholar
  16. 57.
    X. R. Cao, “Semi-Markov Decision Problems and Performance Sensitivity Analysis,” IEEE Transactions on Automatic Control, Vol. 48, 758-769, 2003.CrossRefGoogle Scholar
  17. 169.
    L. Kleinrock, Queueing Systems, Volume I: Theory, John Wiley & Sons, New York, 1975.Google Scholar
  18. 243.
    H. C. Tijms, Stochastic Models: An Algorithmic Approach, John Wiley & Sons, New York, 1994.MATHGoogle Scholar
  19. 45.
    X. R. Cao, “Realization Probability in Closed Jackson Queueing Networks and Its Application,” Advances in Applied Probability, Vol. 19, 708-738, 1987.MATHCrossRefMathSciNetGoogle Scholar
  20. 49.
    X. R. Cao, “Realization Probability and Throughput Sensitivity in a Closed Jackson Network,” Journal of Applied Probability, Vol. 26, 615-624, 1989.MATHCrossRefMathSciNetGoogle Scholar
  21. 50.
    X. R. Cao, “Realization Factors and Sensitivity Analysis of Queueing Networks with State-Dependent Service Rates,” Advances in Applied Probability, Vol. 22, 178-210, 1990.MATHCrossRefMathSciNetGoogle Scholar
  22. 113.
    P. Glasserman, “The Limiting Value of Derivative Estimates Based on Perturbation Analysis,” Communications in Statistics: Stochastic Models, Vol. 6, 229-257, 1990.MATHCrossRefMathSciNetGoogle Scholar
  23. 141.
    Y. C. Ho and X. R. Cao, “Perturbation Analysis and Optimization of Queueing Networks,” Journal of Optimization Theory and Applications, Vol. 40, 559-582, 1983.MATHCrossRefMathSciNetGoogle Scholar
  24. 42.
    X. R. Cao, “Convergence of Parameter Sensitivity Estimates in a Stochastic Experiment,” IEEE Transactions on Automatic Control, Vol. 30, 845-853, 1985.MATHCrossRefGoogle Scholar
  25. 46.
    X. R. Cao, “A Sample Performance Function of Jackson Queueing Networks,” Operations Research, Vol. 36, 128-136, 1988.MATHCrossRefMathSciNetGoogle Scholar
  26. 105.
    M. C. Fu and J. Q. Hu, “Smoothed Perturbation Analysis Derivative Estimation for Markov Chains,” Operations Research Letters, Vol. 15, 241-251, 1994.MATHCrossRefMathSciNetGoogle Scholar
  27. 114.
    P. Glasserman and W. B. Gong, “Smoothed Perturbation Analysis for A Class of Discrete Event System,” IEEE Transactions on Automatic Control, Vol. 35, 1218-1230, 1990.MATHCrossRefMathSciNetGoogle Scholar
  28. 119.
    W. B. Gong and Y. C. Ho, “Smoothed Perturbation Analysis for Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 32, 858-866, 1987.MATHCrossRefMathSciNetGoogle Scholar
  29. 143.
    Y. C. Ho, X. R. Cao, and C. Cassandras, “Infinitesimal and Finite Perturbation Analysis for Queueing Networks,” Automatica, Vol. 19, 439-445, 1983.MATHCrossRefGoogle Scholar
  30. 33.
    P. Bremaud, “Maximal Coupling and Rare Perturbation Sensitivity Analysis,” Queueing Systems: Theory and Applications, Vol. 11, 307-333, 1992.MATHCrossRefMathSciNetGoogle Scholar
  31. 34.
    P. Bremaud and W. B. Gong, “Derivatives of Likelihood Ratios and Smoothed Perturbation Analysis for Routing Problem,” ACM Transactions on Modeling and Computer Simulation, Vol. 3, 134-161, 1993.MATHCrossRefGoogle Scholar
  32. 36.
    P. Bremaud and L. Massoulie, “Maximal Coupling and Rare Perturbation Analysis with a Random Horizon,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 5, 319-342, 1995.MATHCrossRefGoogle Scholar
  33. 37.
    P. Bremaud and F. J. Vazquez-Abad, “On the Pathwise Computation of Derivatives with Respect to the Rate of A Point Process: The Phantom RPA Method,” Queueing Systems: Theory and Applications, Vol. 10, 249-269, 1992.MATHCrossRefMathSciNetGoogle Scholar
  34. 73.
    C. G. Cassandras and S. G. Strickland, “On-Line Sensitivity Analysis of Markov Chains,” IEEE Transactions on Automatic Control, Vol. 34, 76-86, 1989.MATHCrossRefMathSciNetGoogle Scholar
  35. 94.
    L. Dai, and Y. C. Ho, “Structural Infinitesimal Perturbation Analysis (SIPA) for Derivative Estimation of Discrete Event Dynamic Systems,” IEEE Transactions on Automatic Control, Vol. 40, 1154-1166, 1995.MATHCrossRefMathSciNetGoogle Scholar
  36. 102.
    M. Freimer and L. Schruben, “Graphical Representation of IPA Estimation,” Proceedings of the 2001 Winter Simulation Conference, Arlington, Virginia, U.S.A, Vol. 1, 422-427, December 2001.Google Scholar
  37. 106.
    M. C. Fu and J. Q. Hu, “Efficient Design and Sensitivity Analysis of Control Charts Using Monte Carlo Simulation,” Management Science, Vol. 45, 395-413, 1999.CrossRefGoogle Scholar
  38. 108.
    A. A. Gaivoronski, L. Y. Shi, and R. S. Sreenivas, “Augmented Infinitesimal Perturbation Analysis: An Alternate Explanation,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 2, 121-138, 1992.MATHCrossRefGoogle Scholar
  39. 128.
    B. Heidergott, “Infinitesimal Perturbation Analysis for Queueing Networks with General Service Time Distributions,” Queueing Systems: Theory and Applications, Vol. 31, 43-58, 1999.MATHCrossRefMathSciNetGoogle Scholar
  40. 147.
    Y. C. Ho and S. Li, “Extensions of Infinitesimal Perturbation Analysis,” IEEE Transactions on Automatic Control, Vol. 33, 427-438, 1988.MATHCrossRefMathSciNetGoogle Scholar
  41. 156.
    J. Q. Hu, “Convexity of Sample Path Performance and Strong Consistency of Infinitesimal Perturbation Analysis Estimates,” IEEE Transactions on Automatic Control, Vol. 37, 258-262, 1992.CrossRefGoogle Scholar
  42. 185.
    Q. L. Li, and L. M. Liu, “An Algorithmic Approach on Sensitivity Analysis of Perturbed QBD Processes,” Queueing Systems, Vol. 48, 365-397, 2004.MATHCrossRefMathSciNetGoogle Scholar
  43. 214.
    E. L. Plambeck, B. R. Fu, S. M. Robinson, and R. Suri, “Sample-Path Optimization of Convex Stochastic Performance Functions,” Mathematical Programming, Vol. 75, 137-176, 1996.MathSciNetGoogle Scholar
  44. 232.
    R. Suri, “Infinitesimal Perturbation Analysis for General Discrete Event Systems,” Journal of the ACM, Vol. 34, 686-717, 1987.CrossRefMathSciNetGoogle Scholar
  45. 241.
    Q. Y. Tang, P. L’Ecuyer, and H. F. Chen, “Central Limit Theorems for Stochastic Optimization Algorithms Using Infinitesimal Perturbation Analysis,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 10, 5-32, 2000.MATHCrossRefMathSciNetGoogle Scholar
  46. 245.
    S. Uryasev, “Analytic Perturbation Analysis for DEDS with Discontinuous Sample Path Functions,” Communications in Statistics: Stochastic Models, Vol. 13, 457-490, 1997.MATHCrossRefMathSciNetGoogle Scholar
  47. 247.
    F. J. Vazquez-Abad and J. H. Kushner, “Estimation of the Derivative of a Stationary Measure with Respect to a Control Parameter,” Journal of Applied Probability, Vol. 29, 343-352, 1992.MATHCrossRefMathSciNetGoogle Scholar
  48. 251.
    Y. Wardi, M. W. McKinnon, and R. Schuckle, “On Perturbation Analysis of Queueing Networks with Finitely Supported Service Time Distributions,” IEEE Transactions on Automatic Control, Vol. 36, 863-867, 1991.CrossRefMathSciNetGoogle Scholar
  49. 112.
    P. Glasserman, Gradient Estimation Via Perturbation Analysis, Kluwer Academic Publishers, Boston, 1991.MATHGoogle Scholar
  50. 74.
    C. G. Cassandras, G. Sun, C. G. Panayiotou, and Y. Wardi, “Perturbation Analysis and Control of Two-Class Stochastic Fluid Models for Communication Networks,” IEEE Transactions on Automatic Control, Vol. 48, 770-782, 2003.CrossRefMathSciNetGoogle Scholar
  51. 75.
    C. G. Cassandras, Y. Wardi, B. Melamed, G. Sun, and C. G. Panayiotou, “Perturbation Analysis for Online Control and Optimization of Stochastic Fluid Models,” IEEE Transactions on Automatic Control, Vol. 47, 1234-1248, 2002.CrossRefMathSciNetGoogle Scholar
  52. 189.
    Y. Liu and W. B. Gong, “Perturbation Analysis for Stochastic Fluid Queueing Systems,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 12, 391-416, 2002.MATHCrossRefMathSciNetGoogle Scholar
  53. 210.
    C. Panayiotou and C. G. Cassandras, “Infinitesimal Perturbation Analysis and Optimization for Make-to-Stock Manufacturing Systems Based on Stochastic Fluid Models,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 16, 109-142, 2006.MATHCrossRefMathSciNetGoogle Scholar
  54. 211.
    C. Panayiotou, C. G. Cassandras, G. Sun, and Y. Wardi, “Control of Communication Networks Using Infinitesimal Perturbation Analysis of Stochastic Fluid Models,” Advances in Communication Control Networks, Lecture Notes in Control and Information Sciences, Vol. 308, 1-26, 2004.MathSciNetGoogle Scholar
  55. 231.
    G. Sun, C. G. Cassandras, and C. G. Panayiotou, “Perturbation Analysis of Multiclass Stochastic Fluid Models,” Discrete Event Dynamic Systems: Theory and Applications, Vol. 14, 267-307, 2004.MATHCrossRefMathSciNetGoogle Scholar
  56. 252.
    Y. Wardi, B. Melamed, C. G. Cassandras, and C. G. Panayiotou, “Online IPA Gradient Estimators in Stochastic Continuous Fluid Models,” Journal of Optimization Theory and Applications, Vol. 115, 369-405, 2002.MATHCrossRefMathSciNetGoogle Scholar
  57. 262.
    H. Yu and C. G. Cassandras, “Perturbation Analysis of Feedback-Controlled Stochastic Flow Systems,” IEEE Transactions on Automatic Control, Vol. 49, 1317-1332, 2004.CrossRefMathSciNetGoogle Scholar
  58. 263.
    H. Yu and C. G. Cassandras, “Perturbation Analysis of Communication Networks with Feedback Control Using Stochastic Hybrid Models,” Nonlinear Analysis - Theory Methods and Applications, Vol. 65, 1251-1280, 2006.MATHCrossRefMathSciNetGoogle Scholar
  59. 115.
    P. W. Glynn, “Regenerative Structure of Markov Chains Simulated Via Common Random Numbers,” Operations Research Letters, Vol. 4, 49-53, 1985.MATHCrossRefMathSciNetGoogle Scholar
  60. 116.
    P. W. Glynn, “Likelihood Ratio Gradient Estimation: An Overview,” Proceedings of the 1987 Winter Simulation Conference, Atlanta, Georgia, U.S.A, 366-375, December 1987.Google Scholar
  61. 117.
    P. W. Glynn, “Optimization of Stochastic Systems Via Simulation,” Proceedings of the 1989 Winter Simulation Conference, Washington, U.S.A, 90-105, December 1989.Google Scholar
  62. 118.
    P. W. Glynn and P. L’Ecuyer, “Likelihood Ratio Gradient Estimation for Stochastic Recursions,” Advances in Applied Probability, Vol. 27, 1019-1053, 1995.MATHCrossRefMathSciNetGoogle Scholar
  63. 130.
    B. Heidergott and X. R. Cao, “A Note on the Relation Between Weak Derivatives and Perturbation Realization,” IEEE Transactions on Automatic Control, Vol. 47, 1112-1115, 2002.MathSciNetGoogle Scholar
  64. 176.
    P. L’Ecuyer, “A Unified View of the IPA, SF, and LR Gradient Estimation Techniques,” Management Science, Vol. 36, 1364-1383, 1990.MATHCrossRefGoogle Scholar
  65. 177.
    P. L’Ecuyer, “Convergence Rate for Steady-State Derivative Estimators,” Annals of Operations Research, Vol. 39, 121-136, 1992.MATHCrossRefMathSciNetGoogle Scholar
  66. 178.
    P. L’Ecuyer, “On the Interchange of Derivative and Expectation for Likelihood Ratio Derivative Estimators,” Management Science, Vol. 41, 738-748, 1995.MATHCrossRefGoogle Scholar
  67. 179.
    P. L’Ecuyer and G. Perron, “On the Convergence Rates of IPA and FDC Derivative Estimators,” Operations Research, Vol. 42, 643-656, 1994.MATHCrossRefMathSciNetGoogle Scholar
  68. 205.
    M. K. Nakayama and P. Shahabuddin, “Likelihood Ratio Derivative Estimation for Finite-Time Performance Measures in Generalized Semi-Markov Processes,” Management Science, Vol. 44, 1426-1441, 1998.MATHCrossRefGoogle Scholar
  69. 217.
    M. I. Reiman and A. Weiss, “Sensitivity Analysis for Simulations Via Likelihood Ratios,” Operations Research, Vol. 37, 830-844, 1989.MATHCrossRefMathSciNetGoogle Scholar
  70. 221.
    R. V. Rubinstein, Monte Carlo Optimization, Simulation, and Sensitivity Analysis of Queueing Networks, John Wiley & Sons, New York, 1986.Google Scholar
  71. 222.
    R. V. Rubinstein and A. Shapiro, Sensitivity Analysis and Stochastic Optimization by the Score Function Method, John Wiley & Sons, New York, 1993.MATHGoogle Scholar
  72. 44.
    X. R. Cao, “Sensitivity Estimates Based on One Realization of a Stochastic System,” Journal of Statistical Computation and Simulation, Vol. 27, 211-232, 1987.MATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Hong Kong University of Science and TechnologyKowloonHong Kong

Personalised recommendations