Experimental Results for Gradient Estimation and Optimization of a Markov Chain in Steady-State

  • Pierre L’Ecuyer
  • Nataly Giroux
  • Peter W. Glynn
Conference paper
Part of the Lecture Notes in Economics and Mathematical Systems book series (LNE, volume 374)


Infinitesimal perturbation analysis (IPA) and the likelihood ratio (LR) method have drawn tots of attention recently, as ways of estimating the gradient of a performance measure with respect to continuous parameters in dynamic stochastic systems. In this paper, we experiment with the use of these estimators in stochastic approximation algorithms, to perform so-called “single-run optimizations” of steady-state systems, as suggested in [23]. We also compare them to finite-difference estimators, with and without common random numbers. In most cases, the simulation length must be increased from iteration to iteration, otherwise the algorithm converges to the wrong value. We have performed extensive numerical experiments with a simple M/M/1 queue. We state convergence results, but do not give the proofs. The proofs are given in [14].


Service Time Busy Period Stochastic Approximation Gradient Estimator Dynamic Stochastic System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Aleksandrov, V. M., V. J. Sysoyev and V. V. Shemeneva, “Stochastic Optimization”, Engineering Cybernetics, 5 (1963), 11–16.Google Scholar
  2. [2]
    Asmussen, S., Applied Probability and Queues, Wiley, 1987.Google Scholar
  3. [3]
    Bratley, P., B. L. Fox, and L. E. Schrage, A Guide to Simulation, Springer-Verlag, New York, Second Edition, 1987.CrossRefGoogle Scholar
  4. [4]
    Giroux, N. “Optimisation Stochastique de Type Monte Carlo”, Mémoire de maîtrise, dépt. d’informatique, Univ. Laval, jan. 1989.Google Scholar
  5. [5]
    Glynn, P. W. “Likelihood Ratio Gradient Estimation: an Overview”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 366–375.Google Scholar
  6. [6]
    Glynn, P. W. “Optimization of Stochastic Systems Via Simulation”, Proceedings of the Winter Simulation Conference 1989, IEEE Press (1989), 90–105.CrossRefGoogle Scholar
  7. [7]
    Glynn, P. W. “Likelihood Ratio Gradient Estimation for Stochastic Systems”, Communications of the ACM, 33, 10 (1990), 75–84.CrossRefGoogle Scholar
  8. [8]
    Heidelberger, P., Cao, X.-R., Zazanis, M. A. and Suri, R., “Convergence Properties of Infinitesimal Perturbation Analysis Estimates”, Management Science, 34, 11 (1989), 1281–1302.CrossRefGoogle Scholar
  9. [9]
    Ho, Y.-C., “Performance Evaluation and Perturbation Analysis of Discrete Event Dynamic Systems”, IEEE Transactions of Automatic Control, AC-32, 7 (1987), 563–572.Google Scholar
  10. [10]
    Kushner, H. J. and Clark, D. S., Stochastic Approximation Methods for Constrained and Unconstrained Systems, Springer-Verlag, Applied Math. Sciences, vol. 26, 1978.Google Scholar
  11. [11]
    Kushner, H. J. and Shwartz, A., “An Invariant Measure Approach to the Convergence of Stochastic Approximations with State Dependent Noise”, SIAM J. on Control and Optim., 22, 1 (1984), 13–24.CrossRefGoogle Scholar
  12. [12]
    L’Ecuyer, P., “A Unified View of the IPA, SF, and LR Gradient Estimation Techniques”, Management Science, 36, 11 (1990), 1364–1383.CrossRefGoogle Scholar
  13. [13]
    L’Ecuyer, P. and Glynn, P. W., “A Control Varíate Scheme for Likelihood Ratio Gradient Estimation”, In preparation (1990).Google Scholar
  14. [14]
    L’Ecuyer, P., Giroux, N., and Glynn, P. W., “Stochastic Optimization by Simulation: Convergence Proofs and Experimental Results for the GI/G/1 Queue”, manuscript, 1990.Google Scholar
  15. [15]
    Meketon, M. S., “Optimization in Simulation: a Survey of Recent Results”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 58–67.Google Scholar
  16. [16]
    Pflug, G. Ch., “On-line Optimization of Simulated Markovian Processes”, Math. of Oper. Res., 15, 3 (1990), 381–395.CrossRefGoogle Scholar
  17. [17]
    Reiman, M. I. and Weiss, A., “Sensitivity Analysis for Simulation via Likelihood Ratios”, Operalions Research, 37, 5 (1989), 830–844.CrossRefGoogle Scholar
  18. [IS]
    Rubinstein, R. Y., Monte-Carlo Optimization, Simulation and Sensitivity of Queueing Networks, Wiley, 1986.Google Scholar
  19. [19]
    Rubinstein, R. Y., “The Score Function Approach for Sensitivity Analysis of Computer Simulation Models”, Math, and Computers in Simulation, 28 (1986), 351–379.CrossRefGoogle Scholar
  20. [20]
    Rubinstein, R. Y., “Sensitivity Analysis and Performance Extrapolation for Computer Simulation Models”, Operations Research, 37, 1 (1989), 72–81.CrossRefGoogle Scholar
  21. [21]
    Suri, R., “Infinitesimal Perturbation Analysis of General Discrete Event Dynamic Systems”, J. of the ACM, 34, 3 (1987), 686–717.CrossRefGoogle Scholar
  22. [22]
    Suri, R., “Perturbation Analysis: The State of the Art and Research Issues Explained via the GI/G/1 Queue”, Proceedings of the IEEE, 77, 1 (1989), 114–137.CrossRefGoogle Scholar
  23. [23]
    Suri, R. and Leung, Y. T., “Single Run Optimization of Discrete Event Simulations—An Empirical Study Using the M/M/1 Queue”, IIE Transactions, 21, 1 (1989), 35–49.CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1992

Authors and Affiliations

  • Pierre L’Ecuyer
    • 1
  • Nataly Giroux
    • 2
  • Peter W. Glynn
    • 3
  1. 1.Département d’I.R.O.Université de MontréalMontréalCanada
  2. 2.Département d’informatiqueUniversité LavalSte-FoyCanada
  3. 3.Operations Research DepartmentStanford UniversityStanfordUSA

Personalised recommendations