Abstract
Infinitesimal perturbation analysis (IPA) and the likelihood ratio (LR) method have drawn tots of attention recently, as ways of estimating the gradient of a performance measure with respect to continuous parameters in dynamic stochastic systems. In this paper, we experiment with the use of these estimators in stochastic approximation algorithms, to perform so-called “single-run optimizations” of steady-state systems, as suggested in [23]. We also compare them to finite-difference estimators, with and without common random numbers. In most cases, the simulation length must be increased from iteration to iteration, otherwise the algorithm converges to the wrong value. We have performed extensive numerical experiments with a simple M/M/1 queue. We state convergence results, but do not give the proofs. The proofs are given in [14].
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Aleksandrov, V. M., V. J. Sysoyev and V. V. Shemeneva, “Stochastic Optimization”, Engineering Cybernetics, 5 (1963), 11–16.
Asmussen, S., Applied Probability and Queues, Wiley, 1987.
Bratley, P., B. L. Fox, and L. E. Schrage, A Guide to Simulation, Springer-Verlag, New York, Second Edition, 1987.
Giroux, N. “Optimisation Stochastique de Type Monte Carlo”, Mémoire de maîtrise, dépt. d’informatique, Univ. Laval, jan. 1989.
Glynn, P. W. “Likelihood Ratio Gradient Estimation: an Overview”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 366–375.
Glynn, P. W. “Optimization of Stochastic Systems Via Simulation”, Proceedings of the Winter Simulation Conference 1989, IEEE Press (1989), 90–105.
Glynn, P. W. “Likelihood Ratio Gradient Estimation for Stochastic Systems”, Communications of the ACM, 33, 10 (1990), 75–84.
Heidelberger, P., Cao, X.-R., Zazanis, M. A. and Suri, R., “Convergence Properties of Infinitesimal Perturbation Analysis Estimates”, Management Science, 34, 11 (1989), 1281–1302.
Ho, Y.-C., “Performance Evaluation and Perturbation Analysis of Discrete Event Dynamic Systems”, IEEE Transactions of Automatic Control, AC-32, 7 (1987), 563–572.
Kushner, H. J. and Clark, D. S., Stochastic Approximation Methods for Constrained and Unconstrained Systems, Springer-Verlag, Applied Math. Sciences, vol. 26, 1978.
Kushner, H. J. and Shwartz, A., “An Invariant Measure Approach to the Convergence of Stochastic Approximations with State Dependent Noise”, SIAM J. on Control and Optim., 22, 1 (1984), 13–24.
L’Ecuyer, P., “A Unified View of the IPA, SF, and LR Gradient Estimation Techniques”, Management Science, 36, 11 (1990), 1364–1383.
L’Ecuyer, P. and Glynn, P. W., “A Control Varíate Scheme for Likelihood Ratio Gradient Estimation”, In preparation (1990).
L’Ecuyer, P., Giroux, N., and Glynn, P. W., “Stochastic Optimization by Simulation: Convergence Proofs and Experimental Results for the GI/G/1 Queue”, manuscript, 1990.
Meketon, M. S., “Optimization in Simulation: a Survey of Recent Results”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 58–67.
Pflug, G. Ch., “On-line Optimization of Simulated Markovian Processes”, Math. of Oper. Res., 15, 3 (1990), 381–395.
Reiman, M. I. and Weiss, A., “Sensitivity Analysis for Simulation via Likelihood Ratios”, Operalions Research, 37, 5 (1989), 830–844.
Rubinstein, R. Y., Monte-Carlo Optimization, Simulation and Sensitivity of Queueing Networks, Wiley, 1986.
Rubinstein, R. Y., “The Score Function Approach for Sensitivity Analysis of Computer Simulation Models”, Math, and Computers in Simulation, 28 (1986), 351–379.
Rubinstein, R. Y., “Sensitivity Analysis and Performance Extrapolation for Computer Simulation Models”, Operations Research, 37, 1 (1989), 72–81.
Suri, R., “Infinitesimal Perturbation Analysis of General Discrete Event Dynamic Systems”, J. of the ACM, 34, 3 (1987), 686–717.
Suri, R., “Perturbation Analysis: The State of the Art and Research Issues Explained via the GI/G/1 Queue”, Proceedings of the IEEE, 77, 1 (1989), 114–137.
Suri, R. and Leung, Y. T., “Single Run Optimization of Discrete Event Simulations—An Empirical Study Using the M/M/1 Queue”, IIE Transactions, 21, 1 (1989), 35–49.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1992 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
L’Ecuyer, P., Giroux, N., Glynn, P.W. (1992). Experimental Results for Gradient Estimation and Optimization of a Markov Chain in Steady-State. In: Pflug, G., Dieter, U. (eds) Simulation and Optimization. Lecture Notes in Economics and Mathematical Systems, vol 374. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-48914-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-48914-3_2
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-54980-2
Online ISBN: 978-3-642-48914-3
eBook Packages: Springer Book Archive