Experimental Results for Gradient Estimation and Optimization of a Markov Chain in Steady-State
- 57 Downloads
Infinitesimal perturbation analysis (IPA) and the likelihood ratio (LR) method have drawn tots of attention recently, as ways of estimating the gradient of a performance measure with respect to continuous parameters in dynamic stochastic systems. In this paper, we experiment with the use of these estimators in stochastic approximation algorithms, to perform so-called “single-run optimizations” of steady-state systems, as suggested in . We also compare them to finite-difference estimators, with and without common random numbers. In most cases, the simulation length must be increased from iteration to iteration, otherwise the algorithm converges to the wrong value. We have performed extensive numerical experiments with a simple M/M/1 queue. We state convergence results, but do not give the proofs. The proofs are given in .
KeywordsService Time Busy Period Stochastic Approximation Gradient Estimator Dynamic Stochastic System
Unable to display preview. Download preview PDF.
- Aleksandrov, V. M., V. J. Sysoyev and V. V. Shemeneva, “Stochastic Optimization”, Engineering Cybernetics, 5 (1963), 11–16.Google Scholar
- Asmussen, S., Applied Probability and Queues, Wiley, 1987.Google Scholar
- Giroux, N. “Optimisation Stochastique de Type Monte Carlo”, Mémoire de maîtrise, dépt. d’informatique, Univ. Laval, jan. 1989.Google Scholar
- Glynn, P. W. “Likelihood Ratio Gradient Estimation: an Overview”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 366–375.Google Scholar
- Ho, Y.-C., “Performance Evaluation and Perturbation Analysis of Discrete Event Dynamic Systems”, IEEE Transactions of Automatic Control, AC-32, 7 (1987), 563–572.Google Scholar
- Kushner, H. J. and Clark, D. S., Stochastic Approximation Methods for Constrained and Unconstrained Systems, Springer-Verlag, Applied Math. Sciences, vol. 26, 1978.Google Scholar
- L’Ecuyer, P. and Glynn, P. W., “A Control Varíate Scheme for Likelihood Ratio Gradient Estimation”, In preparation (1990).Google Scholar
- L’Ecuyer, P., Giroux, N., and Glynn, P. W., “Stochastic Optimization by Simulation: Convergence Proofs and Experimental Results for the GI/G/1 Queue”, manuscript, 1990.Google Scholar
- Meketon, M. S., “Optimization in Simulation: a Survey of Recent Results”, Proceedings of the Winter Simulation Conference 1987, IEEE Press (1987), 58–67.Google Scholar
- [IS]Rubinstein, R. Y., Monte-Carlo Optimization, Simulation and Sensitivity of Queueing Networks, Wiley, 1986.Google Scholar