Monte Carlo Minimization for One Step Ahead Sequential Control

  • Li-Shya Chen
  • Seymour Geisser
  • Charles J. Geyer
Part of the The IMA Volumes in Mathematics and its Applications book series (IMA, volume 114)


Sequential updating solutions to optimal control problems are typically not available in closed form. We present a method of Monte Carlo calculation of sequential one step ahead updating solutions by simulating realizations from the predictive distribution of model parameters and approximating the predictive expected loss (PEL) by averaging over the simulations. The minimizer of the approximate PEL is taken to approximate the exact PEL. The approximate minimizer is shown to converge to the exact minimizer under mere lower semi-continuity of the loss function and is shown to be asymptotically normal under stronger conditions. Examples are given from the problem of controlling a linear regression model with autoregressive responses (ARX) or with autocorrelated errors and from dynamic input-output models using a variety of loss functions.


Markov Chain Posterior Distribution Loss Function Linear Regression Model Posterior Density 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Attouch, H. (1984). Variational Convergence for Functions and Operators. Boston: Pitman.zbMATHGoogle Scholar
  2. Berry, D.A. and Fristedt, B. (1985). Bandit Problems: Sequential Allocation of Experiments. London, New York: Chapman and Hall.zbMATHGoogle Scholar
  3. Bertsekas, D.P. (1976). Dynamic Programming and Stochastic Control. New York: Academic Press.zbMATHGoogle Scholar
  4. Bertsekas, D.P. and Shreve S.E. (1978). Stochastic Optimal Control: The Discrete Time Case. New York: Academic Press.zbMATHGoogle Scholar
  5. Box, G.P. and Jenkins, G.M. (1976). Time Series Analysis: Forecasting and Control. San Francisco: Holden-Day.zbMATHGoogle Scholar
  6. Chan, K.S. and Geyer, C.J. (1994). Discussion of the paper by Tierney. Ann. Statist. 22, 1747–1758.CrossRefGoogle Scholar
  7. Chib, S. (1993). Bayesian regression with autoregressive errors: A Gibbs sampling approach. Journal of Econometrics 58, 275–294.MathSciNetzbMATHCrossRefGoogle Scholar
  8. Dupačová, J. and Wets, R. (1988). Asymptotic behavior of statistical estimators and of optimal solutions of stochastic optimization problems. Ann. Statist. 16, 1517–1549.MathSciNetzbMATHCrossRefGoogle Scholar
  9. Geyer, C.J. (1992). Practical Markov chain Monte Carlo. Statistical Science 7, 473–483.CrossRefGoogle Scholar
  10. Geyer, C.J. (1994a). On the convergence of Monte Carlo Maximum Likelihood calculations. Journal of the Royal Statistical Society, Ser. B. 56, 261–274.MathSciNetzbMATHGoogle Scholar
  11. Geyer, C.J. (1994b). On the asymptotics of constrained M-estimation. Ann. Statist. 22, 1993–2010.MathSciNetzbMATHCrossRefGoogle Scholar
  12. Geyer, C.J. and Thompson, E.A. (1992). Constrained Monte Carlo maximum likelihood for dependent data (with discussion). Journal of the Royal Statistical Society, Ser. B 54, 657–699.MathSciNetGoogle Scholar
  13. Hess, C. (1996). EPI-convergence of sequences of normal integrands and strong consistency of maximum likelihood estimators. Ann. Statist., 24, 1298–1315.MathSciNetzbMATHCrossRefGoogle Scholar
  14. Jennrich, R.I. (1969). Asymptotic properties of non-linear least squares estimators. Ann. Math. Statist., 40, 633–643.MathSciNetzbMATHCrossRefGoogle Scholar
  15. Kall, P. (1986). Approximation to optimization problems: an elementary review. Math. Oper. Res. 11, 9–18.MathSciNetzbMATHCrossRefGoogle Scholar
  16. Meyn, S.P. and Tweedie, R.L. (1993). Markov Chains and Stochastic Stability. London, New York: Springer-Verlag.zbMATHCrossRefGoogle Scholar
  17. Rockafellar, R.T. (1970). Convex Analysis. Princeton: Princeton University Press.zbMATHGoogle Scholar
  18. Tierney, L. (1994). Markov chains for exploring posterior distributions (with discussion). Ann. Statist. 22, 1701–1762.MathSciNetCrossRefGoogle Scholar
  19. Whittle, P. (1982). Optimization Over Time: Dynamic Programming and Stochastic Control. Chichester, New York: Wiley.zbMATHGoogle Scholar
  20. Zellner, A. (1971). An Introduction to Bayesian Inference in Economics. New York: Wiley.Google Scholar

Copyright information

© Springer Science+Business Media New York 1999

Authors and Affiliations

  • Li-Shya Chen
    • 1
  • Seymour Geisser
    • 2
  • Charles J. Geyer
    • 2
  1. 1.National Chengchi UniversityTaipeiTaiwan
  2. 2.School of StatisticsUniversity of MinnesotaMinneapolisUSA

Personalised recommendations