Simulation Methods

  • Larry Wasserman
Part of the Springer Texts in Statistics book series (STS)


In this chapter we will show how simulation can be used to approximate integrals. Our leading example is the problem of computing integrals in Bayesian inference but the techniques are widely applicable. We will look at three integration methods: (i) basic Monte Carlo integration, (ii) importance sampling, and (iii) Markov chain Monte Carlo (MCMC).


Markov Chain Markov Chain Monte Carlo Bayesian Inference Gibbs Sampling Importance Sampling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Agresti, A. (1990).Categorical Data Analysis.Wiley.Google Scholar
  2. Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle.Second International Symposium on Information Theory 267–281.Google Scholar
  3. Anderson, T. W. (1984). AnIntroduction to Multivariate Statistical Analysis (Second Edition). Wiley.Google Scholar
  4. Barron, A., Schervish, M. J. and Wasserman, L. (1999). The consistency of posterior distributions in nonparametric problems.The Annals of Statistics 27 536–561.CrossRefzbMATHMathSciNetGoogle Scholar
  5. Beecher, H. (1959).Measurement of Subjective Responses.Oxford University Press.Google Scholar
  6. Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society,Series B,Methodological 57 289–300.zbMATHMathSciNetGoogle Scholar
  7. Reran, R. (2000). React scatterplot smoothers: Superefficiency through basis economy.Journal of the American Statistical Association 95 155–171.CrossRefMathSciNetGoogle Scholar
  8. Beran, R. and Dümbgen, L. (1998). Modulation of estimators and confidence sets.The Annals of Statistics 26 1826–1856.CrossRefzbMATHMathSciNetGoogle Scholar
  9. Berger, J. and Wolpert, R. (1984). The Likelihood Principle. Institute of Mathematical Statistics.Google Scholar
  10. Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis(Second Edition). Springer-Verlag.Google Scholar
  11. Berger, J. O. and Delampady, M. (1987). Testing precise hypotheses (c/r: P335–352).Statistical Science 2 317–335.CrossRefzbMATHMathSciNetGoogle Scholar
  12. Berliner, L. M. (1983). Improving on inadmissible estimators in the control problem.The Annals of Statistics 11 814–826.CrossRefzbMATHMathSciNetGoogle Scholar
  13. Bickel, P. J. and Doxsum, K. A. (2000). Mathematical Statistics: Basic Ideas and Selected Topics, Vol. I (Second Edition).Prentice Hall.Google Scholar
  14. Billingsley, P. (1979). Probability and Measure. Wiley.Google Scholar
  15. Bishop, Y. M. M.. Fienberg, S. E. and Holland, P. W. (1975).Discrete Multivariate Analyses: Theory and Practice. MIT Press.Google Scholar
  16. Breiman, L. (1992).Probability. Society for Industrial and Applied Mathematics.Google Scholar
  17. Brinegar, C. S. (1963). Mark Twain and the Quintus Curtius Snodgrass letters: A statistical test of authorship.Journal of the American Statistical Association 58 85–96.CrossRefGoogle Scholar
  18. Carlin, B. P. and Lows, T. A. (1996). Bayes and Empirical Bayes Methods for Data Analysis. Chapman & Hall.Google Scholar
  19. Casella, G. and Berger, R. L. (2002). Statistical Inference. Duxbury Press.Google Scholar
  20. Chaudhuri, P. and Marron, J. S. (1999). Sizer for exploration of structures in curves.Journal of the American Statistical Association 94 807–823.CrossRefzbMATHMathSciNetGoogle Scholar
  21. Cox, D. and Lewis, P. (1966). The Statistical Analysis of Series of Events. Chapman & Hall.Google Scholar
  22. Cox, D. D. (1993). An analysis of Bayesian inference for nonparametric regression.The Annals of Statistics 21 903–923.CrossRefzbMATHMathSciNetGoogle Scholar
  23. Davison, A. C. and Hinkley, D. V. (1997). Bootstrap Methods and Their Application. Cambridge University Press.Google Scholar
  24. Degroot, M. and Schervish, M. (2002). Probability and Statistics (Third Edition). Addison-Wesley.Google Scholar
  25. Devroye, L., Györfi, L. and Lugosi, G. (1996). AProbabilistic Theory of Pattern Recognition. Springer-Verlag.Google Scholar
  26. Diaconis, P. and Freedman, D. (1986). on inconsistent Bayes estimates of location.The Annals of Statistics 14 68–87.Google Scholar
  27. Dobson, A. J. (2001). Anintroduction to generalized linear models. Chapman & Hall.Google Scholar
  28. Donoho, D. L. and Johnstone, I. M. (1994). Ideal spatial adaptation by wavelet shrinkage.Biometrika 81 425–455.CrossRefzbMATHMathSciNetGoogle Scholar
  29. Donoho, D. L. and Johnstone, I. M. (1995). Adapting to unknown smoothness via wavelet shrinkage.Journal of the American Statistical Association 90 1200–1224.CrossRefzbMATHMathSciNetGoogle Scholar
  30. Donoho, D. L. and Johnstone, I. M. (1998). Minimax estimation via wavelet shrinkage.The Annals of Statistics 26 879–921.CrossRefzbMATHMathSciNetGoogle Scholar
  31. Donoho, D. L., Johnstone, I. M., Kerkyacharian, G. and Picard, D. (1995). Wavelet shrinkage: Asymptopia? (Disc: p 337–369).Journal of the Royal Statistical Society,Series B,Methodological 57 301–337.zbMATHMathSciNetGoogle Scholar
  32. Dunsmore, I., Daly, F. ET AL. (1987).M345 Statistical Methods , Unit 9: Categorical Data. The Open University.Google Scholar
  33. Edwards, D. (1995).Introduction to graphical modelling. Springer-Verlag.Google Scholar
  34. Efromovich, S. (1999).Nonparametric Curve Estimation: Methods , Theory and Applications. Springer-Verlag.Google Scholar
  35. Efron, B. (1979). Bootstrap methods: Another look at the jackknife.The Annals of Statistics 7 1–26.CrossRefzbMATHMathSciNetGoogle Scholar
  36. Efron, B., Tibshirani, R., Storey, J. D. and Tusher, V. (2001). Empirical Bayes analysis of a microarray experiment.Journal of the American Statistical Association 96 1151–1160.CrossRefzbMATHMathSciNetGoogle Scholar
  37. Efron, B. and Tibshirani, R. J. (1993).An Introduction to the Bootstrap. Chapman & Hall.Google Scholar
  38. Ferguson, T. (1967).Mathematical Statistics: a Decision Theoretic Approach. Academic Press.Google Scholar
  39. Fisher, R. (1921). On the probable error of a coefficient of correlation deduced from a small sample.Metron 1 1–32.Google Scholar
  40. Freedman, D. (1999). Wald lecture: On the Bernstein-von Mises theorem with infinite-dimensional parameters.The Annals of Statistics 27 1119–1141.zbMATHMathSciNetGoogle Scholar
  41. Friedman, J. H. (1997). On bias, variance, 0/1-loss, and the curse-ofdimensionality.Data Mining and Knowledge Discovery 1 55–77.CrossRefGoogle Scholar
  42. Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. B. (1995).Bayesian Data Analysis. Chapman & Hall.Google Scholar
  43. Ghosal, S., Ghosh, J. K. and Van Der Vaart, A. W. (2000). Convergence rates of posterior distributions.The Annals of Statistics 28 500–531.CrossRefzbMATHMathSciNetGoogle Scholar
  44. Gilks, W. R., Richardson, S. and Spiegelhalter, D. J. (1998).Markov Chain Monte Carlo in Practice. Chapman & Hall.Google Scholar
  45. Grimmett, G. and Stirzaker, D. (1982).Probability and Random Processes. Oxford University Press.Google Scholar
  46. Guttorp, P. (1995).Stochastic Modeling of Scientific Data. Chapman & Hall.Google Scholar
  47. Hardle, W., Kerkyacharian, G., Picard, D. and Tsybakov, A. (1998).Wavelets, Approximation, and Statistical Applications. Springer-Verlag.Google Scholar
  48. Hall, P. (1992).The Bootstrap and Edgeworth Expansion. Springer-Verlag.Google Scholar
  49. Halverson, N., Leitch, E., Pryke, C., Kovac, J., Carlstrom, J., Holzapfel, W., Dragovan, M., Cartwright, J., Mason, B., Padin, S., Pearson, T., Shepherd, M. and Readhead, A. (2002). DASI first results: A measurement of the cosmic microwave background angular power spectrum.Astrophysics Journal 568 38–45.CrossRefGoogle Scholar
  50. Hardle, W. (1990).Applied nonparametric regression. Cambridge University Press.Google Scholar
  51. Hastie, T., Tibshirani, R. and Friedman, J. H. (2001).The Elements of Statistical Learning: Data Mining , Inference , and Prediction. Springer-Verlag.Google Scholar
  52. Herbich, R. (2002). Learning Kernel Classifiers: Theory and Algorithms. MIT Press.Google Scholar
  53. Johnson, R. A. and Wichern, D. W. (1982).Applied Multivariate Statistical Analysis. Prentice-Hall.Google Scholar
  54. Johnson, S. and Johnson, R. (1972).New England Journal of Medicine 287 1122–1125.CrossRefGoogle Scholar
  55. Jordan, M. (2004).Graphical models. In Preparation.Google Scholar
  56. Karr, A. (1993).Probability. Springer-Verlag.Google Scholar
  57. Kass, R. E. and Raftery, A. E. (1995). Bayes factors.Journal of the American Statistical Association 90 773–795.CrossRefzbMATHGoogle Scholar
  58. Kass, R. E. and Wasserman, L. (1996). The selection of prior distributions by formal rules (corr: 1998 v93 p 412).Journal of the American Statistical Association 91 1343–1370.CrossRefzbMATHGoogle Scholar
  59. Larsen, R. J. and Marx, M. L. (1986).An Introduction to Mathematical Statistics and Its Applications (Second Edition). Prentice Hall.Google Scholar
  60. Lauritzen, S. L. (1996).Graphical Models. Oxford University Press.Google Scholar
  61. Lee, A. T. et al. (2001). A high spatial resolution analysis of the maxima-1 cosmic microwave background anisotropy data.Astrophys. J. 561 L1 - L6.CrossRefGoogle Scholar
  62. Lee, P. M. (1997).Bayesian Statistics: An Introduction. Edward Arnold.Google Scholar
  63. Lehmann, E. L. (1986).Testing Statistical Hypotheses (Second Edition). Wiley.Google Scholar
  64. Lehmann, E. L. and Casella, G. (1998).Theory of Point Estimation. Springer-Verlag.Google Scholar
  65. Loader, C. (1999).Local regression and likelihood. Springer-Verlag.Google Scholar
  66. Marron, J. S. and Wand, M. P. (1992). Exact mean integrated squared error.The Annals of Statistics 20 712–736.CrossRefzbMATHMathSciNetGoogle Scholar
  67. Morrison, A., Black, M., Lowe, C., Macmahon, B. and Yusa, S. (1973). Some international differences in histology and survival in breast cancer.International Journal of Cancer 11 261–267.CrossRefGoogle Scholar
  68. Netterfield, C. B. et al. (2002). A measurement by boomerang of multiple peaks in the angular power spectrum of the cosmic microwave background.Astrophys. J. 571 604–614.CrossRefGoogle Scholar
  69. Ogden, R. T. (1997).Essential Wavelets for Statistical Applications and Data Analysis. Birkhäuser.Google Scholar
  70. Pearl, J. (2000).Casuality: models , reasoning , and inference. Cambridge University Press.Google Scholar
  71. Phillips, D. and King, E. (1988). Death takes a holiday: Mortality surrounding major social occasions.Lancet 2 728–732.CrossRefGoogle Scholar
  72. Phillips, D. and Smith, D. (1990). Postponement of death until symbolically meaningful occasions.Journal of the American Medical Association 263 1947–1961.CrossRefGoogle Scholar
  73. Quenouille, M. (1949). Approximate tests of correlation in time series.Journal of the Royal Statistical Society B 11 18–84.MathSciNetGoogle Scholar
  74. Rice, J. A. (1995).Mathematical Statistics and Data Analysis (Second Edition). Duxbury Press.Google Scholar
  75. Robert, C. P. (1994).The Bayesian Choice: A Decision-theoretic Motivation. Springer-Verlag.Google Scholar
  76. Robert, C. P. and Casella, G. (1999).Monte Carlo Statistical Methods. Springer-Verlag.Google Scholar
  77. Robins, J., Scheines, R., Spirtes, P. and Wasserman, L. (2003). Uniform convergence in causal inference.Biometrika (to appear).Google Scholar
  78. Robins, J. M. and Ritov, Y. (1997). Toward a curse of dimensionality appropriate (CODA) asymptotic theory for semi-parametric models.Statistics in Medicine 16 285–319.CrossRefGoogle Scholar
  79. Rosenbaum, P. (2002).Observational Studies. Springer-Verlag.Google Scholar
  80. Rousseauw, J., Du Plessis, J., Benade, A., Jordaan, P., Kotze, J., Jooste, P. and Ferreira, J. (1983). Coronary risk factor screening in three rural communities.South African Medical Journal 64 430–436.Google Scholar
  81. Schervish, M. J. (1995).Theory of Statistics. Springer-Verlag.Google Scholar
  82. Scholkopf, B. and Smola, A. (2002).Learning with Kernels: Support Vector Machines , Regularization , Optimization , and Beyond. MIT Press.Google Scholar
  83. Schwarz, G. (1978). Estimating the dimension of a model.The Annals of Statistics 6 461–464.CrossRefzbMATHMathSciNetGoogle Scholar
  84. Scott, D., Lotto, A., Cole, J. and Gorry, G. (1978). Plasma lipids as collateral risk factors in coronary artery disease: a study of 371 males with chest pain.Journal of Chronic Diseases 31 337–345.CrossRefGoogle Scholar
  85. Scott, D. W. (1992).Multivariate Density Estimation: Theory , Practice , and Visualization. Wiley.Google Scholar
  86. Shao, J. and Tu, D. (1995).The Jackknife and Bootstrap (German). Springer-Verlag.Google Scholar
  87. Shen, X. and Wasserman, L. (2001). Rates of convergence of posterior distributions.The Annals of Statistics 29 687–714.CrossRefzbMATHMathSciNetGoogle Scholar
  88. Shorack, G. R. and Wellner, J. A. (1986).Empirical Processes With Applications to Statistics. Wiley.Google Scholar
  89. Silverman, B. W. (1986).Density Estimation for Statistics and Data Analysis. Chapman & Hall.Google Scholar
  90. Spirtes, P., Glymour, C. N. and Scheines, R. (2000).Causation , prediction , and search. MIT Press.Google Scholar
  91. Taylor, H. M. and Karlin, S. (1994).An Introduction to Stochastic Modeling. Academic Press.Google Scholar
  92. Van Der Laan, M. and Robins, J. (2003).Unified Methods for Censored Longitudinal Data and Causality. Springer Verlag.Google Scholar
  93. Van Der Vaart, A. W. (1998).Asymptotic Statistics. Cambridge University Press.Google Scholar
  94. Vapnik, V. N. (1998).Statistical Learning Theory. Wiley.Google Scholar
  95. Weisberg, S. (1985).Applied Linear Regression. Wiley.Google Scholar
  96. Whittaker, J. (1990).Graphical Models in Applied Multivariate Statistics. Wiley.Google Scholar
  97. Wright, S. (1934). The method of path coefficients.The Annals of Mathematical Statistics 5 161–215.CrossRefGoogle Scholar
  98. Zhao, L. H. (2000). Bayesian aspects of some nonparametric problems.The Annals of Statistics 28 532–552.CrossRefzbMATHMathSciNetGoogle Scholar
  99. Zheng, X. and Loh, W.-Y. (1995). Consistent variable selection in linear models.Journal of the American Statistical Association 90 151–156.CrossRefzbMATHMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2004

Authors and Affiliations

  • Larry Wasserman
    • 1
  1. 1.Department of StatisticsCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations