# Markov Chains and Monte Carlo Markov Chains

• Massimiliano Bonamente
Chapter
Part of the Graduate Texts in Physics book series (GTP)

## Abstract

The theory of Markov chains is rooted in the work of Russian mathematician Andrey Markov, and has an extensive body of literature to establish its mathematical foundations. The availability of computing resources has recently made it possible to use Markov chains to analyze a variety of scientific data, and Monte Carlo Markov chains are now one of the most popular methods of data analysis. The modern-day data analyst will find that Monte Carlo Markov chains are an essential tool that permits tasks that are simply not possible with other methods, such as the simultaneous estimate of parameters for multi-parametric models of virtually any level of complexity. This chapter starts with an introduction to the mathematical properties of Markov chains necessary to understand its implementation as a Monte Carlo Markov chains. The second part is devoted to a description of the implementation of Monte Carlo Markov chains, including the Metropolis-Hasting algorithm and a few tests of convergence.

## Keywords

Markov Chain Posterior Distribution Monte Carlo Markov Chain Stationary Distribution Monte Carlo Markov Chain Method
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

## References

1. 1.
Abramowitz, M., Stegun, I.A.: Handbook of Mathematical Functions: With Formulas, Graphs, and Mathematical Tables. Dover, New York (1970)Google Scholar
2. 2.
Akritas, M.G., Bershady, M.A.: Linear regression for astronomical data with measurement errors and intrinsic scatter. Astrophys. J. 470, 706 (1996). doi:10.1086/177901
3. 3.
Bayes, T., Price, R.: An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. 53 (1763)Google Scholar
4. 4.
Bonamente, M., Swartz, D.A., Weisskopf, M.C., Murray, S.S.: Swift XRT observations of the possible dark galaxy VIRGOHI 21. Astrophys. J. Lett. 686, L71–L74 (2008). doi:10.1086/592819
5. 5.
Bonamente, M., Hasler, N., Bulbul, E., Carlstrom, J.E., Culverhouse, T.L., Gralla, M., Greer, C., Hawkins, D., Hennessy, R., Joy, M., Kolodziejczak, J., Lamb, J.W., Landry, D., Leitch, E.M., Marrone, D.P., Miller, A., Mroczkowski, T., Muchovej, S., Plagge, T., Pryke, C., Sharp, M., Woody, D.: Comparison of pressure profiles of massive relaxed galaxy clusters using the Sunyaev–Zel’dovich and x-ray data. N. J. Phys. 14(2), 025010 (2012). doi:10.1088/1367-2630/14/2/025010
6. 6.
Brooks, S.P., Gelman, A.: General methods for monitoring convergence of iterative simulations. J. Comput. Graph. Stat. 7, 434–455 (1998)
7. 7.
Bulmer, M.G.: Principles of Statistics. Dover, New York (1967)
8. 8.
Carlin, B., Gelfand, A., Smith, A.: Hierarchical Bayesian analysis for changepoint problems. Appl. Stat. 41, 389–405 (1992)
9. 9.
Cash, W.: Parameter estimation in astronomy through application of the likelihood ratio. Astrophys. J. 228, 939–947 (1979). doi:10.1086/156922
10. 10.
Cramer, H.: Mathematical Methods of Statistics. Princeton University Press, Princeton (1946)
11. 11.
Cowan, G.: Statistical Data Analysis, Oxford University Press (1998)Google Scholar
12. 12.
Emslie, A.G., Massone, A.M.: Bayesian confidence limits ETC ETC. ArXiv e-prints (2012)Google Scholar
13. 13.
Fisher, R.A.: On a distribution yielding the error functions of several well known statistics. Proc. Int. Congr. Math. 2, 805–813 (1924)Google Scholar
14. 14.
Gamerman, D.: Markov Chain Monte Carlo. Chapman and Hall CRC, London/New York (1997)
15. 15.
Gehrels, N.: Confidence limits for small numbers of events in astrophysical data. Astrophys. J. 303, 336–346 (1986). doi:10.1086/164079
16. 16.
Gelman, A., Rubin, D.: Inference from iterative simulation using multiple sequences. Stat. Sci. 7, 457–511 (1992)
17. 17.
Gosset, W.S.: The probable error of a mean. Biometrika 6, 1–25 (1908)Google Scholar
18. 18.
Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57(1), 97–109 (1970). doi:10.1093/biomet/57.1.97. http://biomet.oxfordjournals.org/content/57/1/97.abstract
19. 19.
Helmert, F.R.: Die genauigkeit der formel von peters zur berechnung des wahrscheinlichen fehlers director beobachtungen gleicher genauigkeit. Astron. Nachr. 88, 192–218 (1876)
20. 20.
Hubble, E., Humason, M.: The velocity-distance relation among extra-galactic nebulae. Astrophys. J. 74, 43 (1931)
21. 21.
Jeffreys, H.: Theory of Probability. Oxford University Press, London (1939)Google Scholar
22. 22.
Kelly, B.C.: Some aspects of measurement error in linear regression of astronomical data. Astrophys. J. 665, 1489–1506 (2007). doi:10.1086/519947
23. 23.
Kolmogorov, A.: Sulla determinazione empirica di una legge di distribuzione. Giornale dell’ Istituto Italiano degli Attuari 4, 1–11 (1933)
24. 24.
Kolmogorov, A.N.: Foundations of the Theory of Probability. Chelsea, New York (1950)Google Scholar
25. 25.
Lampton, M., Margon, B., Bowyer, S.: Parameter estimation in X-ray astronomy. Astrophys. J. 208, 177–190 (1976). doi:10.1086/154592
26. 26.
27. 27.
Marsaglia, G., Tsang, W., Wang, J.: Evaluating kolmogorov’s distribution. J. Stat. Softw. 8, 1–4 (2003)Google Scholar
28. 28.
Mendel, G.: Versuche über plflanzenhybriden (experiments in plant hybridization). Verhandlungen des naturforschenden Vereines in Brünn pp. 3–47 (1865)Google Scholar
29. 29.
Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21, 1087–1092 (1953). doi:10.1063/1.1699114
30. 30.
Pearson, K., Lee, A.: On the laws on inheritance in men. Biometrika 2, 357–462 (1903)Google Scholar
31. 31.
Plummer, M., Best, N., Cowles, K., Vines, K.: CODA: convergence diagnosis and output analysis for MCMC. R News 6(1), 7–11 (2006). http://CRAN.R-project.org/doc/Rnews/
32. 32.
Press, W., Teukolski, S., Vetterling, W., Flannery, B.: Numerical Recipes 3rd Edition: The Art of Scientific Computing. Cambridge University Press, Cambridge/New York (2007)Google Scholar
33. 33.
Protassov, R., van Dyk, D.A., Connors, A., Kashyap, V.L., Siemiginowska, A.: Statistics, handle with care: detecting multiple model components with the likelihood ratio test. Astrophys. J. 571, 545–559 (2002). doi:10.1086/339856
34. 34.
Raftery, A., Lewis, S.: How many iterations in the gibbs sampler? Bayesian Stat. 4, 763–773 (1992)Google Scholar
35. 35.
Ross, S.M.: Introduction to Probability Models. Academic, San Diego (2003)
36. 36.
Thomson, J.J.: Cathode rays. Philos. Mag. 44, 293 (1897)Google Scholar
37. 37.
Tremaine, S., Gebhardt, K., Bender, R., Bower, G., Dressler, A., Faber, S.M., Filippenko, A.V., Green, R., Grillmair, C., Ho, L.C., Kormendy, J., Lauer, T.R., Magorrian, J., Pinkney, J., Richstone, D.: The slope of the black hole mass versus velocity dispersion correlation. Astrophys. J. 574, 740–753 (2002). doi:10.1086/341002
38. 38.
Wilks, S.S.: Mathematical Statistics. Princeton University Press, Princeton (1943)