Basics of Bayesian Methods

  • Sujit K. Ghosh
Part of the Methods in Molecular Biology book series (MIMB, volume 620)


Bayesian methods are rapidly becoming popular tools for making statistical inference in various fields of science including biology, engineering, finance, and genetics. One of the key aspects of Bayesian inferential method is its logical foundation that provides a coherent framework to utilize not only empirical but also scientific information available to a researcher. Prior knowledge arising from scientific background, expert judgment, or previously collected data is used to build a prior distribution which is then combined with current data via the likelihood function to characterize the current state of knowledge using the so-called posterior distribution. Bayesian methods allow the use of models of complex physical phenomena that were previously too difficult to estimate (e.g., using asymptotic approximations). Bayesian methods offer a means of more fully understanding issues that are central to many practical problems by allowing researchers to build integrated models based on hierarchical conditional distributions that can be estimated even with limited amounts of data. Furthermore, advances in numerical integration methods, particularly those based on Monte Carlo methods, have made it possible to compute the optimal Bayes estimators. However, there is a reasonably wide gap between the background of the empirically trained scientists and the full weight of Bayesian statistical inference. Hence, one of the goals of this chapter is to bridge the gap by offering elementary to advanced concepts that emphasize linkages between standard approaches and full probability modeling via Bayesian methods.

Key words

Bayesian inference hierarchical models likelihood function Monte Carlo methods posterior distribution prior distribution 


  1. 1.
    Press, J. S. and Tanur, J. M. (2001) The Subjectivity of Scientists and the Bayesian Approach. Wiley, New York.CrossRefGoogle Scholar
  2. 2.
    Berry, D. A. (1996) Statistics: A Bayesian Perspective. Wiley, New York.Google Scholar
  3. 3.
    Winkler, R. L. (2003). Introduction to Bayesian Inference and Decision. 2nd Edition.Google Scholar
  4. 4.
    Bolstad, W. M. (2004). Introduction to Bayesian Statistics. John Wiley, New York.CrossRefGoogle Scholar
  5. 5.
    Lee, P. M. (2004). Bayesian Statistics: An Introduction. Arnold, New York.Google Scholar
  6. 6.
    Sivia, D., and Skilling, J. (2006) Data Analysis: A Bayesian Tutorial. Oxford University Press, New York.Google Scholar
  7. 7.
    Berger, J. O. (1985). Statistical Decision Theory and Bayesian Analysis. 2nd Edition, Springer-Verlag, New York.CrossRefGoogle Scholar
  8. 8.
    Bernardo, J. M., and Smith, A. F. M. (1994) Bayesian Theory. Wiley, Chichester.CrossRefGoogle Scholar
  9. 9.
    Box, G. E. P., and Tiao, G. C. (1992) Bayesian inference in statistical analysis. Wiley, New York.CrossRefGoogle Scholar
  10. 10.
    Carlin, B. P., and Louis, T. A. (2008) Bayesian Methods for Data Analysis. 3rd Edition. Chapman & Hall/CRC, Boca Raton, Florida.Google Scholar
  11. 11.
    Congdon, P. (2007) Bayesian Statistical Modelling. 2nd Edition. Wiley, New York.Google Scholar
  12. 12.
    Gelman, A., Carlin, J. B., Stern, H. S., and Rubin, D. B. (2003) Bayesian Data Analysis. 2nd edition. CRC Press, New York.Google Scholar
  13. 13.
    Ghosh, J. K., Delampady, M., and Samanta, S. (2006) An Introduction to Bayesian Analysis. Springer, New York.Google Scholar
  14. 14.
    Robert, C. P. (2001) The Bayesian Choice. Springer Verlag, New York.Google Scholar
  15. 15.
    Bayes, T. (1764) An Essay Toward Solving a Problem in the Doctrine of Chances. Philos. Trans. R. Soc. London 53, 370–418.Google Scholar
  16. 16.
    Laplace, P. S. (1774) Mmoire sur la probabilit des causes par les vnements. Mmoires de mathmatique et de physique presents. lAcadmie royale des sciences par divers savants & lus dans ses assembles 6, 621–656.Google Scholar
  17. 17.
    Kolmogorov, A. N. (1930) Sur la loi forte des grands nombres. Comptes Rendus de l’Academie des. Sciences 191, 910–912.Google Scholar
  18. 18.
    Pitman, E. (1936) Sufficient statistics and intrinsic accuracy. Proc. Camb. Phil. Soc. 32, 567–579.CrossRefGoogle Scholar
  19. 19.
    Koopman, B. (1936) On distribution admitting a sufficient statistic. Trans. Amer. math. Soc. 39, 399–409.CrossRefGoogle Scholar
  20. 20.
    Diaconis, P. and Ylvisaker, D. (1979) Conjugate priors for exponential families. Ann. Stat. 7, 269–281.CrossRefGoogle Scholar
  21. 21.
    Consonni, G., and Veronese, P. (1992) Conjugate priors for exponential families having quadratic variance functions. J. Amer. Stat. Assoc. 87, 1123–1127.CrossRefGoogle Scholar
  22. 22.
    Consonni, G., and Veronese, P. (2001) Conditionally reducible natural exponential families and enriched conjugate priors. Scand. J. Stat. 28, 377–406.CrossRefGoogle Scholar
  23. 23.
    Gutirrez-Pea, E. (1997) Moments for the canonical parameter of an exponential family under a conjugate distribution. Biometrika. 84, 727–732.CrossRefGoogle Scholar
  24. 24.
    Jeffreys, H. (1946) An invariant form for the prior probability estimation problems. Proc. R. Stat. Soc. London (Ser. A). 186, 453–461.CrossRefGoogle Scholar
  25. 25.
    Hartigan, J. A. (1998) The maximum likelihood prior. Ann. Stat. 26, 2083–2103.CrossRefGoogle Scholar
  26. 26.
    Bernardo, J. M. (1979) Reference posterior distributions for Bayesian inference, J. R. Stat. Soc., B. 41, 113–147 (with discussion).Google Scholar
  27. 27.
    Zellner, A. (1996) Models, Prior Information, and Bayesian Analysis. J. Econom. 75, 51–68.CrossRefGoogle Scholar
  28. 28.
    Lhoste, E. (1923) Le Calcul des probabilits appliqu lartillerie, lois de probabilit apriori. Revue dartillerie, Mai-Aot, Berger-Levrault, Paris.Google Scholar
  29. 29.
    Berger, J. O., and Strawderman, W. E. (1996) Choice of hierarchical priors: Admissibility in estimation of normal means. Ann. Stat. 24, 931–951.CrossRefGoogle Scholar
  30. 30.
    Efron, B., and Morris, C. (1975) Data analysis using Stein’s estimator and its generalizations. J. Amer. Stat. Assoc. 70, 311–319.CrossRefGoogle Scholar
  31. 31.
    Davis, P. J., and Rabinowitz, P. (1984) Methods of Numerical Integration. 2nd Edition. Academic Press, New York.Google Scholar
  32. 32.
    Ueberhuber, C. W. (1997) Numerical Computation 2: Methods, Software, and Analysis. Springer-Verlag, Berlin.CrossRefGoogle Scholar
  33. 33.
    Piessens, R., de Doncker‐Kapenga, E., Uberhuber, C., and Kahaner, D. (1983) QUADPACK, A Subroutine Package for Automatic Integration. Springer-Verlag, Berlin.Google Scholar
  34. 34.
    Metropolis, N., and Ulam, S. (1949) The Monte Carlo Method. J. Amer. Stat. Assoc., 44, 335–341.CrossRefGoogle Scholar
  35. 35.
    Devroye, L. (1986) Non-Uniform Random Variate Generation. Springer-Verlag, New York.Google Scholar
  36. 36.
    Gentle, J. E. (2003) Random Number Generation and Monte Carlo Methods, 2nd Edition. Springer-Verlag, New York.Google Scholar
  37. 37.
    Berg, B. A. (2004) Markov chain Monte Carlo Simulations and their Statistical Analysis. World Scientific, Singapore.Google Scholar
  38. 38.
    Chib, S. and Greenberg, E. (1995) Understanding the MetropolisHastings Algorithm. Am. Stat. 49(4), 327–335.Google Scholar

Copyright information

© Humana Press, a part of Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Sujit K. Ghosh
    • 1
  1. 1.Department of StatisticsNorth Carolina State UniversityRaleighUSA

Personalised recommendations