Skip to main content
SpringerLink
Log in
Book cover

Case Studies in Applied Bayesian Data Science pp 137–151Cite as

Bayesian Computation with Intractable Likelihoods

  • Chapter
  • First Online:

Part of the book series: Lecture Notes in Mathematics ((LNM,volume 2259))

Abstract

This chapter surveys computational methods for posterior inference with intractable likelihoods, that is where the likelihood function is unavailable in closed form, or where evaluation of the likelihood is infeasible. We survey recent developments in pseudo-marginal methods, approximate Bayesian computation (ABC), the exchange algorithm, thermodynamic integration, and composite likelihood, paying particular attention to advancements in scalability for large datasets. We also mention R and MATLAB source code for implementations of these algorithms, where they are available.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. P. Alquier, N. Friel, R. Everitt, A. Boland, Noisy Monte Carlo: convergence of Markov chains with approximate transition kernels. Stat. Comput. 26(1–2), 29–47 (2016). https://doi.org/10.1007/s11222-014-9521-x

    Article  MathSciNet  MATH  Google Scholar 

  2. C. Andrieu, G.O. Roberts, The pseudo-marginal approach for efficient Monte Carlo computations. Ann. Statist. 37(2), 697–725 (2009). https://doi.org/10.1214/07-AOS574

    Article  MathSciNet  MATH  Google Scholar 

  3. C. Andrieu, J. Thoms, A tutorial on adaptive MCMC. Stat. Comput. 18(4), 343–373 (2008). https://doi.org/10.1007/s11222-008-9110-y

    Article  MathSciNet  Google Scholar 

  4. C. Andrieu, M. Vihola, Convergence properties of pseudo-marginal Markov chain Monte Carlo algorithms. Ann. Appl. Prob. 25(2), 1030–1077, 04 (2015). https://doi.org/10.1214/14-AAP1022

    Article  MathSciNet  MATH  Google Scholar 

  5. C. Andrieu, A. Doucet, R. Holenstein, Particle Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. B 72(3), 269–342 (2010). https://doi.org/10.1111/j.1467-9868.2009.00736.x

    Article  MathSciNet  MATH  Google Scholar 

  6. M.A. Beaumont, Estimation of population growth or decline in genetically monitored populations. Genetics 164(3), 1139–1160 (2003)

    Google Scholar 

  7. A. Boland, N. Friel, F. Maire, Efficient MCMC for Gibbs random fields using pre-computation. Electron. J. Statist. 12(2), 4138–4179 (2018). https://doi.org/10.1214/18-EJS1504.

    Article  MathSciNet  MATH  Google Scholar 

  8. C.T. Butts, A perfect sampling method for exponential family random graph models. J. Math. Soc. 42(1), 17–36 (2018). https://doi.org/10.1080/0022250X.2017.1396985

    Article  MathSciNet  Google Scholar 

  9. A. Caimo, N. Friel, Bayesian inference for exponential random graph models. Soc. Networks 33(1), 41–55 (2011). https://doi.org/10.1016/j.socnet.2010.09.004

    Article  Google Scholar 

  10. A. Caimo, N. Friel, Bergm: Bayesian exponential random graphs in R. J. Stat. Soft. 61(2), 1–25 (2014). https://doi.org/10.18637/jss.v061.i02

    Article  Google Scholar 

  11. B. Calderhead, M. Girolami, Estimating Bayes factors via thermodynamic integration and population MCMC. Comput. Stat. Data Anal. 53(12), 4028–4045 (2009). https://doi.org/10.1016/j.csda.2009.07.025

    Article  MathSciNet  MATH  Google Scholar 

  12. E. Cameron, A.N. Pettitt, Approximate Bayesian computation for astronomical model analysis: a case study in galaxy demographics and morphological transformation at high redshift. Mon. Not. R. Astron. Soc. 425(1), 44–65 (2012). https://doi.org/10.1111/j.1365-2966.2012.21371.x

    Article  Google Scholar 

  13. M.-H. Chen, Q.-M. Shao, J.G. Ibrahim, Monte Carlo Methods in Bayesian Computation. Springer Series in Statistics (Springer, New York, 2000)

    Book  MATH  Google Scholar 

  14. J.A. Christen, C. Fox, Markov chain Monte Carlo using an approximation. J. Comput. Graph. Stat. 14(4), 795–810 (2005). https://doi.org/10.1198/106186005X76983

    Article  MathSciNet  Google Scholar 

  15. L. Cucala, J.-M. Marin, C.P. Robert, D.M. Titterington, A Bayesian reassessment of nearest-neighbor classification. J. Am. Stat. Assoc. 104(485), 263–273 (2009). https://doi.org/10.1198/jasa.2009.0125

    Article  MathSciNet  MATH  Google Scholar 

  16. P. Del Moral, A. Doucet, A. Jasra, An adaptive sequential Monte Carlo method for approximate Bayesian computation. Stat. Comput. 22(5), 1009–20 (2012). https://doi.org/10.1007/s11222-011-9271-y

    Article  MathSciNet  MATH  Google Scholar 

  17. A.P. Dempster, N.M. Laird, D.B. Rubin, Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B 39(1), 1–38 (1977).

    MathSciNet  MATH  Google Scholar 

  18. A. Doucet, M. Pitt, G. Deligiannidis, R. Kohn, Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator. Biometrika 102(2), 295–313 (2015). https://doi.org/10.1093/biomet/asu075

    Article  MathSciNet  MATH  Google Scholar 

  19. C.C. Drovandi, A.N. Pettitt, Estimation of parameters for macroparasite population evolution using approximate Bayesian computation. Biometrics 67(1), 225–233 (2011). https://doi.org/10.1111/j.1541-0420.2010.01410.x

    Article  MathSciNet  MATH  Google Scholar 

  20. C.C. Drovandi, A.N. Pettitt, M.J. Faddy, Approximate Bayesian computation using indirect inference. J. R. Stat. Soc. Ser. C 60(3), 317–337 (2011). https://doi.org/10.1111/j.1467-9876.2010.00747.x

    Article  MathSciNet  Google Scholar 

  21. C.C. Drovandi, A.N. Pettitt, A. Lee, Bayesian indirect inference using a parametric auxiliary model. Stat. Sci. 30(1), 72–95 (2015). https://doi.org/10.1214/14-STS498

    Article  MathSciNet  MATH  Google Scholar 

  22. C.C. Drovandi, M.T. Moores, R.J. Boys, Accelerating pseudo-marginal MCMC using Gaussian processes. Comput. Stat. Data Anal. 118, 1–17 (2018). https://doi.org/10.1016/j.csda.2017.09.002

    Article  MathSciNet  MATH  Google Scholar 

  23. P. Erdős, A. Rényi, On random graphs. Publ. Math. Debr. 6, 290–297 (1959)

    Google Scholar 

  24. R.G. Everitt, Bayesian parameter estimation for latent Markov random fields and social networks. J. Comput. Graph. Stat. 21(4), 940–960 (2012). https://doi.org/10.1080/10618600.2012.687493

    Article  MathSciNet  Google Scholar 

  25. P. Fearnhead, V. Giagos, C. Sherlock, Inference for reaction networks using the linear noise approximation. Biometrics 70(2), 457–466 (2014). https://doi.org/10.1111/biom.12152

    Article  MathSciNet  MATH  Google Scholar 

  26. O. Frank, D. Strauss, Markov graphs. J. Amer. Stat. Assoc. 81(395), 832–842 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  27. N. Friel, Bayesian inference for Gibbs random fields using composite likelihoods, in ed. by C. Laroque, J. Himmelspach, R. Pasupathy, O. Rose, A.M. Uhrmacher, Proceedings of the 2012 Winter Simulation Conference (WSC) (2012), pp. 1–8. https://doi.org/10.1109/WSC.2012.6465236

  28. N. Friel, A.N. Pettitt, Likelihood estimation and inference for the autologistic model. J. Comp. Graph. Stat. 13(1), 232–246 (2004). https://doi.org/10.1198/1061860043029

    Article  MathSciNet  Google Scholar 

  29. N. Friel, A.N. Pettitt, Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. Ser. B 70(3), 589–607 (2008). https://doi.org/10.1111/j.1467-9868.2007.00650.x

    Article  MathSciNet  MATH  Google Scholar 

  30. N. Friel, A.N. Pettitt, R. Reeves, E. Wit, Bayesian inference in hidden Markov random fields for binary data defined on large lattices. J. Comp. Graph. Stat. 18(2), 243–261 (2009). https://doi.org/10.1198/jcgs.2009.06148

    Article  MathSciNet  Google Scholar 

  31. A. Gelman, X.-L. Meng, Simulating normalizing constants: from importance sampling to bridge sampling to path sampling. Statist. Sci. 13(2), 163–185 (1998). https://doi.org/10.1214/ss/1028905934

    Article  MathSciNet  MATH  Google Scholar 

  32. C.J. Geyer, L. Johnson, potts: Markov Chain Monte Carlo for Potts Models. R package version 0.5-2 (2014). http://CRAN.R-project.org/package=potts

  33. A. Golightly, D.A. Henderson, C. Sherlock, Delayed acceptance particle MCMC for exact inference in stochastic kinetic models. Stat. Comput. 25(5), 1039–1055 (2015). https://doi.org/10.1007/s11222-014-9469-x

    Article  MathSciNet  MATH  Google Scholar 

  34. A. Grelaud, C.P. Robert, J.-M. Marin, F. Rodolphe, J.-F. Taly, ABC likelihood-free methods for model choice in Gibbs random fields. Bayesian Anal. 4(2), 317–336 (2009). https://doi.org/10.1214/09-BA412

    Article  MathSciNet  MATH  Google Scholar 

  35. M.L. Huber, A bounding chain for Swendsen-Wang. Random Struct. Algor. 22(1), 43–59 (2003). https://doi.org/10.1002/rsa.10071

    Article  MathSciNet  MATH  Google Scholar 

  36. M.L. Huber, Perfect Simulation (Chapman & Hall/CRC Press, London/Boca Raton, 2016)

    Book  MATH  Google Scholar 

  37. P.E. Jacob, A.H. Thiery, On nonnegative unbiased estimators. Ann. Statist. 43(2), 769–784 (2015). https://doi.org/10.1214/15-AOS1311

    Article  MathSciNet  MATH  Google Scholar 

  38. M. Järvenpää, M. Gutmann, A. Vehtari, P. Marttinen, Gaussian process modeling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria. Ann. Appl. Stat. 12(4), 2228–2251 (2018). https://doi.org/10.1214/18-AOAS1150

    Article  MathSciNet  MATH  Google Scholar 

  39. A. Lee, K. Łatuszyński, Variance bounding and geometric ergodicity of Markov chain Monte Carlo kernels for approximate Bayesian computation. Biometrika 101(3), 655–671 (2014). https://doi.org/10.1093/biomet/asu027

    Article  MathSciNet  MATH  Google Scholar 

  40. A.-M. Lyne, M. Girolami, Y. Atchadé, H. Strathmann, D. Simpson, On Russian roulette estimates for Bayesian inference with doubly-intractable likelihoods. Statist. Sci. 30(4), 443–467 (2015). https://doi.org/10.1214/15-STS523

    Article  MathSciNet  MATH  Google Scholar 

  41. F. Maire, R. Douc, J. Olsson, Comparison of asymptotic variances of inhomogeneous Markov chains with application to Markov chain Monte Carlo methods. Ann. Statist. 42(4), 1483–1510, 08 (2014). https://doi.org/10.1214/14-AOS1209

    Article  MathSciNet  MATH  Google Scholar 

  42. J.-M. Marin, C.P. Robert, Bayesian Core: A Practical Approach to Computational Bayesian Statistics. Springer Texts in Statistics (Springer, New York, 2007)

    Google Scholar 

  43. P. Marjoram, J. Molitor, V. Plagnol, S. Tavaré, Markov chain Monte Carlo without likelihoods. Proc. Natl Acad. Sci. 100(26), 15324–15328 (2003). https://doi.org/10.1073/pnas.0306899100

    Article  Google Scholar 

  44. C.A. McGrory, D.M. Titterington, R. Reeves, A.N. Pettitt, Variational Bayes for estimating the parameters of a hidden Potts model. Stat. Comput. 19(3), 329–340 (2009). https://doi.org/10.1007/s11222-008-9095-6

    Article  MathSciNet  Google Scholar 

  45. C.A. McGrory, A.N. Pettitt, R. Reeves, M. Griffin, M. Dwyer, Variational Bayes and the reduced dependence approximation for the autologistic model on an irregular grid with applications. J. Comput. Graph. Stat. 21(3), 781–796 (2012). https://doi.org/10.1080/10618600.2012.632232

    Article  MathSciNet  Google Scholar 

  46. T.J. McKinley, I. Vernon, I. Andrianakis, N. McCreesh, J.E. Oakley, R.N. Nsubuga, M. Goldstein, R.G. White, et al., Approximate Bayesian computation and simulation-based inference for complex stochastic epidemic models. Statist. Sci. 33(1), 4–18 (2018). https://doi.org/10.1214/17-STS618

    Article  MathSciNet  MATH  Google Scholar 

  47. F.J. Medina-Aguayo, A. Lee, G.O. Roberts, Stability of noisy Metropolis-Hastings. Stat. Comput. 26(6), 1187–1211 (2016). https://doi.org/10.1007/s11222-015-9604-3

    Article  MathSciNet  MATH  Google Scholar 

  48. E. Meeds, M. Welling, GPS-ABC: Gaussian process surrogate approximate Bayesian computation, in Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, Quebec City, Canada (2014)

    Google Scholar 

  49. A. Mira, J. Møller, G.O. Roberts, Perfect slice samplers. J. R. Stat. Soc. Ser. B 63(3), 593–606 (2001). https://doi.org/10.1111/1467-9868.00301

    Article  MathSciNet  MATH  Google Scholar 

  50. J. Møller, A.N. Pettitt, R. Reeves, K.K. Berthelsen, An efficient Markov chain Monte Carlo method for distributions with intractable normalising constants. Biometrika 93(2), 451–458 (2006). https://doi.org/10.1093/biomet/93.2.451

    Article  MathSciNet  MATH  Google Scholar 

  51. M.T. Moores, D. Feng, K. Mengersen, bayesImageS: Bayesian Methods for Image Segmentation Using a Potts Model. R package version 0.5-3 (2014). URL http://CRAN.R-project.org/package=bayesImageS

  52. M.T. Moores, C.C. Drovandi, K. Mengersen, C.P. Robert, Pre-processing for approximate Bayesian computation in image analysis. Stat. Comput. 25(1), 23–33 (2015). https://doi.org/10.1007/s11222-014-9525-6

    Article  MathSciNet  MATH  Google Scholar 

  53. M.T. Moores, G.K. Nicholls, A.N. Pettitt, K. Mengersen, Scalable Bayesian inference for the inverse temperature of a hidden Potts model. Bayesian Anal. 15, 1–27 (2020). https://doi.org/10.1214/18-BA1130.

    Article  MathSciNet  MATH  Google Scholar 

  54. I. Murray, Z. Ghahramani, D.J.C. MacKay, MCMC for doubly-intractable distributions, in Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence, Arlington (AUAI Press, Tel Aviv-Yafo, 2006), pp. 359–366

    Google Scholar 

  55. G.K. Nicholls, C. Fox, A. Muir Watt, Coupled MCMC with a randomized acceptance probability (2012).Preprint arXiv:1205.6857 [stat.CO]. https://arxiv.org/abs/1205.6857

  56. C.J. Oates, T. Papamarkou, M. Girolami, The controlled thermodynamic integral for Bayesian model evidence evaluation. J. Am. Stat. Assoc. 111(514), 634–645 (2016). https://doi.org/10.1080/01621459.2015.1021006

    Article  MathSciNet  Google Scholar 

  57. H.E. Ogden, On asymptotic validity of naive inference with an approximate likelihood. Biometrika 104(1), 153–164 (2017). https://doi.org/10.1093/biomet/asx002

    Article  MathSciNet  MATH  Google Scholar 

  58. S. Okabayashi, L. Johnson, C.J. Geyer, Extending pseudo-likelihood for Potts models. Statistica Sinica 21, 331–347 (2011)

    MathSciNet  MATH  Google Scholar 

  59. E. Olbrich, T. Kahle, N. Bertschinger, N. Ay, J. Jost, Quantifying structure in networks. Eur. Phys. J. B 77(2), 239–247 (2010). https://doi.org/10.1140/epjb/e2010-00209-0

    Article  Google Scholar 

  60. P.D. O’Neill, D.J. Balding, N.G. Becker, M. Eerola, D. Mollison, Analyses of infectious disease data from household outbreaks by Markov chain Monte Carlo methods. J. R. Stat. Soc. Ser. C 49(4), 517–542 (2000). https://doi.org/10.1111/1467-9876.00210

    Article  MathSciNet  MATH  Google Scholar 

  61. A.N. Pettitt, N. Friel, R. Reeves, Efficient calculation of the normalizing constant of the autologistic and related models on the cylinder and lattice. J. R. Stat. Soc. Ser. B 65(1), 235–246 (2003). https://doi.org/10.1111/1467-9868.00383

    Article  MathSciNet  MATH  Google Scholar 

  62. M.K. Pitt, R. dos Santos Silva, P. Giordani, R. Kohn, On some properties of Markov chain Monte Carlo simulation methods based on the particle filter. J. Econometr. 171(2), 134–151 (2012). https://doi.org/10.1016/j.jeconom.2012.06.004

    Article  MathSciNet  MATH  Google Scholar 

  63. D. Prangle, Lazy ABC. Stat. Comput. 26(1), 171–185 (2016). https://doi.org/10.1007/s11222-014-9544-3

    Article  MathSciNet  Google Scholar 

  64. J.K. Pritchard, M.T. Seielstad, A. Perez-Lezaun, M.W. Feldman, Population growth of human Y chromosomes: a study of Y chromosome microsatellites. Mol. Biol. Evol. 16(12), 1791–1798 (1999). https://doi.org/10.1093/oxfordjournals.molbev.a026091

    Article  Google Scholar 

  65. J.G. Propp, D.B. Wilson, Exact sampling with coupled Markov chains and applications to statistical mechanics. Random Struct. Algor. 9(1–2), 223–252 (1996). https://doi.org/10.1002/(SICI)1098-2418(199608/09)9:1/2<223::AID-RSA14>3.0.CO;2-O

    Article  MathSciNet  MATH  Google Scholar 

  66. R. Reeves, A.N. Pettitt, Efficient recursions for general factorisable models. Biometrika 91(3), 751–757 (2004). https://doi.org/10.1093/biomet/91.3.751

    Article  MathSciNet  MATH  Google Scholar 

  67. G.O. Roberts, J.S. Rosenthal, Examples of adaptive MCMC. J. Comput. Graph. Stat. 18(2), 349–367 (2009). https://doi.org/10.1198/jcgs.2009.06134

    Article  MathSciNet  Google Scholar 

  68. T. Rydén, D.M. Titterington, Computational Bayesian analysis of hidden Markov models. J. Comput. Graph. Stat. 7(2), 194–211 (1998). https://doi.org/10.1080/10618600.1998.10474770

    MathSciNet  Google Scholar 

  69. C. Sherlock, A.H. Thiery, G.O. Roberts, J.S. Rosenthal, On the efficiency of pseudo-marginal random walk Metropolis algorithms. Ann. Statist. 43(1), 238–275, 02 (2015). https://doi.org/10.1214/14-AOS1278

    Article  MathSciNet  MATH  Google Scholar 

  70. C. Sherlock, A. Golightly, D.A. Henderson, Adaptive, delayed-acceptance MCMC for targets with expensive likelihoods. J. Comput. Graph. Stat. 26(2), 434–444 (2017). https://doi.org/10.1080/10618600.2016.1231064

    Article  MathSciNet  Google Scholar 

  71. A.M. Stuart, A.L. Teckentrup, Posterior consistency for Gaussian process approximations of Bayesian posterior distributions. Math. Comp. 87, 721–753 (2018). https://doi.org/10.1090/mcom/3244

    Article  MathSciNet  MATH  Google Scholar 

  72. R.H. Swendsen, J.-S. Wang, Nonuniversal critical dynamics in Monte Carlo simulations. Phys. Rev. Lett. 58, 86–88 (1987). https://doi.org/10.1103/PhysRevLett.58.86

    Article  Google Scholar 

  73. M.A. Tanner, W.H. Wong, The calculation of posterior distributions by data augmentation. J. Am. Stat. Assoc. 82(398), 528–40 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  74. C. Varin, N. Reid, D. Firth, An overview of composite likelihood methods. Statistica Sinica 21, 5–42 (2011)

    MathSciNet  MATH  Google Scholar 

  75. R.D. Wilkinson, Accelerating ABC methods using Gaussian processes, in ed. by S. Kaski, J. Corander, Proceedings of the 17th International Conference on Artificial Intelligence and Statistics AISTATS (JMLR: Workshop and Conference Proceedings) , vol. 33 (2014), pp. 1015–1023

    Google Scholar 

Download references

Acknowledgements

This research was conducted by the Australian Research Council Centre of Excellence for Mathematical and Statistical Frontiers (project number CE140100049) and funded by the Australian Government.

Author information

Authors and Affiliations

  1. National Institute for Applied Statistics Research Australia, School of Mathematics & Applied Statistics, University of Wollongong, Keiraville, NSW, Australia

    Matthew T. Moores

  2. School of Mathematical Sciences, Queensland University of Technology, Brisbane, QLD, Australia

    Anthony N. Pettitt & Kerrie L. Mengersen

Authors
  1. Matthew T. Moores
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anthony N. Pettitt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kerrie L. Mengersen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matthew T. Moores .

Editor information

Editors and Affiliations

  1. Mathematical Sciences, Queensland University of Technology, Brisbane, QLD, Australia

    Kerrie L. Mengersen

  2. I2M, CNRS, Centrale Marseille, Aix-Marseille University, Marseille, France

    Pierre Pudlo

  3. CEREMADE, Université Paris Dauphine, Paris, France

    Christian P. Robert

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Moores, M.T., Pettitt, A.N., Mengersen, K.L. (2020). Bayesian Computation with Intractable Likelihoods. In: Mengersen, K., Pudlo, P., Robert, C. (eds) Case Studies in Applied Bayesian Data Science. Lecture Notes in Mathematics, vol 2259. Springer, Cham. https://doi.org/10.1007/978-3-030-42553-1_6

Download citation

Publish with us

Policies and ethics

Search

Navigation