Advertisement

PharmacoEconomics

, Volume 35, Issue 6, pp 613–624 | Cite as

Bayesian Methods for Calibrating Health Policy Models: A Tutorial

  • Nicolas A. MenziesEmail author
  • Djøra I. Soeteman
  • Ankur Pandya
  • Jane J. Kim
Practical Application

Abstract

Mathematical simulation models are commonly used to inform health policy decisions. These health policy models represent the social and biological mechanisms that determine health and economic outcomes, combine multiple sources of evidence about how policy alternatives will impact those outcomes, and synthesize outcomes into summary measures salient for the policy decision. Calibrating these health policy models to fit empirical data can provide face validity and improve the quality of model predictions. Bayesian methods provide powerful tools for model calibration. These methods summarize information relevant to a particular policy decision into (1) prior distributions for model parameters, (2) structural assumptions of the model, and (3) a likelihood function created from the calibration data, combining these different sources of evidence via Bayes’ theorem. This article provides a tutorial on Bayesian approaches for model calibration, describing the theoretical basis for Bayesian calibration approaches as well as pragmatic considerations that arise in the tasks of creating calibration targets, estimating the posterior distribution, and obtaining results to inform the policy decision. These considerations, as well as the specific steps for implementing the calibration, are described in the context of an extended worked example about the policy choice to provide (or not provide) treatment for a hypothetical infectious disease. Given the many simplifications and subjective decisions required to create prior distributions, model structure, and likelihood, calibration should be considered an exercise in creating a reasonable model that produces valid evidence for policy, rather than as a technique for identifying a unique theoretically optimal summary of the evidence.

Notes

Author Contributions

N.A.M., J.J.K., A.P., and D.I.S. conceived the paper. N.A.M. wrote the paper. J.J.K., A.P., and D.I.S. edited the paper.

Compliance with Ethical Standards

Funding

NAM received funding from National Institutes of Health/National Institute of Allergy and Infectious Diseases (Grant No. 7R01AI112438-02).

Conflict of interest

The authors report no conflicts of interest directly relevant to the content of this article.

Supplementary material

40273_2017_494_MOESM1_ESM.pdf (1 mb)
Supplementary material 1 (PDF 1065 kb)

References

  1. 1.
    Vanni T, Karnon J, Madan J, et al. Calibrating models in economic evaluation: a seven-step approach. Pharmacoeconomics. 2011;29(1):35–49.CrossRefGoogle Scholar
  2. 2.
    Enns EA, Cipriano LE, Simons CT, Kong CY. Identifying best-fitting inputs in health-economic model calibration: a Pareto frontier approach. Med Decis Making. 2015;35(2):170–82.CrossRefGoogle Scholar
  3. 3.
    Waller LA, Smith D, Childs JE, Real LA. Monte Carlo assessments of goodness-of-fit for ecological simulation models. Ecol Model. 2003;164(1):49–63.CrossRefGoogle Scholar
  4. 4.
    Wong RK, Storlie CB, Lee T. A frequentist approach to computer model calibration. J Royal Stat Soc Ser B (Stat Method). (In press).Google Scholar
  5. 5.
    De Finetti B. La Previsoin: ses lois logiques, ses sources subjectives. Annales de l’Institut Henri Poincare. 1937;7:1–68.Google Scholar
  6. 6.
    Ramsey FP. Truth and probability. In: Braithwaite RB, editor. Foundations of mathematics and other essays. London: Routledge & Keegan Paul; 1931.Google Scholar
  7. 7.
    Raiffa H, Schaifer R. Applied statistical decision theory. Boston: Harvard Business School; 1961.Google Scholar
  8. 8.
    van der Steen A, van Rosmalen J, Kroep S, et al. Calibrating parameters for microsimulation disease models: a review and comparison of different goodness-of-fit criteria. Med Decis Making. 2016;36(5):652–65.CrossRefGoogle Scholar
  9. 9.
    Brown T, Grassly NC, Garnett G, Stanecki K. Improving projections at the country level: the UNAIDS Estimation and Projection Package 2005. Sex Transm Infect. 2006;82(Suppl. 3):iii34–40.Google Scholar
  10. 10.
    Gilbert JA, Meyers LA, Galvani AP, Townsend JP. Probabilistic uncertainty analysis of epidemiological modeling to guide public health intervention policy. Epidemics. 2014;6:37–45.CrossRefGoogle Scholar
  11. 11.
    Weinstein MC, Siegel JE, Gold MR, et al. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA. 1996;276(15):1253–8.CrossRefGoogle Scholar
  12. 12.
    Briggs AH, Weinstein MC, Fenwick EA, et al. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Med Decis Making. 2012;32(5):722–32.CrossRefGoogle Scholar
  13. 13.
    Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.CrossRefGoogle Scholar
  14. 14.
    Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Econ. 1999;8(3):257–61.CrossRefGoogle Scholar
  15. 15.
    Fenwick E, O’Brien BJ, Briggs A. Cost-effectiveness acceptability curves: facts, fallacies and frequently asked questions. Health Econ. 2004;13(5):405–15.CrossRefGoogle Scholar
  16. 16.
    Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res. 2001;10(4):277–303.CrossRefGoogle Scholar
  17. 17.
    Ades AE, Sculpher M, Sutton A, et al. Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics. 2006;24(1):1–19.CrossRefGoogle Scholar
  18. 18.
    Jackson CH, Jit M, Sharples LD, De Angelis D. Calibration of complex models through Bayesian evidence synthesis: a demonstration and tutorial. Med Decis Making. 2015;35(2):148–61.CrossRefGoogle Scholar
  19. 19.
    Welton NJ, Ades AE. Estimation of markov chain transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis, and model calibration. Med Decis Making. 2005;25(6):633–45.CrossRefGoogle Scholar
  20. 20.
    Ades AE, Welton NJ, Caldwell D, et al. Multiparameter evidence synthesis in epidemiology and medical decision-making. J Health Serv Res Policy. 2008;13(Suppl. 3):12–22.CrossRefGoogle Scholar
  21. 21.
    R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2013.Google Scholar
  22. 22.
    Rubin D. Using the SIR algorithm to simulate posterior distributions. Bayesian Stat. 1988;3:395–402.Google Scholar
  23. 23.
    Kish L. Survey sampling. New York: Wiley; 1965.Google Scholar
  24. 24.
    Raftery AE, Bao L. Estimating and projecting trends in HIV/AIDS generalized epidemics using incremental mixture importance sampling. Biometrics. 2010;66(4):1162–73.CrossRefGoogle Scholar
  25. 25.
    Whyte S, Walsh C, Chilcott J. Bayesian calibration of a natural history model with application to a population model for colorectal cancer. Med Decis Making. 2011;31(4):625–41.CrossRefGoogle Scholar
  26. 26.
    Rutter CM, Miglioretti DL, Savarino JE. Bayesian calibration of microsimulation models. J Am Stat Assoc. 2009;104(488):1338–50.CrossRefGoogle Scholar
  27. 27.
    Lunn DJ, Thomas A, Best N, Spiegelhalter D. WinBUGS: a Bayesian modelling framework: concepts, structure, and extensibility. Stat Comput. 2000;10:325–37.CrossRefGoogle Scholar
  28. 28.
    Carpenter B, Gelman A, Hoffman M, et al. Stan: a probabilistic programming language. J Stat Softw. (In press).Google Scholar
  29. 29.
    Sunnaker M, Busetto AG, Numminen E, et al. Approximate Bayesian computation. PLoS Comput Biol. 2013;9(1):e1002803.CrossRefGoogle Scholar
  30. 30.
    Beaumont MA, Zhang W, Balding DJ. Approximate Bayesian computation in population genetics. Genetics. 2002;162(4):2025–35.PubMedPubMedCentralGoogle Scholar
  31. 31.
    Cowles MK, Carlin BP. Markov Chain Monte Carlo convergence diagnostics: a comparative review. J Am Stat Assoc. 1996;91(434):883–904.CrossRefGoogle Scholar
  32. 32.
    Brooks SP, Gelman A. General methods for monitoring convergence of iterative simulations. J Comput Graph Stat. 1998;7(4):434–55.Google Scholar
  33. 33.
    Gelman A, Meng XL, Stern H. Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica. 1996;6(4):733–60.Google Scholar
  34. 34.
    Gelman A. Exploratory data analysis for complex models. J Comput Graph Stat. 2004;13(4):755–79.CrossRefGoogle Scholar
  35. 35.
    Garrett ES, Zeger SL. Latent class model diagnosis. Biometrics. 2000;56(4):1055–67.CrossRefGoogle Scholar
  36. 36.
    Iman RL, Helton JC, Campbell JE. An approach to sensitivity analysis of computer models. Part II: ranking of input variables, response surface validation, distribution effect, and technique synopsis variable assessment. J Qual Technol. 1981;13:232–40.CrossRefGoogle Scholar
  37. 37.
    Iman RL, Helton JC, Campbell JE. An approach to sensitivity analysis of computer models. Part I: introduction, input variable selection and preliminary variable assessment. J Qual Technol. 1981;13:174–83.CrossRefGoogle Scholar
  38. 38.
    Rutter CM, Savarino JE. An evidence-based microsimulation model for colorectal cancer: validation and application. Cancer Epidemiol Biomark Prev. 2010;19(8):1992–2002.CrossRefGoogle Scholar
  39. 39.
    Farah M, Birrell P, Conti S, De Angelis D. Bayesian emulation and calibration of a dynamic epidemic model for A/H1N1 influenza. J Am Stat Assoc. 2014;109(508):1398–411.CrossRefGoogle Scholar
  40. 40.
    Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27(7):533–45.CrossRefGoogle Scholar
  41. 41.
    Kong CY, McMahon PM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value Health. 2009;12(4):521–9.CrossRefGoogle Scholar
  42. 42.
    Cevik M, Ergun MA, Stout NK, et al. Using active learning for speeding up calibration in simulation models. Med Decis Making. 2016;36(5):581–93.CrossRefGoogle Scholar
  43. 43.
    Gelman A, Shalizi CR. Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol. 2013;66:8–38.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Nicolas A. Menzies
    • 1
    • 2
    Email author
  • Djøra I. Soeteman
    • 2
  • Ankur Pandya
    • 2
    • 3
  • Jane J. Kim
    • 2
    • 3
  1. 1.Department of Global Health and PopulationHarvard T.H. Chan School of Public HealthBostonUSA
  2. 2.Center for Health Decision ScienceHarvard T.H. Chan School of Public HealthBostonUSA
  3. 3.Department of Health Policy and ManagementHarvard T.H. Chan School of Public HealthBostonUSA

Personalised recommendations