Mathematical simulation models are commonly used to inform health policy decisions. These health policy models represent the social and biological mechanisms that determine health and economic outcomes, combine multiple sources of evidence about how policy alternatives will impact those outcomes, and synthesize outcomes into summary measures salient for the policy decision. Calibrating these health policy models to fit empirical data can provide face validity and improve the quality of model predictions. Bayesian methods provide powerful tools for model calibration. These methods summarize information relevant to a particular policy decision into (1) prior distributions for model parameters, (2) structural assumptions of the model, and (3) a likelihood function created from the calibration data, combining these different sources of evidence via Bayes’ theorem. This article provides a tutorial on Bayesian approaches for model calibration, describing the theoretical basis for Bayesian calibration approaches as well as pragmatic considerations that arise in the tasks of creating calibration targets, estimating the posterior distribution, and obtaining results to inform the policy decision. These considerations, as well as the specific steps for implementing the calibration, are described in the context of an extended worked example about the policy choice to provide (or not provide) treatment for a hypothetical infectious disease. Given the many simplifications and subjective decisions required to create prior distributions, model structure, and likelihood, calibration should be considered an exercise in creating a reasonable model that produces valid evidence for policy, rather than as a technique for identifying a unique theoretically optimal summary of the evidence.
This is a preview of subscription content, log in to check access.
N.A.M., J.J.K., A.P., and D.I.S. conceived the paper. N.A.M. wrote the paper. J.J.K., A.P., and D.I.S. edited the paper.
Compliance with Ethical Standards
NAM received funding from National Institutes of Health/National Institute of Allergy and Infectious Diseases (Grant No. 7R01AI112438-02).
Conflict of interest
The authors report no conflicts of interest directly relevant to the content of this article.
Vanni T, Karnon J, Madan J, et al. Calibrating models in economic evaluation: a seven-step approach. Pharmacoeconomics. 2011;29(1):35–49.CrossRefGoogle Scholar
Enns EA, Cipriano LE, Simons CT, Kong CY. Identifying best-fitting inputs in health-economic model calibration: a Pareto frontier approach. Med Decis Making. 2015;35(2):170–82.CrossRefGoogle Scholar
Waller LA, Smith D, Childs JE, Real LA. Monte Carlo assessments of goodness-of-fit for ecological simulation models. Ecol Model. 2003;164(1):49–63.CrossRefGoogle Scholar
Wong RK, Storlie CB, Lee T. A frequentist approach to computer model calibration. J Royal Stat Soc Ser B (Stat Method). (In press).Google Scholar
De Finetti B. La Previsoin: ses lois logiques, ses sources subjectives. Annales de l’Institut Henri Poincare. 1937;7:1–68.Google Scholar
Ramsey FP. Truth and probability. In: Braithwaite RB, editor. Foundations of mathematics and other essays. London: Routledge & Keegan Paul; 1931.Google Scholar
Raiffa H, Schaifer R. Applied statistical decision theory. Boston: Harvard Business School; 1961.Google Scholar
van der Steen A, van Rosmalen J, Kroep S, et al. Calibrating parameters for microsimulation disease models: a review and comparison of different goodness-of-fit criteria. Med Decis Making. 2016;36(5):652–65.CrossRefGoogle Scholar
Brown T, Grassly NC, Garnett G, Stanecki K. Improving projections at the country level: the UNAIDS Estimation and Projection Package 2005. Sex Transm Infect. 2006;82(Suppl. 3):iii34–40.Google Scholar
Gilbert JA, Meyers LA, Galvani AP, Townsend JP. Probabilistic uncertainty analysis of epidemiological modeling to guide public health intervention policy. Epidemics. 2014;6:37–45.CrossRefGoogle Scholar
Weinstein MC, Siegel JE, Gold MR, et al. Recommendations of the panel on cost-effectiveness in health and medicine. JAMA. 1996;276(15):1253–8.CrossRefGoogle Scholar
Briggs AH, Weinstein MC, Fenwick EA, et al. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Med Decis Making. 2012;32(5):722–32.CrossRefGoogle Scholar
Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.CrossRefGoogle Scholar
Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Econ. 1999;8(3):257–61.CrossRefGoogle Scholar
Fenwick E, O’Brien BJ, Briggs A. Cost-effectiveness acceptability curves: facts, fallacies and frequently asked questions. Health Econ. 2004;13(5):405–15.CrossRefGoogle Scholar
Sutton AJ, Abrams KR. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res. 2001;10(4):277–303.CrossRefGoogle Scholar
Ades AE, Sculpher M, Sutton A, et al. Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics. 2006;24(1):1–19.CrossRefGoogle Scholar
Jackson CH, Jit M, Sharples LD, De Angelis D. Calibration of complex models through Bayesian evidence synthesis: a demonstration and tutorial. Med Decis Making. 2015;35(2):148–61.CrossRefGoogle Scholar
Welton NJ, Ades AE. Estimation of markov chain transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis, and model calibration. Med Decis Making. 2005;25(6):633–45.CrossRefGoogle Scholar
Ades AE, Welton NJ, Caldwell D, et al. Multiparameter evidence synthesis in epidemiology and medical decision-making. J Health Serv Res Policy. 2008;13(Suppl. 3):12–22.CrossRefGoogle Scholar
R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2013.Google Scholar
Rubin D. Using the SIR algorithm to simulate posterior distributions. Bayesian Stat. 1988;3:395–402.Google Scholar
Iman RL, Helton JC, Campbell JE. An approach to sensitivity analysis of computer models. Part II: ranking of input variables, response surface validation, distribution effect, and technique synopsis variable assessment. J Qual Technol. 1981;13:232–40.CrossRefGoogle Scholar
Iman RL, Helton JC, Campbell JE. An approach to sensitivity analysis of computer models. Part I: introduction, input variable selection and preliminary variable assessment. J Qual Technol. 1981;13:174–83.CrossRefGoogle Scholar
Rutter CM, Savarino JE. An evidence-based microsimulation model for colorectal cancer: validation and application. Cancer Epidemiol Biomark Prev. 2010;19(8):1992–2002.CrossRefGoogle Scholar
Farah M, Birrell P, Conti S, De Angelis D. Bayesian emulation and calibration of a dynamic epidemic model for A/H1N1 influenza. J Am Stat Assoc. 2014;109(508):1398–411.CrossRefGoogle Scholar
Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27(7):533–45.CrossRefGoogle Scholar
Kong CY, McMahon PM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value Health. 2009;12(4):521–9.CrossRefGoogle Scholar
Cevik M, Ergun MA, Stout NK, et al. Using active learning for speeding up calibration in simulation models. Med Decis Making. 2016;36(5):581–93.CrossRefGoogle Scholar
Gelman A, Shalizi CR. Philosophy and the practice of Bayesian statistics. Br J Math Stat Psychol. 2013;66:8–38.CrossRefGoogle Scholar