Trusting the Results of Model-Based Economic Analyses: Is there a Pragmatic Validation Solution?
Models have become a nearly essential component of health technology assessment. This is because the efficacy and safety data available from clinical trials are insufficient to provide the required estimates of impact of new interventions over long periods of time and for other populations and subgroups. Despite more than five decades of use of these decision-analytic models, decision makers are still often presented with poorly validated models and thus trust in their results is impaired. Among the reasons for this vexing situation are the artificial nature of the models, impairing their validation against observable data, the complexity in their formulation and implementation, the lack of data against which to validate the model results, and the challenges of short timelines and insufficient resources. This article addresses this crucial problem of achieving models that produce results that can be trusted and the resulting requirements for validation and transparency, areas where our field is currently deficient. Based on their differing perspectives and experiences, the authors characterize the situation and outline the requirements for improvement and pragmatic solutions to the problem of inadequate validation.
The authors thank Isaac Coro Ramos, Pepijn Vemer, George A.K van Voorn, Maiwenn J. AI, Talitha L. Feenstra, and Chloé Herpin for their useful comments and suggestions.
SG drafted Sects. 2 and 3.1; MS drafted Sect. 3.2; JM drafted Sect. 3.3; and JJC reviewed these materials and integrated them into the paper. All authors participated in writing and editing Sects. 1, 2, and 4.
Compliance with Ethical Standards
No funding was received for the preparation of this article.
Conflict of interest
Salah Ghabri and Matt Stevenson have no conflicts of interest that are directly relevant to the contents of this article. Jörgen Möller and J. Jaime Caro are employed by Evidera, a company that provides consulting and other research services to pharmaceutical, device, government, and non-government organizations. The opinions expressed in this article are those of the authors and do not necessarily represent the views of their institutions.
- 6.Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 4th ed. Ottawa: Canadian Agency for Drugs and Technologiesin Health; 2017. https://www.cadth.ca/dv/guidelineseconomic-evaluation-health-technologies-canada-4th-edition. Accessed 15 Oct 2017.
- 7.National Institute for Health and Care Excellence (NICE). Single technology appraisal: user guide for company evidence submission template; 2015. https://www.nice.org.uk/process/pmg24/chapter/cost-effectiveness. Accessed 15 Oct 2017.
- 8.Belgian Health Care Knowledge Centre (KCE). Belgian guidelines for economic evaluations and budget impact analysis; 2015. https://kce.fgov.be/sites/default/files/page_documents/KCE_183_economic_evaluations_second_edition_Report.pdf. Accessed 6 Aug 2018.
- 9.Haute Autorité de Santé (HAS). Choices in methods for economic evaluation; 2012. https://www.has-sante.fr/portail/upload/docs/application/pdf/2012-10/choices_in_methods_for_economic_evaluation.pdf. Accessed 15 Oct 2017.
- 10.Department of Health, Commonwealth of Australia. Guidelines for preparing submissions to the Pharmaceutical Benefits Advisory Committee (PBAC), version 5.0; 2016. https://pbac.pbs.gov.au/content/information/files/pbac-guidelines-version-5.pdf. Accessed 6 Aug 2018.
- 11.Zorginstitut Nederland. Guideline for the conduct of economic evaluations in healthcare; 2016. https://english.zorginstituutnederland.nl/publications/reports/2016/06/16/guideline-for-economic-evaluations-in-healthcare. Accessed 6 Aug 2018.
- 13.Journal Officiel de la République Francaise. Décret n°2012-1116 du 2 octobre 2012 relatif aux missions médico-économiques de la Haute Autorité de Santé. JORF n°0231 du 4 octobre 2012; page 15522 texte n°8. https://www.legifrance.gouv.fr/;jsessionid=EEAEBD3C6A9DB9AFF25516B14F46DBDC.tplgfr32s_3. Accessed 6 Aug 2018.
- 14.Ghabri S, Herpin C. Economic model validation: a pilot study on manufacturers submissions. In: Presented at ISPOR 20th Annual European Congress; 2017. https://www.ispor.org/docs/default-source/presentations/1328.pdf?sfvrsn=bba5258b_1. Accessed 6 Aug 2018.
- 18.Huang M, Latimer N, Zhang Y, et al. Estimating the long-term outcomes associated with immuno-oncology therapies: challenges and approaches for overall survival extrapolations. Value Outcomes Spotlight. 2018;2018:28–30.Google Scholar
- 19.National Institute for Health and Care Excellence (NICE). Guide to the processes of technology appraisal. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-technology-appraisals/technology-appraisal-processes-guide-apr-2018.pdf. Accessed 6 Aug 2018.
- 21.Tikhonova I, Hoyle MW, Snowsill TM, Cooper C, Varley-Campbell JL, Rudin CE, Mujica Mota RE. Azacitidine for treating acute myeloid leukaemia with more than 30 % bone marrow blasts: an Evidence Review Group perspective of a National Institute for Health and Care Excellence Single Technology Appraisal. Pharmacoeconomics. 2017;35:363–73.CrossRefGoogle Scholar
- 22.National Institute for Health and Care Excellence (NICE). Regorafenib for previously treated advanced hepatocellular carcinoma. https://www.nice.org.uk/guidance/ta514/documents/final-appraisal-determination-document. Accessed 6 Aug 2018.
- 23.National Institute for Health and Care Excellence (NICE). Cenegermin for treating neurotrophic keratitis. https://www.nice.org.uk/guidance/gid-ta10131/documents/appraisal-consultation-document. Accessed 6 Aug 2018.
- 24.National Institute for Health and Care Excellence (NICE). Golimumab for treating non-radiographic axial spondyloarthritis. https://www.nice.org.uk/guidance/ta497. Accessed 6 Aug 2018.
- 25.Ransohoff DF, Feinstein AR. Editorial: is decision modeling useful in clinical medicine. Yale J Biol Med. 1976;41:761–7.Google Scholar
- 28.Brailsford SC, Hilton NA. A comparison of discrete event simulation and system dynamics for modelling health care systems. In: Riley J, editor. Planning for the future: health service quality and emergency accessibility. Operational Research Applied to Health Services (ORAHS). Glasgow: Glasgow Caledonian University; 2001.Google Scholar
- 30.Wang J, Carroll JM. Behind Linus’s Law: a preliminary analysis of open source software peer review practices in Mozilla and Python. In: Proceedings of the 2011 international conference on collaboration technologies and systems; 2011; p. 117–24.Google Scholar