Evaluating the impact of health policies: using a difference-in-differences approach

  • Sahar Saeed
  • Erica E. M. MoodieEmail author
  • Erin C. Strumpf
  • Marina B. Klein
Hints & Kinks

Constrained healthcare resources worldwide have made evaluating the impact of population health interventions increasingly important to maximize health and equity, while minimizing costs. However, the effects of population-level exposures such as health policies can seldom be evaluated through randomized controlled trials (RCTs). The following article will examine how the difference-in-differences method can be used to estimate the causal effect of such interventions. While this method was formalized and is extensively used in the field of economics (Meyer 1995), its first application is believed to have originated in the field of public health in 1855 (Snow 1855). The difference-in-differences method emulates a randomized design by measuring changes in outcomes over time between exposed and control groups. But unlike an RCT where the researcher randomly assigns exposure status; in a difference-in-differences design, researchers use “natural experiments” to assign exposure status,...



This study was funded through support by Doctoral Awards funded to SS by the Canadian Institutes of Health Research and the Canadian Hepatitis C Network. ECS and EEMM are supported by a Chercheur boursier Junior 2 from the Fonds de Recherche Santé (FRQ-S). The Canadian HIV-HCV Coinfection Cohort Study is supported by the Fonds de recherche du Québec-Santé (FRQ-S); Réseau SIDA/maladies infectieuses, the Canadian Institutes of Health Research (CIHR FDN 143270) and the CIHR Canadian HIV Trials Network (CTN222).

Compliance with ethical standards

Conflict of interest

Authors SS, EEMM and ECS declare that they have no conflicts of interest. None of the authors feel in conflict of interest with regard to this study, and there was no pharmaceutical industry support to conduct this study although MBK has received research grants for investigator-initiated trials from Merck and ViiV Healthcare and consulting fees from ViiV Healthcare, Bristol-Meyers Squibb, Merck, Gilead and AbbVie.


  1. Abadie A (2005) Semiparametric difference-in-differences estimators. Rev Econ Stud 72:1–19CrossRefGoogle Scholar
  2. Abadie A, Diamond A, Hainmueller J (2010) Synthetic control methods for comparative case studies: estimating the effect of California’s tobacco control program. J Am Stat Assoc 105:493–505. CrossRefGoogle Scholar
  3. Bendavid E, Holmes CB, Bhattacharya J, Miller G (2012) HIV development assistance and adult mortality in Africa. JAMA 307:2060–2067. CrossRefPubMedPubMedCentralGoogle Scholar
  4. Bertrand M, Duflo E, Mullainathan S (2002) How much should we trust differences-in-differences estimates? The National Bureau of Economic Research, 8841Google Scholar
  5. Bilgel F, Galle B (2015) Financial incentives for kidney donation: a comparative case study using synthetic controls. J Health Econ 43:103–117. CrossRefPubMedGoogle Scholar
  6. Dimick JB, Ryan AM (2014) Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA 312:2401–2402. CrossRefPubMedGoogle Scholar
  7. Harper S, Strumpf EC (2017) Primary enforcement of mandatory seat belt laws and motor vehicle crash deaths American. J Prev Med 53:176–183. CrossRefGoogle Scholar
  8. Jena AB, Goldman DP, Seabury SA (2015) Incidence of sexually transmitted infections after human papillomavirus vaccination among adolescent females. JAMA Intern Med 175:617–623. CrossRefPubMedPubMedCentralGoogle Scholar
  9. King M, Essick C, Bearman P, Ross JS (2013) Medical school gift restriction policies and physician prescribing of newly marketed psychotropic medications: difference-in-differences analysis. BMJ (Clin Res Ed) 346:f264. CrossRefGoogle Scholar
  10. Kmenta J (2010) Mostly harmless econometrics: an empiricist’s companion. Springer, BerlinGoogle Scholar
  11. Kondo MC, Keene D, Hohl BC, MacDonald JM, Branas CC (2015) A difference-in-differences study of the effects of a new abandoned building remediation strategy on safety. PLoS ONE 10:e0129582. CrossRefPubMedPubMedCentralGoogle Scholar
  12. Liang KY, Zeger SL (1993) Regression analysis for correlated data. Annu Rev Public Health 14:43–68. CrossRefPubMedGoogle Scholar
  13. Lipsitch M, Tchetgen Tchetgen E, Cohen T (2010) Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology 21:383–388. CrossRefPubMedPubMedCentralGoogle Scholar
  14. Little RJ, Rubin DB (2000) Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annu Rev Public Health 21:121–145. CrossRefPubMedGoogle Scholar
  15. McCormick D, Hanchate AD, Lasser KE, Manze MG, Lin M, Chu C, Kressin NR (2015) Effect of Massachusetts healthcare reform on racial and ethnic disparities in admissions to hospital for ambulatory care sensitive conditions: retrospective analysis of hospital episode statistics. BMJ (Clin Res Ed) 350:h1480. CrossRefGoogle Scholar
  16. McKinnon B, Harper S, Kaufman JS, Bergevin Y (2015) Removing user fees for facility-based delivery services: a difference-in-differences evaluation from ten sub-Saharan African countries. Health Policy Plan 30:432–441. CrossRefPubMedGoogle Scholar
  17. Meyer BD (1995) Natural and quasi-experiments in economics. J Bus Econ Stat 13:151–161. CrossRefGoogle Scholar
  18. Noémi K, Richard G, Dominik H, James TA, Silviya N, Matt S (2016) Examination of the synthetic control method for evaluating health policies with multiple treated units health. Economics 25:1514–1528. CrossRefGoogle Scholar
  19. Puhani PA (2012) The treatment effect, the cross difference, and the interaction term in nonlinear “difference-in-differences” models. Econ Lett 115:85–87. CrossRefGoogle Scholar
  20. Raifman J, Moscoe E, Austin SB, McConnell M (2017) Difference-in-differences analysis of the association between state same-sex marriage policies and adolescent suicide attempts. JAMA Pediatr 171:350–356. CrossRefPubMedPubMedCentralGoogle Scholar
  21. Riddell CA, Kaufman JS, Hutcheon JA, Strumpf EC, Teunissen PW, Abenhaim HA (2014) Effect of uterine rupture on a hospital’s future rate of vaginal birth after cesarean delivery. Obstet Gynecol 124:1175–1181. CrossRefPubMedGoogle Scholar
  22. Ryan AM, Burgess JF Jr, Dimick JB (2015) Why we should not be indifferent to specification choices for difference-in-differences. Health Serv Res 50:1211–1235. CrossRefPubMedGoogle Scholar
  23. Saeed S, Moodie EEM, Strumpf EC, Klein MB (2018) Segmented generalized mixed effect models to evaluate health outcomes. Int J Public Health 63:547–551. CrossRefPubMedGoogle Scholar
  24. Snow J (1855) On the mode of communication of cholera, 2nd edn. John Churchill, LondonGoogle Scholar
  25. Strumpf EC, Harper S, Kaufman JS (2017) Fixed effects and difference in differences. In: Oakes JM and Kaufman JS (eds) Methods in social epidemiology. Jossey-Bass, San FranciscoGoogle Scholar
  26. Wharam JF, Landon BE, Galbraith AA, Kleinman KP, Soumerai SB, Ross-Degnan D (2007) Emergency department use and subsequent hospitalizations among members of a high-deductible health plan. JAMA 297:1093–1102. CrossRefPubMedGoogle Scholar

Copyright information

© Swiss School of Public Health (SSPH+) 2019

Authors and Affiliations

  • Sahar Saeed
    • 1
  • Erica E. M. Moodie
    • 1
    Email author
  • Erin C. Strumpf
    • 1
  • Marina B. Klein
    • 2
  1. 1.Department of Epidemiology, Biostatistics, and Occupational HealthMcGill UniversityMontrealCanada
  2. 2.Division of Infectious Diseases/Chronic Viral Illness Service, Department of Medicine, Glen SiteMcGill University Health CentreMontrealCanada

Personalised recommendations