Advertisement

A Primer for Interpreting and Designing Difference-in-Differences Studies in Higher Education Research

  • Fernando FurquimEmail author
  • Daniel Corral
  • Nicholas Hillman
Living reference work entry
Part of the Higher Education: Handbook of Theory and Research book series (HATR, volume 35)

Abstract

Though randomized control trials continue to serve as the “gold standard” of causal inference, they are neither feasible nor desirable in numerous instances. Even in the absence of randomized trials, higher education researchers have at their disposal several statistical tools for estimating causal relationships. One such method is difference-in-differences, a powerful and intuitive approach to causal evaluation that exploits variation in the timing and coverage of policies. The method lends itself well to studying higher education policies and initiatives, as these frequently diffuse over time and across space in ways that may permit for causal inference. Difference-in-differences has become one of the most widely used methods for causal inference in higher education research. We use this chapter to introduce new researchers to this method with an overview of difference-in-differences models, common threats to their validity, and robustness checks. We then present extensions of the method, including event study models and variation in treatment timing. We illustrate these methods throughout the chapter by analyzing the effect of hurricanes on enrollment at affected colleges using data from the Integrated Postsecondary Education Data System and provide Stata code for replication of the analysis.

Keywords

Policy evaluation Causal inference Quasi-experimental design Natural experiments Counterfactuals Difference-in-differences Event study models Parallel trends assumptions Fixed effects models Variation in treatment timing Heterogeneous effects Robustness checks Multiple comparison groups Clustered and bootstrapped standard errors Hurricane Katrina Geography of college choice Enrollment trends 

References

  1. Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505.CrossRefGoogle Scholar
  2. Andrews, R. J., DesJardins, S., & Ranchhod, V. (2010). The effects of the Kalamazoo promise on college choice. Economics of Education Review, 29(5), 722–737.CrossRefGoogle Scholar
  3. Angrist, J. (2004). American education research changes tack. Oxford Review of Economic Policy, 20(2), 198–212.CrossRefGoogle Scholar
  4. Angrist, J. D., & Pischke, J. S. (2009). Mostly harmless econometrics: An empiricist's companion. Princeton: Princeton University Press.CrossRefGoogle Scholar
  5. Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455.CrossRefGoogle Scholar
  6. Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2019). Synthetic difference in differences (No. w25532). National Bureau of Economic Research.Google Scholar
  7. Ashenfelter, O. (1978). Estimating the effect of training programs on earnings. The Review of Economics and Statistics, 60(1), 47–57.CrossRefGoogle Scholar
  8. Athey, S., & Imbens, G. W. (2018). Design-based analysis in difference-in-differences settings with staggered adoption (No. w24963). National Bureau of Economic Research.Google Scholar
  9. Baker, D. J. (2019). Pathways to racial equity in higher education: Modeling the antecedents of state affirmative action bans. American Educational Research Journal. Advance online publication.  https://doi.org/10.3102/0002831219833918.CrossRefGoogle Scholar
  10. Belasco, A. S., Rosinger, K. O., & Hearn, J. C. (2015). The test-optional movement at America’s selective liberal arts colleges: A boon for equity or something else? Educational Evaluation and Policy Analysis, 37(2), 206–223.CrossRefGoogle Scholar
  11. Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust differences-in- differences estimates? The Quarterly Journal of Economics, 119(1), 249–275.CrossRefGoogle Scholar
  12. Betts, J. R., & McFarland, L. L. (1995). Safe port in a storm: The impact of labor market conditions on community college enrollments. Journal of Human Resources, 30(4), 741–765.CrossRefGoogle Scholar
  13. Bielby, R. M., House, E., Flaster, A., & DesJardins, S. L. (2013). Instrumental variables: Conceptual issues and an application considering high school course taking. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. XXVIII, pp. 263–321). Dordrecht: Springer.CrossRefGoogle Scholar
  14. Brewer, M., Crossley, T. F., & Joyce, R. (2018). Inference with difference-in-differences revisited. Journal of Econometric Methods, 7(1), 1–16.CrossRefGoogle Scholar
  15. Bumphus, W. G., & Royal, A. M. (2007). In our own words: Disaster recovery post-Katrina. In A. W. Villadsen & G. E. De los Santos (Eds.), Hoping for the best while preparing for the worst: Disasters, emergencies and the community college (pp. 24–30). Phoenix: League for Innovation in the Community College.Google Scholar
  16. Charles, K., Hurst, E., & Notowidigdo, M. (2018). Housing booms and busts, labor market opportunities, and college attendance. American Economic Review, 108(10), 2947–2994.CrossRefGoogle Scholar
  17. Conley, T. G., & Taber, C. R. (2011). Inference with “`difference in differences” with a small number of policy changes. The Review of Economics and Statistics, 93(1), 113–125.CrossRefGoogle Scholar
  18. Cortes, K. E. (2010). Do bans on affirmative action hurt minority students? Evidence from the Texas Top 10% Plan. Economics of Education Review, 29(6), 1110–1124.CrossRefGoogle Scholar
  19. Dache-Gerbino, A. (2018). College desert and oasis: A critical geographic analysis of local college access. Journal of Diversity in Higher Education, 11(2), 97–116.CrossRefGoogle Scholar
  20. Daw, J. R., & Hatfield, L. A. (2018). Matching and regression to the mean in difference-in-differences analysis. Health Services Research, 53(6), 4138–4156.CrossRefGoogle Scholar
  21. Deryugina, T., Kawano, L., & Levitt, S. (2018). The economic impact of Hurricane Katrina on its victims: Evidence from individual tax returns. American Economic Journal: Applied Economics, 10(2), 202–233.Google Scholar
  22. DeWaard, J., Curtis, K., & Fussell, E. (2016). Population recovery in New Orleans after Hurricane Katrina: Exploring the potential role of stage migration in migration systems. Population and Environment, 37(4), 449–463.CrossRefGoogle Scholar
  23. Dynarski, S. M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279–288.CrossRefGoogle Scholar
  24. Foote, A., & Grosz, M. (2019). The effect of local labor market downturns on postsecondary enrollment and program choice. Education Finance and Policy. Advance online publication.Google Scholar
  25. Freyaldenhoven, S., Hansen, C., & Shapiro, J. M. (2018). Pre-event trends in the panel event-study design (No. w24565). National Bureau of Economic Research.Google Scholar
  26. Furquim, F., & Glasener, K. M. (2017). A quest for equity? Measuring the effect of QuestBridge on economic diversity at selective institutions. Research in Higher Education, 58(6), 646–671.CrossRefGoogle Scholar
  27. Furuta, J. (2017). Rationalization and student/school personhood in US college admissions: The rise of test-optional policies, 1987 to 2015. Sociology of Education, 90(3), 236–254.CrossRefGoogle Scholar
  28. Fussell, E. (2015). The long-term recovery of New Orleans’ population after Hurricane Katrina. American Behavioral Scientist, 59(10), 1231–1245.CrossRefGoogle Scholar
  29. Galea, S., Brewin, C., Gruber, M., Jones, R., King, D., King, L., McNally, R., Ursano, R., Petukhova, M., & Kessler, R. (2007). Exposure to hurricane-related stressors and mental illness after Hurricane Katrina. Archives of General Psychiatry, 64(12), 1427–1434.CrossRefGoogle Scholar
  30. Garces, L. M. (2013). Understanding the impact of affirmative action bans in different graduate fields of study. American Educational Research Journal, 50(2), 251–284.CrossRefGoogle Scholar
  31. Garces, L. M., & Mickey-Pabello, D. (2015). Racial diversity in the medical profession: The impact of affirmative action bans on underrepresented student of color matriculation in medical schools. The Journal of Higher Education, 86(2), 264–294.CrossRefGoogle Scholar
  32. Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2016). Impact evaluation in practice. Washington, DC: The World Bank.CrossRefGoogle Scholar
  33. Goodman-Bacon, A. (2018). Difference-in-differences with variation in treatment timing (No. w25018). National Bureau of Economic Research.Google Scholar
  34. Groen, J., & Polivka, A. (2008). The effect of Hurricane Katrina on the labor market outcomes of evacuees. American Economic Review, 98(2), 43–48.CrossRefGoogle Scholar
  35. Heckman, J. J., & Smith, J. A. (1999). The pre-programme earnings dip and the determinants of participation in a social programme: Implications for simple programme evaluation strategies. The Economic Journal, 109(457), 313–348.CrossRefGoogle Scholar
  36. Heller, D. E. (1999). The effects of tuition and state financial aid on public college enrollment. The Review of Higher Education, 23(1), 65–89.CrossRefGoogle Scholar
  37. Hemelt, S., & Marcotte, D. (2011). The impact of tuition increases on enrollment at public colleges and universities. Educational Evaluation and Policy Analysis, 33(4), 435–457.CrossRefGoogle Scholar
  38. Hillman, N. W. (2017). Geospatial analysis in higher education research. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (Vol. XXXII, pp. 529–575).CrossRefGoogle Scholar
  39. Hillman, N. W., & Orians, E. L. (2013). Community colleges and labor market conditions: How does enrollment demand change relative to local unemployment rates? Research in Higher Education, 54(7), 765–780.CrossRefGoogle Scholar
  40. Hillman, N. W., Tandberg, D. A., & Gross, J. P. K. (2014). Performance funding in higher education: Do financial incentives impact college completions? The Journal of Higher Education, 85(6), 826–857.CrossRefGoogle Scholar
  41. Hillman, N. W., Tandberg, D. A., & Fryar, A. H. (2015). Evaluating the impacts of “new” performance funding in higher education. Educational Evaluation and Policy Analysis, 37(4), 501–519.CrossRefGoogle Scholar
  42. Hillman, N. W., Fryar, A. H., & Crespín-Trujillo, V. (2018). Evaluating the impact of performance funding in Ohio and Tennessee. American Educational Research Journal, 55(1), 144–170.CrossRefGoogle Scholar
  43. Holland, P. W. (1986). Statistics and causal inference. Journal of the American statistical Association, 81(396), 945–960.CrossRefGoogle Scholar
  44. Hubbard, D. (2018). The impact of local labor market shocks on college choice: Evidence from plant closings in Michigan. Unpublished manuscript. Retrieved from https://www.aeaweb.org/conference/2019/preliminary/paper/Q67dtN77
  45. Iacus, S. M., King, G., & Porro, G. (2012). Causal inference without balance checking: Coarsened exact matching. Political Analysis, 20(1), 1–24.CrossRefGoogle Scholar
  46. Jaquette, O., Kramer, D. A., & Curs, B. R. (2018). Growing the pie? The effect of responsibility center management on tuition revenue. The Journal of Higher Education, 89(5), 637–676.CrossRefGoogle Scholar
  47. Jarvie, J., & Landsberg, M. (2018, September 17). When a Category None hurricane is worse than a Category 5. Los Angeles Times. Retrieved from https://www.latimes.com
  48. Johnson, G., & Rainey, S. (2007). Hurricane Katrina’s impact on three historically black colleges and universities (HBCUs): Voices from displaced students. Race, Gender & Class, 14(1), 100–119.Google Scholar
  49. Kahn-Lang, A., & Lang, K. (2019). The promise and pitfalls of differences-in-differences: Reflections on 16 and pregnant and other applications. Journal of Business & Economic Statistics, 1–14.Google Scholar
  50. Kates, R. W., Colten, C. E., Laska, S., & Leatherman, S. P. (2006). Reconstruction of New Orleans after Hurricane Katrina: a research perspective. Proceedings of the National Academy of Sciences, 103(40), 14653–14660.CrossRefGoogle Scholar
  51. Kothari, S. P., & Warner, J. B. (2007). Econometrics of event studies. In B. E. Eckbo (Ed.), Handbook of empirical corporate finance (pp. 3–36). North Holland: Elsevier.CrossRefGoogle Scholar
  52. Ladd, A., Gill, D., & Marszalek, J. (2007). Riders from the storm: Narratives of relocated New Orleans college students in the aftermath of Hurricane Katrina. Journal of Public Management & Social Policy, 13(2), 51–80.Google Scholar
  53. Lafortune, J., Rothstein, J., & Schanzenbach, D. W. (2018). School finance reform and the distribution of student achievement. American Economic Journal: Applied Economics, 10(2), 1–26.Google Scholar
  54. Lall, R. (2016). How multiple imputation makes a difference. Political Analysis, 24(4), 414–433.CrossRefGoogle Scholar
  55. Lapid, P. (2018). Expanding college access: The impact of new universities on local enrollment. Unpublished manuscript. Retrieved from https://www.ocf.berkeley.edu/~palapid/wp-content/uploads/2019/03/newU-SEA-18_11_05.pdf
  56. Lechner, M. (2010). The estimation of causal effects by difference-in-difference methods. Foundations and Trends in Econometrics, 4(3), 165–224.CrossRefGoogle Scholar
  57. Li, A. Y., & Kennedy, A. I. (2018). Performance funding policy effects on community college outcomes: Are short-term certificates on the rise? Community College Review, 46(1), 3–39.CrossRefGoogle Scholar
  58. Liang, K. Y., & Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika, 73(1), 13–22.CrossRefGoogle Scholar
  59. Long, B. T. (2011). The new financial aid policies: Their impact on access and equity for low-income students. In L. M. Stulberg & S. L. Weinberg (Eds.), Diversity in American higher education: Toward a more comprehensive approach. New York: Routledge Books.Google Scholar
  60. Lowe, S., & Rhodes, J. (2012). Community college re-enrollment after Hurricane Katrina. Journal of College Student Retention, 14(2), 227–249.CrossRefGoogle Scholar
  61. Mangan, K., & O’Leary, B. (2010). New Orleans colleges 5 years after Katrina. The Chronicle of Higher Education. https://www.chronicle.com/article/New-Orleans-Colleges-5-Years/124107. Accessed 29 May 2019.
  62. McCall, B. P., & Bielby, R. M. (2012). Regression discontinuity design: Recent developments and a guide to practice for researchers in higher education. In J. C. Smart & M. B. Paulsen (Eds.), Higher education: Handbook of theory and research (Vol. XXVII, pp. 249–290). Dordrecht: Springer.CrossRefGoogle Scholar
  63. Murnane, R., & Willett, J. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford, UK: Oxford University Press.Google Scholar
  64. Ngo, F., & Astudillo, S. (2019). California DREAM: The impact of financial aid for undocumented community college students. Educational Researcher, 48(1), 5–18.CrossRefGoogle Scholar
  65. Reynolds, C. L., & DesJardins, S. L. (2009). The use of matching methods in higher education research: Answering whether attendance at a 2-year institution results in differences in educational attainment. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. XXIV, pp. 47–97). Dordrecht: Springer.CrossRefGoogle Scholar
  66. Rios-Aguilar, C., & Titus, M. (2018). Spatial research in higher education: Expanding understanding and identifying inequities. New Directions in Institutional Research, 2018(180), 5–10.CrossRefGoogle Scholar
  67. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688.CrossRefGoogle Scholar
  68. Rubin, D. B. (2005). Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association, 100(469), 322–331.CrossRefGoogle Scholar
  69. Sacerdote, B. (2012). When the saints go marching out: Long-term outcomes for student evacuees from Hurricanes Katrina and Rita. American Economic Journal: Applied Economics, 4(1), 109–135.Google Scholar
  70. Savoie, E. J., O’Neil, R. M., & Rabban, D. M. (2007). Report of an AAUP special committee: Hurricane Katrina and New Orleans universities. Academe, 93(3), 59–126.Google Scholar
  71. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
  72. Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103(484), 1334–1344.CrossRefGoogle Scholar
  73. St. Clair, T., & Cook, T. D. (2015). Difference-in-differences methods in public finance. National Tax Journal, 68(2), 319–338.CrossRefGoogle Scholar
  74. Stange, K. (2015). Differential pricing in undergraduate education: Effects on degree production by field. Journal of Policy Analysis and Management, 34(1), 107–135.CrossRefGoogle Scholar
  75. Strauss, V. (2018, September 4). The real story of New Orleans and its charter schools. The Washington Post. Retrieved from https://www.washingtonpost.com
  76. Strezhnev, A. (2018). Semiparametric weighting estimators for multi-period difference-in-differences designs. Working paper. Retried from https://www.antonstrezhnev.com/research
  77. Tandberg, D. A., & Hillman, N. W. (2014). State higher education performance funding: Data, outcomes, and policy implications. Journal of Education Finance, 29(3), 222–243.Google Scholar
  78. Turley, R. (2009). College proximity: Mapping access to opportunity. Sociology of Education, 82(2), 126–146.CrossRefGoogle Scholar
  79. U.S. Census Bureau. (2018a). Metropolitan and micropolitan: About. Retrieved from https://www.census.gov/programs-surveys/metro-micro/about.html
  80. U.S. Census Bureau. (2018b). Metropolitan and micropolitan: Guidance for data users. Retrieved from https://www.census.gov/programs-surveys/metro-micro/guidance.html
  81. U.S. Census Bureau. (2019). Resident population in Orleans Parish, LA [LAORLE0POP]. Retrieved from FRED, Federal Reserve Bank of St. Louis; https://fred.stlouisfed.org/series/LAORLE0POP. 13 Apr 2019.
  82. U.S. Department of Education. (2019). Integrated postsecondary education data system. https://nces.ed.gov/ipeds/datacenter/Data.aspx
  83. U.S. Department of Homeland Security. (2019). OpenFEMA dataset: Disaster declarations summaries – V1. Retrieved from https://www.fema.gov/openfema-dataset-disaster-declarations-summaries-v1
  84. Vigdor, J. (2008). The economic aftermath of Hurricane Katrina. Journal of Economic Perspectives, 22(4), 135–154.CrossRefGoogle Scholar
  85. Wolfers, J. (2006). Did unilateral divorce laws raise divorce rates? A reconciliation and new results. American Economic Review, 96(5), 1802–1820.CrossRefGoogle Scholar
  86. Zhang, L. (2010). The use of panel data models in higher education policy studies. In J. C. Smart (Ed.), Higher education: Handbook of theory and research (Vol. XXV, pp. 307–349). Dordrecht: Springer.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Fernando Furquim
    • 1
    Email author
  • Daniel Corral
    • 2
  • Nicholas Hillman
    • 2
  1. 1.Center for the Study of Higher and Postsecondary Education, School of EducationUniversity of MichiganAnn ArborUSA
  2. 2.University of Wisconsin-MadisonMadisonUSA

Section editors and affiliations

  • Laura W. Perna
    • 1
  • Ann Austin
    • 2
  • Linda Eisenmann
  • Pamela Eddy
    • 3
  • Adrianna Kezar
    • 4
  • Anne-Marie Nunez
  • Shouping Hu
  • Anna Neumann
    • 5
  • Nicholas A Bowman
  • Marvin Titus
  • Nicholas Hillman
    • 6
  1. 1.Graduate School of EducationUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.Michigan State UniversityEast LansingUSA
  3. 3.College of William and MaryWilliamsburgUSA
  4. 4.School of EducationUniversity of Southern CaliforniaLos AngelesUSA
  5. 5.Higher and Postsecondary EducationTeachers College, Columbia UniversityNew York CityUSA
  6. 6.University of WisconsinMadisonUSA

Personalised recommendations