Time series robustness checks to test the effects of the 1996 Australian firearm law on cause-specific mortality

Abstract

Objectives

Many studies utilize time series methods to identify causal effects without accounting for an underlying time trend. We show that accounting for trends changes the conclusions in the study of Chapman et al. (JAMA, 316(3), 291–299, 2016), who evaluated the impact of the Australian firearm law in 1996. We also introduce a new empirical method that tests whether their empirical strategy can actually identify a causal effect that is also useful for panel analyses.

Methods

We use national data from the Australian Bureau of Statistics, assembled in annual counts of: total firearm deaths, firearm suicides, and firearm homicides. These data are used in an independent re-analysis of the impact of the 1996 Australian firearm law that accounts for underlying stochastic trends. We then estimate a series of artificially created interruptions using interrupted times series analysis in a time frame before 1996, to test for changes in the slope of mortality across several years prior to the actual regulatory changes. This tests whether the empirical model produces effects in years other than the year of the intervention, thereby testing if the results can simply be replicated at random using other interruption years.

Results

Controlling for stochastic trends produces less statistical evidence of the impact of the firearm law on firearm mortality than previously reported by Chapman et al. (JAMA, 316(3), 291–299, 2016). Introducing artificial interruptions in 1990 through 1995 produces statistically significant decreases in all firearm-related mortality measures well above the expected type 1 error. Overall, 19 out of the 36 artificial interruption models we tested were found to be statistically significant, suggesting that the empirical model can be implemented in multiple non-intervention years with results similar to the true 1996 interruption year.

Conclusions

Current evidence showing decreases in firearm mortality after the 1996 Australian national firearm law relies on an empirical model that may have limited ability to identify the true effects of the law.

This is a preview of subscription content, log in to check access.

Fig. 1

Notes

  1. 1.

    We also stratified the pre-treatment period into two unique periods, 1979–1986 and 1986–1996, and estimated artificial interruptions in 1982 and 1993, respectively. We similarly find statistically significant impacts of the law and the interaction in Chapman et al.’s (2016) model well above expected levels. Such subsample analysis speaks to the multiple comparison problem outlined in Hansen (2001), which may be a potential concern in the results of the five artificial interruptions in the same time series.

References

  1. Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505.

    Article  Google Scholar 

  2. Andreyeva, E., & Ukert, B. (2017). Do firearm regulations work? Evidence from the Australian National Firearms Agreement.

  3. Bernal, J. L., Cummins, S., & Gasparrini, A. (2017). Interrupted time series regression for the evaluation of public health interventions: A tutorial. International Journal of Epidemiology, 46(1), 348–355.

    Google Scholar 

  4. Branas, C. C., Kastanaki, A. E., Michalodimitrakis, M., Tzougas, J., Kranioti, E. F., Theodorakis, P. N., Carr, B. G., & Wiebe, D. J. (2015). The impact of economic austerity and prosperity events on suicide in Greece: A 30-year interrupted time-series analysis. British Medical Journal Open, 5(1), 1–11.

    Google Scholar 

  5. Campbell, D. T., & Stanley, J. C. (1966). Experimental and quasi-experimental designs for research. Skokie, IL: Rand McNally.

    Google Scholar 

  6. Carmichael, F., & Ward, R. (2001). Male unemployment and crime in England and Wales. Economics Letters, 73(1), 111–115.

    Article  Google Scholar 

  7. Chapman, S., Alpers, P., Agho, K., & Jones, M. (2006). Australia’s 1996 gun law reforms: Faster falls in firearm deaths, firearm suicides, and a decade without mass shootings. Injury Prevention, 12(6), 365–372.

    Article  Google Scholar 

  8. Chapman, S., Alpers, P., & Jones, M. (2016). Association between gun law reforms and intentional firearm deaths in Australia, 1979–2013. JAMA, 316(3), 291–299.

    Article  Google Scholar 

  9. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston, MA: Houghton Mifflin.

    Google Scholar 

  10. D’Alessio, S. J., & Stolzenberg, L. (1995). The impact of sentencing guidelines on jail incarceration in Minnesota. Criminology, 33(2), 283–302.

    Article  Google Scholar 

  11. Dugan, L. (2010). Estimating effects over time for single and multiple units. In A. Piquero & D. Weisburd (Eds.), Handbook of quantitative criminology (pp. 741–763). New York, NY: Springer.

    Google Scholar 

  12. Enders, W. (2014). Applied econometric time series (4th ed.). New York, NY: Wiley.

    Google Scholar 

  13. Gagné, M., Robitaille, Y., Hamel, D., & St-Laurent, D. (2010). Firearms regulation and declining rates of male suicide in Quebec. Injury Prevention, ip-2009.

  14. Greene, W. H. (2003). Econometric analysis. Delhi, India: Pearson Education.

    Google Scholar 

  15. Hansen, B. E. (2001). The new econometrics of structural change: Dating breaks in U.S. labor productivity. The Journal of Economic Perspectives, 15, 117–128.

    Article  Google Scholar 

  16. Humphreys, D. K., Gasparrini, A., & Wiebe, D. J. (2017). Evaluating the impact of Florida’s “stand your ground” self-defense law on homicide and suicide by firearm: An interrupted time series study. JAMA Internal Medicine, 177(1), 44–50.

    Article  Google Scholar 

  17. Klieve, H., Barnes, M., & De Leo, D. (2009). Controlling firearms use in Australia: Has the 1996 gun law reform produced the decrease in rates of suicide with this method? Social Psychiatry and Psychiatric Epidemiology, 44(4), 285.

    Article  Google Scholar 

  18. Laliotis, I., Ioannidis, J. P., & Stavropoulou, C. (2016). Total and cause-specific mortality before and after the onset of the Greek economic crisis: An interrupted time-series analysis. The Lancet Public Health, 1(2), e56–e65.

    Article  Google Scholar 

  19. Lee, D. Y., & Holoviak, S. J. (2006). Unemployment and crime: An empirical investigation. Applied Economics Letters, 13(12), 805–810.

    Article  Google Scholar 

  20. Lee, W.-S., & Suardi, S. (2010). The Australian firearms buyback and its effect on gun deaths. Contemporary Economic Policy, 28(1), 65–79.

    Article  Google Scholar 

  21. McCleary, R., McDowall, D., & Bartos, B. J. (2017). Design and analysis of time series experiments. New York, NY: Oxford University Press.

    Google Scholar 

  22. Messner, S. F., Raffalovich, L. E., & McMillan, R. (2001). Economic deprivation and changes in homicide arrest rates for white and black youths, 1967–1998: A national time-series analysis. Criminology, 39(3), 591–614.

    Article  Google Scholar 

  23. Pridemore, W. A., Chamlin, M. B., & Cochran, J. K. (2007). An interrupted time-series analysis of Durkheim’s social deregulation thesis: The case of the Russian Federation. Justice Quarterly, 24(2), 271–290.

    Article  Google Scholar 

  24. Pridemore, W. A., Chamlin, M. B., & Trahan, A. (2008). A test of competing hypotheses about homicide following terrorist attacks: An interrupted time series analysis of September 11 and Oklahoma City. Journal of Quantitative Criminology, 24(4), 381–396.

    Article  Google Scholar 

  25. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York, NY: Houghton Mifflin.

    Google Scholar 

  26. Spittal, M. J., Pirkis, J., Miller, M., & Studdert, D. M. (2012). Declines in the lethality of suicide attempts explain the decline in suicide deaths in Australia. PLoS One, 7(9), e44565.

    Article  Google Scholar 

  27. Tang, C. F., & Lean, H. H. (2007). Will inflation increase crime rate? New evidence from bounds and modified Wald tests. Global Crime, 8(4), 311–323.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Benjamin Ukert.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ukert, B., Andreyeva, E. & Branas, C.C. Time series robustness checks to test the effects of the 1996 Australian firearm law on cause-specific mortality. J Exp Criminol 14, 141–154 (2018). https://doi.org/10.1007/s11292-017-9313-3

Download citation

Keywords

  • Australia
  • Firearm regulation
  • Interrupted time series
  • Methods