Evaluation Analytics for Public Health: Has Reducing Air Pollution Reduced Death Rates in the United States?

  • Louis Anthony Cox Jr.
  • Douglas A. Popken
  • Richard X. Sun
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 270)


An aim of applied science in general, and of epidemiology in particular, is to draw sound causal inferences from observations. For public health policy analysts and epidemiologists, this includes drawing inferences about whether historical changes in exposures have actually caused the consequences predicted for, or attributed to, them. The example of the Dublin coal-burning ban introduced in Chap.  1 suggests that accurate evaluation of the effect of interventions is not always easy, even when data are plentiful. Students are taught to develop hypotheses about causal relations, devise testable implications of these causal hypotheses, carry out the tests, and objectively report and learn from the results to refute or refine the initial hypotheses. For at least the past two decades, however, epidemiologists and commentators on scientific methods and results have raised concerns that current practices too often lead to false-positive findings and to mistaken attributions of causality to mere statistical associations (Lehrer 2012; Sarewitz 2012; Ottenbacher 1998; Imberger et al. 2011). Formal training in epidemiology may be a mixed blessing in addressing these concerns. As discussed in Chap.  2, concepts such as “attributable risk,” “population attributable fraction,” “burden of disease,” “etiologic fraction,” and even “probability of causation” are solidly based on relative risks and related measures of statistical association; they do not necessarily reveal anything about predictive, manipulative, structural, or explanatory (mechanistic) causation (e.g., Cox 2013; Greenland and Brumback 2002). Limitations of human judgment and inference, such as confirmation bias (finding what we expect to find), motivated reasoning (concluding what it pays us to conclude), and overconfidence (mistakenly believing that our own beliefs are more accurate than they really are), do not spare health effects investigators. Experts in the health effects of particular compounds are not always also experts in causal analysis, and published causal conclusions are often unwarranted, as reviewed in Chap.  2, with a pronounced bias toward finding “significant” effects where none actually exists (false positives) (Lehrer 2012; Sarewitz 2012; Ioannidis 2005; The Economist 2013).


  1. Angrist JD, Pischke J-S (2009) Mostly harmless econometrics: an empiricist’s companion. Princeton University Press, PrincetonGoogle Scholar
  2. Beelen R, Stafoggia M, Raaschou-Nielsen O et al (2014) Long-term exposure to air pollution and cardiovascular mortality: an analysis of 22 European cohorts. Epidemiology 25(3):368–378CrossRefGoogle Scholar
  3. Buka I, Koranteng S, Osornio-Vargas AR (2006) The effects of air pollution on the health of children. Paediatr Child Health 11(8):513–516Google Scholar
  4. Campbell DT, Stanley JC (1966) Experimental and quasi-experimental designs for research. Rand McNally, ChicagoGoogle Scholar
  5. Centers for Disease Control and Prevention (CDC) (2014) Wonder “compressed mortality, 1999-2010” database.
  6. Cesaroni G, Badaloni C, Gariazzo C, Stafoggia M, Sozzi R, Davoli M, Forastiere F (2013) Long-term exposure to urban air pollution and mortality in a cohort of more than a million adults in Rome. Environ Health Perspect 121(3):324–331CrossRefGoogle Scholar
  7. Clancy L, Goodman P, Sinclair H, Dockery DW (2002) Effect of air-pollution control on death rates in Dublin, Ireland: an intervention study. Lancet 360(9341):1210–1214CrossRefGoogle Scholar
  8. Cox LA Jr (2013) Improving causal inference in risk analysis. Risk Analysis 33(10):1762–1771CrossRefGoogle Scholar
  9. Dai L, Zanobetti A, Koutrakis P, Schwartz JD (2014) Associations of fine particulate matter species with mortality in the United States: a multicity time-series analysis. Environ Health Perspect 122(8):837–842CrossRefGoogle Scholar
  10. Dominici F, Greenstone M, Sunstein CR (2014) Particulate matter matters. Science 344(18):257–258CrossRefGoogle Scholar
  11. Eichler M, Didelez V (2010) On Granger causality and the effect of interventions in time series. Lifetime Data Anal 16(1):3–32CrossRefGoogle Scholar
  12. EPA (2006)Expanded expert judgment assessment of the concentration-response relationship between PM2.5 exposure and mortality.
  13. EPA (U.S. Environmental Protection Agency) (2011) The benefits and costs of the Clean Air Act from 1990 to 2020. Final report—Rev. A. Office of Air and Radiation, WashingtonGoogle Scholar
  14. Fann N, Lamson AD, Anenberg SC, Wesson K, Risley D, Hubbell BJ (2012) Estimating the national public health burden associated with exposure to ambient PM2.5 and ozone. Risk Anal 32(1):81–95CrossRefGoogle Scholar
  15. Freedman DA (2004) Graphical models for causation, and the identification problem. Eval Rev 28(4):267–293CrossRefGoogle Scholar
  16. Friedman N, Goldszmidt M (1998) Learning Bayesian networks with local structure. In: Jordan MI (ed) Learning in graphical models. MIT Press, Cambridge, pp 421–459CrossRefGoogle Scholar
  17. Friedman MS, Powell KE, Hutwagner L, Graham LM, Teague WG (2001) Impact of changes in transportation and commuting behaviors during the 1996 Summer Olympic Games in Atlanta on air quality and childhood asthma. JAMA 285(7):897–905CrossRefGoogle Scholar
  18. Gilmour S, Degenhardt L, Hall W, Day C (2006) Using intervention time series analyses to assess the effects of imperfectly identifiable natural events: a general method and example. BMC Med Res Methodol 6:16CrossRefGoogle Scholar
  19. Greenland S, Brumback B (2002) An overview of relations among causal modelling methods. Int J Epidemiol 31(5):1030–1037CrossRefGoogle Scholar
  20. Greven S, Dominici F, Zeger S (2011) AN approach to the estimation of chronic air pollution health effects using spatio-temporal information. J Am Stat Assoc 106(494):396–406CrossRefGoogle Scholar
  21. Hack CE, Haber LT, Maier A, Shulte P, Fowler B, Lotz WG, Savage RE Jr (2010) A Bayesian network model for biomarker-based dose response. Risk Anal 30(7):1037–1051CrossRefGoogle Scholar
  22. Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE, Finkelstein J (2006) The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 13(1):16–23CrossRefGoogle Scholar
  23. Harvard School of Public Health (2002) Press release: “ban on coal burning in dublin cleans the air and reduces death rates”.
  24. Health Effects Institute (HEI) (2010) Impact of improved air quality during the 1996 Summer Olympic Games in Atlanta on multiple cardiovascular and respiratory outcomes. HEI research report #148. April, 2010. Peel JL, Klein M, Dana Flanders W, Mulholland JA, Tolbert PE. Health Effects Institute, Boston.
  25. Health Effects Institute (HEI) (2013) Did the Irish coal bans improve air quality and health? HEI Update, Summer, 2013. Accessed 1 Feb 2014
  26. Helfenstein U (1991) The use of transfer function models, intervention analysis and related time series methods in epidemiology. Int J Epidemiol 20(3):808–815CrossRefGoogle Scholar
  27. Hernán MA, Robins JM (2018) Causal inference. Chapman and Hall/CRC, Boca Raton. Forthcoming. See
  28. Hill AB (1965) The environment and disease: association or causation? Proc R Soc Med 58(5):295–300Google Scholar
  29. Imberger G, Vejlby AD, Hansen SB, Møller AM, Wetterslev J (2011) Statistical multiplicity in systematic reviews of anesthesia interventions: a quantification and comparison between Cochrane and non-Cochrane reviews. PLoS One 6(12):e28422CrossRefGoogle Scholar
  30. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2(8):e124. CrossRefGoogle Scholar
  31. Kelly F, Armstrong B, Atkinson R, Anderson HR, Barratt B, Beevers S, Cook D, Green D, Derwent D, Mudway I, Wilkinson P, HEI Health Review Committee (2011) The London low emission zone baseline study. Res Rep Health Eff Inst 163:3–79Google Scholar
  32. Krstić G (2011) Apparent temperature and air pollution vs. elderly population mortality in Metro Vancouver. PLoS One 6(9):e25101CrossRefGoogle Scholar
  33. Künzli N, Tager IB (1997) The semi-individual study in air pollution epidemiology: a valid design as compared to ecologic studies. Environ Health Perspect 105(10):1078–1083CrossRefGoogle Scholar
  34. Lamm SH, Hall TA, Engel E, White LD, Ructer FH (1994) PM 10 particulates: are they the major determinant in pediatric respiratory admissions in Utah County, Utah (1985-1989)? Ann Occup Hyg 38:969–972Google Scholar
  35. Lehrer J (2012) Trials and errors: why science is failing us. Wired.
  36. Lepeule J, Laden F, Dockery D, Schwartz J (2012) Chronic exposure to fine particles and mortality: an extended follow-up of the Harvard Six Cities Study from 1974 to 2009. Environ Health Perspect 120(7):965–970CrossRefGoogle Scholar
  37. Lipsitch M, Tchetgen Tchetgen E, Cohen T (2010) Negative controls: a tool for detecting confounding and bias in observational studies. Epidemiology 21(3):383–388CrossRefGoogle Scholar
  38. Maclure M (1991) Taxonomic axes of epidemiologic study designs: a refutationist perspective. J Clin Epidemiol 44(10):1045–1053CrossRefGoogle Scholar
  39. Moore KL, Neugebauer R, van der Laan MJ, Tager IB (2012) Causal inference in epidemiological studies with strong confounding. Stat Med 31(13):1380–1404. CrossRefGoogle Scholar
  40. NHS (2012) Air pollution ‘kills 13,000 a year’ says study.
  41. Ottenbacher KJ (1998) Quantitative evaluation of multiplicity in epidemiology and public health research. Am J Epidemiol 147:615–619CrossRefGoogle Scholar
  42. Pelucchi C, Negri E, Gallus S, Boffetta P, Tramacere I, La Vecchia C (2009) Long-term particulate matter exposure and mortality: a review of European epidemiological studies. BMC Public Health 9:453CrossRefGoogle Scholar
  43. Pope CA 3rd (1989) Respiratory disease associated with community air pollution and a steel mill, Utah Valley. Am J Public Health 79(5):623–628CrossRefGoogle Scholar
  44. Powell H, Lee D, Bowman A (2012) Estimating constrained concentration–response functions between air pollution and health. Environmetrics 23(3):228–237CrossRefGoogle Scholar
  45. Rothman KJ, Greenland S (2005) Causation and causal inference in epidemiology. Am J Public Health 95 Suppl 1:S144–S150CrossRefGoogle Scholar
  46. Sarewitz D (2012) Beware the creeping cracks of bias. Nature 485:149CrossRefGoogle Scholar
  47. Savitz DA (2012) Commentary: a niche for ecologic studies in environmental epidemiology. Epidemiology 23(1):53–54CrossRefGoogle Scholar
  48. Stebbings JH Jr (1978) Panel studies of acute health effects of air pollution. II. A methodologic study of linear regression analysis of asthma panel data. Environ Res 17(1):10–32CrossRefGoogle Scholar
  49. The Economist (2013) Trouble at the lab: scientists like to think of science as self-correcting. To an alarming degree, it is not.
  50. U.S. Environmental Protection Agency (EPA) (2014)
  51. Ward AC (2009) The role of causal criteria in causal inferences: Bradford Hill’s “aspects of association”. Epidemiol Perspect Innov 6:2CrossRefGoogle Scholar
  52. Wittmaack K (2007) The big ban on bituminous coal sales revisited: serious epidemics and pronounced trends feign excess mortality previously attributed to heavy black-smoke exposure. Inhal Toxicol 19(4):343–350CrossRefGoogle Scholar
  53. Yang Y, Li R, Li W, Wang M, Cao Y, Wu Z, Xu Q (2013) The association between ambient air pollution and daily mortality in Beijing after the 2008 Olympics: a time series study. PLoS One 8(10):e76759CrossRefGoogle Scholar
  54. Yim SH, Barrett SR (2012) Public health impacts of combustion emissions in the United Kingdom. Environ Sci Technol 46(8):4291–4296CrossRefGoogle Scholar
  55. Zeger SL, Dominici F, McDermott A, Samet JM (2008) Mortality in the Medicare population and chronic exposure to fine particulate air pollution in urban centers (2000-2005). Environ Health Perspect 116(12):1614–1619. CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Louis Anthony Cox Jr.
    • 1
  • Douglas A. Popken
    • 2
  • Richard X. Sun
    • 3
  1. 1.Cox AssociatesDenverUSA
  2. 2.Cox AssociatesLittletonUSA
  3. 3.Cox AssociatesEast BrunswickUSA

Personalised recommendations