Estimating Treatment Effects: Matching Quantification to the Question

  • Thomas A. Loughran
  • Edward P. Mulvey


In criminal justice as well as other areas, practitioners and/or policy makers often wish to know whether something “works” or is “effective.” Does a certain form of family therapy reduce troubled adolescents’ involvement in crime more than what would be seen if they were on probation? Does a jail diversion policy substantially increase indicators of community adjustment for mentally ill individuals who are arrested and processed under this policy? If so, by how much?

Trying to gauge the impact of programs or policies is eminently logical for several reasons. Obviously, this type of information is important from a traditional cost-benefit perspective. Knowing the overall impact of a program in terms of tangible and measurable benefits to some target group of interest is necessary to assess whether an investment in the program buys much. For instance, a drug rehabilitation program, which requires a large fixed cost of opening plus additional considerable operating expenses, should be able to show that this investment is worth it in terms of reduced drug use or criminal activity among its clients. Quantifiable estimates about the impact of policies or programs are also important in assessing the overall social benefit of particular approaches; it is often useful to know how much a recent change in policy has affected some subgroup in an unintended way. For instance, more stringent penalties for dealing crack, rather than powdered cocaine, appears to have provided only a marginal decrease in drug trafficking at the expense of considerable racial disparity in sentencing. Informed practice and policy rests on empirical quantifications of how much outcomes shift when certain approaches or policies are put into place.


Treatment Effect Propensity Score Instrumental Variable Treatment Assignment Policy Question 


  1. Angrist JD (1990) Lifetime earnings and the Vietnam era draft lottery: evidence from social security administrative records. Am Econ Rev 80:313–335Google Scholar
  2. Angrist JD (2004) Treatment effect heterogeneity in theory and practice, The Royal Economic Society Sargan Lecture. Econ J 114:C52–C83CrossRefGoogle Scholar
  3. Angrist JD (2006) Instrumental variables methods in experimental criminological research: what, why, and how. J Exp Criminol 2:23–44CrossRefGoogle Scholar
  4. Angrist J, Imbens G, Rubin DB (1996) Identification of causal effects using instrumental variables. J Am Stat Assoc 91:444–455CrossRefGoogle Scholar
  5. Heckman JJ (1997) Instrumental variables: a study of implicit behavioral assumptions used in making program evaluations. J Hum Resour 32(2):441–462CrossRefGoogle Scholar
  6. Heckman JJ, Smith JA (1995) Assessing the case for social experiments. J Econ Perspect 9(2):85–110Google Scholar
  7. Holland PW (1986) Statistics and causal inference. J Am Stat Assoc 81:945–960CrossRefGoogle Scholar
  8. Berk RA, Sherman LW (1988) Police response to family violence incidents: an analysis of an experimental design with incomplete randomization. J Am Stat Assoc 83(401):70–76CrossRefGoogle Scholar
  9. Imbens GW, Angrist JD (1994) Identification and estimation of local average treatment effects. Econometrica 62: 467–475CrossRefGoogle Scholar
  10. LaLonde RJ (1986) Evaluating the econometric evaluations of training programs with experimental data. Am Econ Rev 76:604–620Google Scholar
  11. Manski CF (1995) Identification problems in the social sciences. Harvard University Press, CambridgeGoogle Scholar
  12. McCaffrey DF, Ridgeway G, Morral AR (2004) Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychol Methods 9(4):403–425CrossRefGoogle Scholar
  13. Needleman HL, Riess JA, Tobin MJ, Biesecker GE, Greenhouse JB (1996) Bone lead levels and delinquent behavior. J Am Med Assoc 275(5):363–369CrossRefGoogle Scholar
  14. Nevin R (2000) How lead exposure relates to temporal changes in IQ, violent crime, and unwed pregnancy. Environ Res 83(1):1–22CrossRefGoogle Scholar
  15. Neyman JS (1923) On the application of probability theory to agricultural experiments. Essay on principles. Section 9. Stat Sci 4:465–480Google Scholar
  16. Ridgeway G (2006) Assessing the effect of race bias in post-traffic stop outcomes using propensity scores. J Quant Criminol 22(1):1–29CrossRefGoogle Scholar
  17. Robins JM, Greenland S, Hu F-C (1999) Estimation of the causal effect of a time-varying exposure on the marginal mean of a repeated binary outcome. J Am Stat Assoc 94:687–700CrossRefGoogle Scholar
  18. Robins JM, Hernan MA, Brumback B (2000) Marginal structural models and causal inference in epidemiology. Epidemiology 11(5):550–560CrossRefGoogle Scholar
  19. Rosenbaum PR (2002) Observational studies, 2nd edn. Springer-Verlag, New YorkGoogle Scholar
  20. Rosenbaum P, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika 70:41–55CrossRefGoogle Scholar
  21. Rosenbaum PR, Rubin DB (1985) The bias due to incomplete matching. Biometrics 41:103–116CrossRefGoogle Scholar
  22. Rubin DB (1974) Estimating causal effects of treatments in randomized and nonrandomized studies. J Educ Psychol 66:688–701CrossRefGoogle Scholar
  23. Rubin DB (1977) Assignment to treatment groups on the basis of a covariate. J Educ Stat 2:1–26CrossRefGoogle Scholar
  24. Rubin DB (1978) Bayesian inference for causal effects: the role of randomization. Ann Stat 6:34–58CrossRefGoogle Scholar
  25. Shadish WR, Cook TD, Campbell DT (2001) Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin, BostonGoogle Scholar
  26. Sherman LW, Berk RA (1984) The specific deterrent effects of arrest for domestic assault. Am Sociol Rev 49(2):261–272CrossRefGoogle Scholar
  27. Weisburd D, Lum C, Petronsino A (2001) Does research design affect study outcomes in criminal justice? Ann Am Acad Pol Soc Sci 578:50–70CrossRefGoogle Scholar
  28. Wright JP, Dietrich KN, Ris MD, Hornung RW, Wessel SD, Lanphear BP, Ho M, Rae MN (2008) Association of prenatal and childhood blood lead concentrations with criminal arrests in early adulthood. PLoS Med 5:e101CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Thomas A. Loughran
    • 1
  • Edward P. Mulvey
    • 2
  1. 1.Department of CriminologyUniversity of South FloridaTampaUSA
  2. 2.Western Psychiatric Institute and ClinicUniversity of Pittsburgh School of MedicinePittsburghUSA

Personalised recommendations