Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot

Abstract

Objectives

In 2013, the Chicago Police Department conducted a pilot of a predictive policing program designed to reduce gun violence. The program included development of a Strategic Subjects List (SSL) of people estimated to be at highest risk of gun violence who were then referred to local police commanders for a preventive intervention. The purpose of this study is to identify the impact of the pilot on individual- and city-level gun violence, and to test possible drivers of results.

Methods

The SSL consisted of 426 people estimated to be at highest risk of gun violence. We used ARIMA models to estimate impacts on city-level homicide trends, and propensity score matching to estimate the effects of being placed on the list on five measures related to gun violence. A mediation analysis and interviews with police leadership and COMPSTAT meeting observations help understand what is driving results.

Results

Individuals on the SSL are not more or less likely to become a victim of a homicide or shooting than the comparison group, and this is further supported by city-level analysis. The treated group is more likely to be arrested for a shooting.

Conclusions

It is not clear how the predictions should be used in the field. One potential reason why being placed on the list resulted in an increased chance of being arrested for a shooting is that some officers may have used the list as leads to closing shooting cases. The results provide for a discussion about the future of individual-based predictive policing programs.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. 1.

    This is only one component of the larger long-term collaboration which broadly explores whether and how crime can be predicted.

  2. 2.

    Here, “numerical instability” refers to the likelihood that the estimates that an SSL member’s calculated risk of a specified thousands of times more likely to be killed is due more to statistical artifacts from fitting the quadratic curve than an accurate estimate.

  3. 3.

    Initially, CPD said they would put all the highest-risk individuals on the list; however, they decided to vet the list through their Deployment Operations Center (DOC), who made some changes to who would appear on the list, and therefore, the 426 individuals did not represent the highest scoring individuals based on the model.

  4. 4.

    Details of GVRS are available at http://directives.chicagopolice.org/directives/data/a7a57bf0-136d1d31-16513-6d1d-382b311ddf65fd3a.html

  5. 5.

    It is important to note that Chicago has gone through transformative changes over this time period, including a new Superintendent in 2011 and the integration of COMPSTAT to provide oversight. In addition to changes in leadership and management style, CPD has implemented a large number of homicide reduction strategies during this time period, including the multiple changes to the Gang Violence Reduction Strategy, and gang call-ins across different districts starting in 2010.

  6. 6.

    While there is always the possibility that the groups are different on unobservable variables, we have captured many of the important research-validated criminogenic factors. That is why we specify the approach reduces, rather than eliminates, bias.

  7. 7.

    With the exception of the winter of 2012 which did not experience the same degree of a “cooling off” period.

  8. 8.

    Most arrestees were not incapacitated for any significant period of time, but rather were booked into the Cook County jail and released within a few hours to a few days.

References

  1. Abrahamse, A. F., Ebener, P. A., Greenwood, P. W., Fitzgerald, N., & Kosin, T. E. (1991). An experimental evaluation of the Phoenix repeat offender program. Justice Quarterly, 8(2), 141–168.

    Article  Google Scholar 

  2. Aitchison, J., & Dunsmore, I. R. (1980). Statistical prediction analysis. CUP Archive.

  3. Auerhahn, K. (1999). Selective incapacitation and the problem of prediction. Criminology, 37(4), 703–734.

    Article  Google Scholar 

  4. Bar-Hillel, M. (1980). The base-rate fallacy in probability judgments. Acta Psychologica, 44(3), 211–233.

    Article  Google Scholar 

  5. Beck, C., & McCue, C. (2009). Predictive policing: what can we learn from Wal-Mart and Amazon about fighting crime in a recession? Police Chief, 76(11), 18.

    Google Scholar 

  6. Becker, G. S. (1993). Nobel lecture: the economic way of looking at behavior. Journal of Political Economy, 101, 385–409.

    Article  Google Scholar 

  7. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B: Methodological, 57, 289–300.

    MathSciNet  MATH  Google Scholar 

  8. Berk, R. (2008). Forecasting methods in crime and justice. Annual Review of Law and Social Science, 4, 219–238.

    Article  Google Scholar 

  9. Berk, R. (2011). Asymmetric loss functions for forecasting in criminal justice settings. Journal of Quantitative Criminology, 27(1), 107–123.

    Article  Google Scholar 

  10. Berk, R., & Bleich, J. (2013). Statistical procedures for forecasting criminal behavior. Criminology and Public Policy, 12(3), 513–544.

    Article  Google Scholar 

  11. Berk, R., Sherman, L., Barnes, G., Kurtz, E., & Ahlman, L. (2009). Forecasting murder within a population of probationers and parolees: a high stakes application of statistical learning. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(1), 191–211.

    MathSciNet  Article  Google Scholar 

  12. Berry, M. J., & Linoff, G. S. (2004). Data mining techniques: for marketing, sales, and customer relationship management. Indianapolis, IN: John Wiley & Sons.

  13. Blumstein, A. (1986). Criminal Careers and “Career Criminals” (Vol. 2). Washington, DC: National Academies.

  14. Bordua, D. J., & Reiss, A. J., Jr. (1966). Command, control, and charisma: reflections on police bureaucracy. American Journal of Sociology, 72, 68–76.

    Article  Google Scholar 

  15. Braga, A. (2005). Hot spots policing and crime prevention: a systematic review of randomized controlled trials. Journal of Experimental Criminology, 1(3), 317–342.

    Article  Google Scholar 

  16. Braga, A., & Weisburd, D. L. (2010). Policing problem places: Crime hot spots and effective prevention. New York, NY: Oxford University Press on Demand.

  17. Braga, A., & Weisburd, D. L. (2012). The effects of focused deterrence strategies on crime: a systematic review and meta-analysis of the empirical evidence. Journal of Research in Crime and Delinquency, 49(3), 323–358.

    Article  Google Scholar 

  18. Braga, A., Papachristos, A. V., & Hureau, D. M. (2012). The effects of hot spots policing on crime: An updated systematic review and meta-analysis. Justice Quarterly, 1–31.

  19. Bruinsma, G., & Weisburd, D. (2014). Encyclopedia of Criminology and Criminal Justice. New York: Springer.

    Google Scholar 

  20. Caldwell, M. F., Vitacco, M., & Van Rybroek, G. J. (2006). Are violent delinquents worth treating? A cost–benefit analysis. Journal of Research in Crime and Delinquency, 43(2), 148–168.

    Article  Google Scholar 

  21. Chicago Police Department. (2014).Gang Violence Reuction Strategy: General Order G10-01. Chicago, IL.

  22. Chinman, M., Imm, P., & Wandersman, A. (2004). Getting To Outcomes™ 2004. Santa Monica: Rand Corporation.

    Google Scholar 

  23. Cohen, J., Gorr, W. L., & Olligschlaeger, A. M. (2007). Leading indicators and spatial interactions: a crime‐forecasting model for proactive police deployment. Geographical Analysis, 39(1), 105–127.

    Article  Google Scholar 

  24. Cope, N. (2004). ‘Intelligence led policing or policing led intelligence?’Integrating volume crime analysis into policing. British Journal of Criminology, 44(2), 188–203.

    Article  Google Scholar 

  25. Cornish, D. B., & Clarke, R. V. (2014). The reasoning criminal: Rational choice perspectives on offending. New Brunswick, NJ: Transaction Publishers.

  26. Dvoskin, J. A., & Heilbrun, K. (2001). Risk assessment and release decision-making: Toward resolving the great debate. American Academy of Psychiatry and the Law, 29, 6–10.

    CAS  Google Scholar 

  27. Eck, J., Chainey, S., Cameron, J., & Wilson, R. (2005). Mapping crime: Understanding hotspots (Vol. NCJ 209393). Washington, DC: National Institute of Justice.

    Google Scholar 

  28. Erbentraut, J. (2014). Chicago’s controversial new police program prompts fear of racial profiling. The Huffington Post.

  29. Foster, E. M., & Jones, D. (2006). Can a costly intervention be cost-effective?: an analysis of violence prevention. Archives of General Psychiatry, 63(11), 1284–1291.

    PubMed  PubMed Central  Article  Google Scholar 

  30. Funk, M. J., Westreich, D., Wiesen, C., Stürmer, T., Brookhart, M. A., & Davidian, M. (2011). Doubly robust estimation of causal effects. American Journal of Epidemiology, 173(7), 761–767.

    PubMed  PubMed Central  Article  Google Scholar 

  31. Gendreau, P., Little, T., & Goggin, C. (1996). A meta-anallysis of the predictors of adult offender recidivism: what works! Criminology, 34(4), 575–608.

    Article  Google Scholar 

  32. Gorr, W., & Harries, R. (2003). Introduction to crime forecasting. International Journal of Forecasting, 19(4), 551–555.

    Article  Google Scholar 

  33. Gottfredson, M., & Hirschi, T. (1986). The true value of Lambda would appear to be zero: an essay on career criminals, criminal careers, selective incapacitation, cohort studies, and related topics*. Criminology, 24(2), 213–234.

    Article  Google Scholar 

  34. Greenwood, P. W., & Abrahamse, A. F. (1982). Selective incapacitation. Santa Monica: Rand Corporation.

    Google Scholar 

  35. Groff, E. R., & La Vigne, N. G. (2002). Forecasting the future of predictive crime mapping. Crime Prevention Studies, 13, 29–58.

    Google Scholar 

  36. Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: the clinical–statistical controversy. Psychology, Public Policy, and Law, 2(2), 293.

    Article  Google Scholar 

  37. Huber, P. J. (1973). Robust regression: asymptotics, conjectures and Monte Carlo. The Annals of Statistics, 1, 799–821.

    MathSciNet  MATH  Article  Google Scholar 

  38. Hunt, P., Saunders, J., & Hollywood, J. S. (2014). Evaluation of the Shreveport Predictive Policing Experiment. Santa Monica: RAND Corporation.

    Google Scholar 

  39. Kang, J. D., & Schafer, J. L. (2007). Demystifying double robustness: a comparison of alternative strategies for estimating a population mean from incomplete data. Statistical Science, 25, 523–539.

  40. Kennedy, D. M. (1996). Pulling levers: chronic offenders, high-crime settings, and a theory of prevention. Valparaiso University Law Review, 31, 449.

    Google Scholar 

  41. Kovandzic, T. V., Sloan, J. J., III, & Vieraitis, L. M. (2004). “Striking out” as crime reduction policy: the impact of “three strikes” laws on crime rates in US cities. Justice Quarterly, 21(2), 207–239.

    Article  Google Scholar 

  42. Lewin, J., & Wernick, M. (2015). Chicago Police Department Data Analytics and Predictive Policing. Paper presented at the International Association of Chief’s of Police, Chicago, IL.

  43. Lipsey, M. W. (1999). Can intervention rehabilitate serious delinquents? The Annals of the American Academy of Political and Social Science, 564(1), 142–166.

    Article  Google Scholar 

  44. Litwack, T. R., & 2. (2001). Actuarial versus clinical assessments of dangerousness. Psychology, Public Policy, and Law, 7, 409.

    Article  Google Scholar 

  45. Llenas, B. (2014). Brave New World of “Predictive Policing” Raises Specter of High-Tech Racial Profiling, Fox News Latino.

  46. Loeber, R., & Farrington, D. P. (1998). Serious and violent juvenile offenders: Risk factors and successful interventions. Thousand Oaks, CA: Sage Publications.

  47. Lum, C., Koper, C. S., & Telep, C. W. (2011). The evidence-based policing matrix. Journal of Experimental Criminology, 7(1), 3–26.

    Article  Google Scholar 

  48. Martin, S. E., & Sherman, L. W. (1986). Selective apprehension: a police strategy for repeat offenders. Criminology, 24, 155.

    Article  Google Scholar 

  49. Mazerolle, L. G., Kadleck, C., & Roehl, J. (1998). Controlling drug and disorder problems: the role of place managers. Criminology, 36(2), 371–404.

    Article  Google Scholar 

  50. Mazerolle, L. G., Ready, J., Terrill, W., & Waring, E. (2000). Problem-oriented policing in public housing: the Jersey City evaluation. Justice Quarterly, 17(1), 129–158.

    Article  Google Scholar 

  51. McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403.

    PubMed  Article  Google Scholar 

  52. McCleary, R., Hay, R. A., Meidinger, E. E., & McDowall, D. (1980). Applied time series analysis for the social sciences. Beverly Hills: Sage Publications.

    Google Scholar 

  53. McCord, J. (2003). Cures that harm: unanticipated outcomes of crime prevention programs. The Annals of the American Academy of Political and Social Science, 587(1), 16–30.

    Article  Google Scholar 

  54. McGarrell, E. F., Chermak, S., Wilson, J. M., & Corsaro, N. (2006). Reducing homicide through a “lever‐pulling” strategy. Justice Quarterly, 23(02), 214–231.

    Article  Google Scholar 

  55. Mohler, G. O., Short, M. B., Malinowski, S., Johnson, M., Tita, G. E., Bertozzi, A. L., & Brantingham, P. J. (2015). Randomized controlled field trials of predictive policing. Journal of the American Statistical Association, 110(512), 1399–1411.

  56. Ohri, A. (2013). Forecasting and Time Series Models R for Business Analytics (pp. 241–258). Springer.

  57. Papachristos, A. (2009). Murder by structure: dominance relations and the social structure of gang homicide. American Journal of Sociology, 115(1), 74–128.

    MathSciNet  Article  Google Scholar 

  58. Papachristos, A. V., & Kirk, D. S. (2015). Changing the street dynamic. Criminology and Public Policy, 14(3), 525–558.

    Article  Google Scholar 

  59. Papachristos, A., Braga, A., & Hureau, D. (2011). Six-degrees of violent victimization: Social networks and the risk of gunshot injury.

  60. Papachristos, A., Braga, A., & Hureau, D. (2012). Social networks and the risk of gunshot injury. Journal of Urban Health, 89(6), 992–1003.

    PubMed  PubMed Central  Article  Google Scholar 

  61. Pate, T., Bowers, R. A., & Parks, R. (1976). Three approaches to criminal apprehension in Kansas City: An evaluation report. Washington, DC: Police Foundation.

    Google Scholar 

  62. Perry, W. L., McInnis, B., Price, C. C., Smith, S. C., & Hollywood, J. S. (2013). Predictive policing: The role of crime forecasting in law enforcement operations. Santa Monica, CA: Rand Corporation.

  63. Quinsey, V. L., Harris, G. T., & Rice, M. E. (2000). Violent Offenders: Appraising and Managing Risk. Psychiatric Services, 51(3), 395

  64. Ratcliffe, J. (2002). Intelligence-led policing and the problems of turning rhetoric into practice. Policing and Society, 12(1), 53–66.

    MathSciNet  Article  Google Scholar 

  65. Ratcliffe, J. (2005). The effectiveness of police intelligence management: a New Zealand case study. Police Practice and Research, 6(5), 435–451.

    Article  Google Scholar 

  66. Ratcliffe, J. H. (2012). Intelligence-led policing. New York, NY: Routledge.

  67. Ratcliffe, J. H., & Guidetti, R. (2008). State police investigative structure and the adoption of intelligence-led policing. Policing: An International Journal of Police Strategies and Management, 31(1), 109–128.

    Article  Google Scholar 

  68. Ridgeway, G. (2013). Linking prediction and prevention. Criminology and Public Policy, 12(3), 545–550.

    Article  Google Scholar 

  69. Ridgeway, G., & MacDonald, J. M. (2009). Doubly robust internal benchmarking and false discovery rates for detecting racial bias in police stops. Journal of the American Statistical Association, 104(486), 661–668.

    MathSciNet  CAS  MATH  Article  Google Scholar 

  70. Ridgeway, G., Braga, A. A., Tita, G., & Pierce, G. L. (2011). Intervening in gun markets: an experiment to assess the impact of targeted gun-law messaging. Journal of Experimental Criminology, 7(1), 103–109.

    Article  Google Scholar 

  71. Ridgeway, G., McCaffrey, D., Morral, A., Burgette, L., & Griffin, B. A. (2014). Toolkit for Weighting and Analysis of Nonequivalent Groups: A tutorial for the twang package. R vignette. RAND.

  72. Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

    MathSciNet  MATH  Article  Google Scholar 

  73. Sailor, W., Dunlap, G., Sugai, G., & Horner, R. (2008). Handbook of positive behavior support. New York, NY: Springer.

  74. Sherman, L. W. (1986). Policing communities: what works? Crime and justice, 343–386.

  75. Sherman, L. W. (1992). The influence of criminology on criminal law: evaluating arrests for misdemeanor domestic violence. Journal of Criminal Law and Criminology, 83, 1–45.

    Article  Google Scholar 

  76. Sherman, L. W., & Berk, R. A. (1984). The specific deterrent effects of arrest for domestic assault. American Sociological Review, 49, 261–272.

    CAS  PubMed  Article  Google Scholar 

  77. Sherman, L. W., & Weisburd, D. (1995). General deterrent effects of police patrol in crime “hot spots”: a randomized, controlled trial. Justice Quarterly, 12(4), 625–648.

    Article  Google Scholar 

  78. Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising: A report to the United States Congress. Washington, DC: US Department of Justice, Office of Justice Programs.

    Google Scholar 

  79. Silver, E., & Miller, L. L. (2002). A cautionary note on the use of actuarial risk assessment tools for social control. Crime and Delinquency, 48(1), 138–161.

    Article  Google Scholar 

  80. Starr, S. B. (2014). Evidence-based sentencing and the scientific rationalization of discrimination. Stanford Law Review, 66, 803–953.

    Google Scholar 

  81. Stroud, M. (2014). The Minority Report: Chicago’s new police computer predicts crimes, but is it racist? The Verge.

  82. Tonry, M. (1987). Prediction and classification: legal and ethical issues. Crime and Justice, 9, 367–413.

    Article  Google Scholar 

  83. Weisburd, D., & Mazerolle, L. G. (2000). Crime and disorder in drug hot spots: implications for theory and practice in policing. Police Quarterly, 3(3), 331–349.

    Article  Google Scholar 

  84. Welsh, B. C., & Rocque, M. (2014). When crime prevention harms: a review of systematic reviews. Journal of Experimental Criminology, 10(3), 245–266.

    Article  Google Scholar 

  85. Wright, K. N., Clear, T. R., & Dickson, P. (1984). Universal applicability of probation risk‐assessment instruments. Criminology, 22(1), 113–134.

    Article  Google Scholar 

  86. Yang, M., Wong, S. C., & Coid, J. (2010). The efficacy of violence prediction: a meta-analytic comparison of nine risk assessment tools. Psychological Bulletin, 136(5), 740.

    PubMed  Article  Google Scholar 

  87. Zikopoulos, P., & Eaton, C. (2011). Understanding big data: Analytics for enterprise class hadoop and streaming data. McGraw-Hill Osborne Media.

Download references

Acknowledgments

We would like to thank the Chicago Police Department and Dr. Miles Wernick from the Illinois Institute of Technology for their participation and support of this evaluation. We would also like to acknowledge research assistance provided by Sam Cooper and Alessandra Sienra-Canas. This publication was made possible by Award Number 2009-IJ-CX-K114 - Predictive Policing Analytic & Evaluation Research Support awarded by the National Institute of Justice, Office of Justice Programs. The opinions, findings, conclusions and recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the Department of Justice.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jessica Saunders.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Saunders, J., Hunt, P. & Hollywood, J.S. Predictions put into practice: a quasi-experimental evaluation of Chicago’s predictive policing pilot. J Exp Criminol 12, 347–371 (2016). https://doi.org/10.1007/s11292-016-9272-0

Download citation

Keywords

  • Predictive policing
  • Program evaluation
  • Propensity score matching
  • Quasi-experimental design
  • Risk assessment
  • Time series analysis