An Introduction to Experimental Criminology

Abstract

Experimental criminology is scientific knowledge about crime and justice discovered from random assignment of different conditions in large field tests. This method is the preferred way to estimate the average effects of one variable on another, holding all other variables constant (Campbell and Stanley 1963; Cook and Campbell 1979). While the experimental method is not intended to answer all the research questions in criminology, it can be used far more often than most criminologists assume (Federal Judicial Center, 1981). Opportunities are particularly promising in partnership with criminal justice agencies.

The highest and best use of experimental criminology is to develop and test theoretically coherent ideas about reducing harm (Sherman 2006, 2007), rather than just “evaluating” government programs. Those tests, in turn, can help to accumulate an integrated body of grounded theory (Glaser and Strauss 1967) in which experimental evidence plays a crucial role. When properly executed, randomized field experiments provide the ideal tests of theories about both the prevention and causation of crime.

Appendix 1

Crim-PORT 1.0:

Criminological Protocol for Operating Randomized Trials

@ 2009 by Lawrence W. Sherman and Heather Strang

INSTRUCTIONS: Please use this form to enter information directly into the WORD document as the protocol for your registration on Cambridge University’s Jerry Lee Centre of Experimental Criminology’s Registry of EXperiments in Policing Strategy and Tactics (REX-POST) or the separate Registry of Experiments in Corrections Strategy and Tactics (REX-COST) at http://www.crim.cam.ac.uk/experiments.

CONTENTS:

• 1. Name and Hypotheses

• 2. Organizational Framework

• 3. Unit of Analysis

• 4. Eligibility Criteria

• 5. Pipeline: Recruitment or Extraction of Cases

• 6. Timing

• 7. Random Assignment

• 8. Treatment and Comparison Elements

• 9. Measuring and Managing Treatments

• 10. Measuring Outcomes

• 11. Analysis Plan

• 12. Due Date and Dissemination Plan

1. Name and Hypotheses

• A. Name of Experiment _____________

• B. Principal Investigator

  (Name) _____________

  (Employer) _____________

• C. 1st Co-Principal Investigator

  (Name) _____________

  (Employer) _____________

• D. 2d Co-Principal Investigator (Name) _____________

  (Employer) _____________

• E. General Hypothesis_____________: (Experimental or Primary Treatment) _____________ causes (less or more) _____________ (crime or justice outcome) _____________ than (comparison or control treatment) _____________.

• F. Specific Hypotheses_____________:

  • 1. List all variations of treatment delivery to be tested.

  • 2. List all variations of outcome measures to be tested.

  • 3. List all subgroups to be tested for all varieties of outcome measures.

2. Organizational Framework: Check only one from a, b, c, or d

• A. In-House delivery of treatments, data collection and analysis _____________

• B. Dual Partnership: Operating agency delivers treatments with independent research organization providing random assignment, data collection, analysis _____________

  • Name of Operating Agency _____________

  • Name of Research Organization _____________

• C. Multi-Agency Partnership: Operating agencies delivers treatments with independent research organization providing random assignment, data collection, analysis

  • Name of Operating Agency

  • 1 _____________

  • Name of Operating Agency

  • 2 _____________

  • Name of Operating Agency

  • 3 _____________

  • Name of Research

  • Organization _____________

• D. Other Framework (describe in detail).

3. Unit of Analysis

Check only one

• _____________A. People (describe role: offenders, victims, etc.) _____________

• _____________B. Places (describe category: school, corner, face-block, etc.) _____________

• _____________C. Situations (describe: police-citizen encounters, fights, etc.) _____________

• _____________D. Other (describe) _____________

4. Eligibility Criteria

• A. Criteria Required (list all)

• B. Criteria for Exclusion (list all)

5. Pipeline: Recruitment or Extraction of Cases (answer all questions)

• A. Where will cases come from?

• B. Who will obtain them?

• C. How will they be identified?

• D. How will each case be screened for eligibility?

• E. Who will register the case identifiers prior to random assignment?

• F. What social relationships must be maintained to keep cases coming?

• G. Has a Phase I (no-control, “dry-run”) test of the pipeline and treatment process been conducted? If so,

  • How many cases were attempted to be treated

  • How many treatments were successfully delivered

  • How many cases were lost during treatment delivery

6. Timing: Cases come into the experiment in (check only one)

• A. A trickle-flow process, one case at a time _____________

• B. A single batch assignment _____________

• C. Repeated batch assignments _____________

• D. Other (describe below) _____________

7. Random Assignment

• A. How is random assignment sequence to be generated?

  (coin-toss, every Nth case, and other nonrandom tools are banned from CCR-RCT).

  Check one from 1, 2 or 3 below

  • 1. Random numbers table → case number sequence → sealed envelopes with case numbers outside and treatment assignment inside, with 2-sheet paper surrounding treatment _____________

  • 2. Random numbers case–treatment generator program in secure computer _____________

  • 3. Other (please describe below) _____________

• B. Who is entitled to issue random assignments of treatments?

  Role:

  Organization:

• C. How will random assignments be recorded in relation to case registration?

  Name of data base:

  Location of data entry:

  Persons performing data entry:

8. Treatment and Comparison Elements

• A. Experimental or Primary Treatment

  • 1. What elements must happen, with dosage level (if measured) indicated.

    • Element A:

    • Element B:

    • Element C:

    • Other Elements:

  • 2. What elements must not happen, with dosage level (if measured) indicated.

    • Element A:

    • Element B:

    • Element C:

    • Other Elements:

• B. Control or Secondary Comparison Treatment

  • 3. What elements must happen, with dosage level (if measured) indicated.

    • Element A:

    • Element B:

    • Element C:

    • Other Elements:

  • 4. What elements must not happen, with dosage level (if measured) indicated.

    • Element A:

    • Element B:

    • Element C:

    • Other Elements:

9. Measuring and Managing Treatments

• A. Measuring

  • 1. How will treatments be measured?

  • 2. Who will measure them?

  • 3. How will data be collected?

  • 4. How will data be stored?

  • 5. Will data be audited?

  • 6. If audited, who will do it?

  • 7. How will data collection reliability be estimated?

  • 8. Will data collection vary by treatment type?

    If so, how?

• B. Managing

  • 1. Who will see the treatment measurement data?

  • 2. How often will treatment measures be circulated to key leaders?

  • 3. If treatment integrity is challenged, whose responsibility is correction?

10. Measuring and Monitoring Outcomes

• A. Measuring

  • 1. How will outcomes be measured?

  • 2. Who will measure them?

  • 3. How will data be collected?

  • 4. How will data be stored?

  • 5. Will data be audited?

  • 6. If audited, who will do it?

  • 7. How will data collection reliability be estimated?

  • 8. Will data collection vary by treatment type?

    If so, how?

• B. Monitoring

  • 1. How often will outcome data be monitored?

  • 2. Who will see the outcome monitoring data?

  • 3. When will outcome measures be circulated to key leaders?

  • 4. If experiment finds early significant differences, what procedure is to be followed?

11. Analysis Plan

• A. Which outcome measure is considered to be the primary indicator of a difference between experimental treatment and comparison group?

• B. What is the minimum sample size to be used to analyze outcomes?

• C. Will all analyses employ an intention-to-treat framework?

• D. What is the threshold below which the percent Treatment-as-Delivered would be so low as to bar any analysis of outcomes?

• E. Who will do the data analysis?

• F. What statistic will be used to estimate effect size?

• G. What statistic will be used to calculate P values?

• H. What is the magnitude of effect needed for a P = 0. 05 difference to have an 80% chance of detection with the projected sample size (optional but recommended calculation of power curve) for the primary outcome measure.

12. Dissemination Plan

• A. What is the date by which the project agrees to file its first report on CCR-RCT? (report of delay, preliminary findings, or final result).

• B. Does the project agree to file an update every 6 months from date of first report until date of final report?

• C. Will preliminary and final results be published, in a 250-word abstract, on CCR-RCT as soon as available?

• D. Will CONSORT requirements be met in the final report for the project? (See http://www.consort-statement.org/)

• E. What organizations will need to approve the final report? (include any funders or sponsors)

• F. Do all organizations involved agree that a final report shall be published after a maximum review period of 6 months from the principal investigator’s certification of the report as final?

• G. Does principal investigator agree to post any changes in agreements affecting items 12A to 12F above?

• H. Does principal investigator agree to file a final report within 2 years of cessation of experimental operations, no matter what happened to the experiment? (e.g., “random assignment broke down after 3 weeks and the experiment was cancelled” or “only 15 cases were referred in the first 12 months and experiment was suspended”).

An Introduction to Experimental Criminology: Lawrence W. Sherman

• Background Experimental criminology (EC) is scientific knowledge about crime and justice discovered from random assignment of different conditions in large field tests.

  • 1. This method is the preferred way to estimate the average effects of one variable on another, holding all other variables constant

  • 2. While the experimental method is not intended to answer all research questions in criminology, it can be used far more often than most criminologists assume

    • Opportunities are particularly promising in partnership with criminal justice agencies

  Note: The goal of this chapter is to help its readers improve the design and conduct of criminological experiments. This chapter’s method is to describe the necessary steps and preferred decisions in planning, conducting, completing, analyzing, reporting, and synthesizing high-quality randomized controlled trials (RCTs) in criminology.

• EC use The highest and best use of experimental criminology is to develop and test theoretically coherent ideas about reducing harm (Sherman 2006, 2007), rather than just “evaluating” government programs.

  • Those tests, in turn, can help to accumulate an integrated body of grounded theory in which experimental evidence plays a crucial role.

  • The advantages depend entirely on the capability of the experimenters to insure success in achieving the many necessary elements of an unbiased comparison:

    • 1. Many randomized field experiments in criminology suffer flaws that could have been avoided with better planning.

• Metaphors for experiments The success of experimental criminology may depend on choosing the right metaphor.

  • The most useful metaphor is constructing a building.

    • The recurrent metaphor of constructing a building helps to illustrate the order of steps to take for best results.

    • The steps presented in this chapter begin with the intellectual property of every experiment: formulating the research question.

  • Once a protocol is agreed and approved, the experimenters (like builders) must find and “contract” with a wide range of agents and others to best construct and sustain the experiment.

  • When and if all these steps are completed, the experiment will be ready for analysis.

  • This chapter briefly maps out those principles and the arguments for and against fundamentally different analytic approaches in EC.

  Part 1:Intellectual Property: Formulating the Research Question

  • Great experiments Great experiments in criminology are arguably based on three criteria:

    • 1. They test theoretically central hypotheses – experimentalists can do the most good for science when they are the most focused on the theoretical implications of their experiments.

    • 2. They eliminate as many competing explanations as possible – it is the capacity to limit ambiguity by eliminating competing explanations that makes EC so important to criminological theory.

    • 3. They show one intervention to be far more cost effective than others – rising interest in this principle alone has done more to encourage evidence-based government programs than any other.

      • Experiments must be planned to measure costs of delivering programs, both in a start-up phase and in a “rollout” model with perhaps more efficiencies from mass production.

    Note: Putting these criteria together in the formulation of an experimental research question may seem to be more a matter of “art” than of science. Such a judgment would demean the importance of intuition, inspiration, and insight in science, as in many fields involving complex decisions.

  Part 2: Social Foundation: Developing a “Field Station”

  • Field stations The history of experimental field science shows many examples of research centered in what looks much like an indoor laboratory – but with a crucial difference – studies consider questions that cannot be answered in a laboratory

    • Field research stations have collected various kinds of observational data systematically in the same places for at least 300 years.

  • By the 1950s, hospitals associated with medical schools took on the same character as field stations, linking teaching and research with a large number of clinical randomized controlled trials (RCT)

  • From at least the 1960s, similar concentrations of field experiments have been found in the criminal justice system

  • The concept of a field station where data are recorded and experiments can last for many decades is an explicit vision for how to conduct experiments in criminology

  • Social elements The key to holding an experiment together is understanding a cognitive map of its social elements which include the funders, the executive leadership of an operating agency, the mid-level operating liaison person, the agents delivering treatments, and where necessary the agents providing cases.

  Part 3: Deciding on the Experimental Protocol

  • Experiment blueprints The future is clear: experimental criminology will need to design blueprints and stick to them, absent approval from oversight bodies.

    • It can be argued that EC will be substantially improved by wider use of experimental protocols.

    • One reason is so many RCTs in criminology have either violated good design standards or failed to report fundamental information.

    • CONSORT – CONsolidated Standards On Reporting of Trials can lead to better planning of experiments with protocols that anticipate reporting requirements.

    • The CONSORT checklist includes 22 reporting elements

  Note: CONSORT alone is not enough. A reporting system does not tell you how to design a protocol for an experiment before it starts. It only tells you what readers need to know after it is finished. Included in this chapter is the first version of a standard protocol format for experiments in criminology. The appendix lays out the elements of the protocol.

  Table 1

  Part 4: “Contracts”: Recruiting, Consulting, and Training of Key People

  • Using the construction metaphor, experiments must be built by contractors (who may also have been the architect) who know how to “contract” with the right people (for money or pro bono love of the research).

    • Principal investigators in experimental criminology will generally be PhD-level academics.

    • The experimental criminologist – or “random assigner,” serving as principal investigator should be seen supported and held accountable as the primary leader of the experiment

    • Other key roles are the field coordinator, the agency liaison, the data manager, and the agent supplying the cases

  Part 5: Starting and Sustaining the Experiment

  • Design How an experiment begins depends on how it must be designed.

    • There is nothing easier than launching a “batch” random assignment project, and nothing harder than launching a “trickle-flow” random assignment project.

      • 1. While months or years of preparation may be required for batch random assignment, they sometimes offer a capacity to literally push a button that will launch the experiment.

      • 2. Trickle-flow experiments, in contrast, may require the cooperation of hundreds of people to supply eligible cases for random assignment – After launch, the even greater challenge with trickle-flow experiments is to sustain the case flow.

  Part 6: Supplying: Obtaining Cases

  • Supplying cases The biggest challenge in trickle-flow experiments is to find and extract the cases that are “leaking” out of the experimental sample.

    • Increasingly, information technology applications can be used to identify the missed cases and the agents who could have referred them to the experimenters.

    • If the agency strongly supports the experiment, there may be ways to encourage agents who do not refer the cases to start doing so.

    • But if the agency will not, or cannot, attempt to persuade those who can contribute cases to an experiment, the only tool left is the ingenuity of the site coordinator or principal investigator.

  Part 7: Screening for Eligibility

  • Avoiding ineligible cases The best solution to the problem of including ineligible cases is to prevent it prior to random assignment.

    • The best time to exclude ineligible cases is when writing the budget: making sure that you spend enough money to have an independent check on the eligibility of cases.

  Part 8: Assigning Treatments

  • Assignment system Saving money on random assignment costs is penny wise and pound foolish.

    • The chance for biased selection of eligible cases can be virtually eliminated by having research staff answer the phone by a secure computer, take the identifying details of the officers and the suspects, and then open a numbered envelope sealed with red sealing wax

    • The credibility of an independent random assignment system is well worth the increased budget.

    • It is important to design a protocol that separates random assignment from operating staff.

  Part 9: Delivering Treatments Consistently

  • Protocol matters Anything that creates differences within the units of analysis, or in the way treatments are delivered, can cause a misleading result: no significant difference despite a “true,” underlying difference.

    • While eligibility criteria can limit the differences in cases (for example, in age or prior record), it is much harder for a protocol to insure consistency in the treatment

    • The best plan is to invest heavily in measurements, such as observations and interviews – even then, however, it is much harder to deliver consistency than to measure consistency

    • Repeating experiments done in Canberra Australia and then again in England showed that protocol matters and that as a result, consistency was far higher in England.

  Part 10: Measuring Treatments Delivered

  • Failure to measure The failure to measure treatment delivery is one of the most common in experimental criminology.

    • Numerous experiments assume that once treatment is assigned it will be delivered; yet when budgets are invested in measuring delivery, it shows at least some portion of cases in which delivery did not occur.

    • Great sums can be saved by using two kinds of electronic data:

      • 1. One is the Automatic Radio Locator System (ARLS) that will record where each and every police officer is at all times.

      • 2. The other technology is CCTV cameras, which are trained on many hot spots and can record what happens 100% of the time.

  Part 11: Measuring Outcomes

  • Principles Several principles can help in measuring outcomes.

    • Choose universal measures over low response rate measures

    • Choose crime frequency over prevalence

    • Choose a seriousness index over categorical counts

    • Choose one measure as primary at the outset

  Part 12: Analyzing Results

  • Analysis issues There are two issues to supplement what textbooks say on analysis of results.

    • 1. The first is to seek simplicity in analysis.

    • 2. The other is test policies, not treatments – in general, Intention-To-Treat analysis makes the most sense in keeping the analysis simple.

      • As long as the experiment is limited to the random assignment of policy and cannot control treatment, the honest thing to do is to analyze the effects of policy.

  Part 13: Communicating Results

  • Avoid complexity Simplicity is also a great virtue in communicating results.

    • Academics inclined to making fine distinctions are often impatient with simplicity, but they lack evidence to support the claim that complexity will lead to better policymaking or even better science.

  Part 14: Synthesizing Results

  • The goal of synthesis The goal of research synthesis is to draw conclusions from a universe of all tests of a single hypothesis.

    • Randomized experiments are especially valuable for systematic reviews and meta-analysis of accumulated tests of a program or policy.

    • Even those who are generally critical of the statistical basis of meta-analysis are ready to endorse its use when only randomized experiments are included in the calculations

  Part 15: Becoming a Random Assigner

  • Recruitment Who makes the best random assigner?

    • Experimental criminology requires a more extroverted personality than is needed for scholarship in general.

    • Experimental criminology may also require a greater readiness to accept the big problems that cannot be changed quickly, in order to attack smaller problems that can be.

    • The best experimental criminology will feature the best traits of scholarship in general: erudition, broad theoretical vision, a nuanced grasp of causal inference, and abiding curiosity.

  • Appendix The appendix contains a Criminological Protocol for Operating Randomized Trials.

This information map was prepared with the support of the Jerry Lee Foundation and the assistance of Herbert Fayer.