Response to: “Evaluating energy efficiency policy: understanding the ‘energy policy epistemology’ may explain the lack of demand for randomized controlled trials,” by Adam Cooper, Energy Efficiency, published online 26 January 2018
- 129 Downloads
In January 2018, Adam Cooper (Energy Efficiency, 2018) published a paper that principally responds to a paper by Vine et al. (Energy Efficiency, 7, 627–640, 2014) that makes the case for promoting greater use of experimental research designs, such as randomized controlled trials (RCTs), in the evaluation of energy efficiency policy. Cooper focuses on the barriers that public policy administrators face—arguing that there are alternative, rational reasons why RCTs are not commonly used in energy policy, providing an overlapping but somewhat different “energy policy epistemology” than in Vine (2014).
KeywordsExperimental design Evaluation Energy policy Epistemology RCT Longitudinal panel design
We appreciate Cooper’s contribution to energy policy epistemology with the view that additional examples (i.e., data) will eventually illuminate the scheme most useful to policy-makers for designing evaluations indicating effective, or the more effective, programs. We comment that the more well-defined policy makers can be about the precise meaning of a successful program, the easier the resolution of epistemological issues. Accordingly, this note only responds to a few of Cooper’s points.
First, while we clearly advocate the use of RCTs in discovering effective energy efficiency programs, his characterization of our argument suggests that this is the only testing methodology that we advocate. In fact, we dedicated considerable discussion both to the limitations of RCTs and to consideration of other robust testing techniques that he does not discuss or acknowledge. A central point of our argument should not be missed: namely, robust testing should be used in evaluating the elements of energy efficiency programs to provide clear evidence of whether a program worked or not. When RCTs are or cannot be conducted, other types of evidence are welcomed!
Second, we agree that there are subtle but fundamental epistemological reasons that can discourage the use of RCTs in energy policy evaluations: principally, that parties working in government-sponsored programs are reluctant to implement policies (including research designs) that “control” decision-making by individuals in the market that are supposed to be outside the control of government. This important consideration was left out of our discussion, and his highlighting of this issue adds value to the discussion of when and how RCTs can be used. Recognizing a situation’s applicable “energy policy epistemology” does not necessarily preclude the use of RCTs—but is another “condition” that needs to be understood and “worked with” in designing and conducting the selected evaluation approach.
Third, his suggestion that large, longitudinal panel designs will somehow provide the kinds of insights that robust testing will produce needs significant development, as this may also be an unfruitful diversion. More development of the context and approach is required. Certain longitudinal designs may provide strong evidence of program impacts, such as the interrupted time series designs that we discuss, but these designs are challenging: they are difficult to construct and depend heavily on the artfulness and statistical sophistication of analysts and readers. Results from them will always be more or less controversial, although they would certainly be better than what we are currently seeing. And we agree with Cooper: this type of study should not preclude the continued use of RCTs and ethnographic and other qualitative approaches both within and without the policy and academy contexts.
Compliance with ethical standards
Conflict of interest
The authors declare that they have no conflict of interest.
- Cooper, A. (2018). Evaluating energy efficiency policy: understanding the ‘energy policy epistemology’ may explain the lack of demand for randomized controlled trials. Energy Efficiency, published online 26 January 2018.Google Scholar