Abstract
Policies to combat climate change should be supported by evidence regarding their effectiveness. But what kind of evidence is that? And what tools should one use to gather such evidence? Many argue that randomized controlled trials (RCTs) are the gold standard when it comes to evaluating the effects of policies. As a result, there has been a push for climate change policies to be evaluated using RCTs. We argue that this push is misguided. After explaining why RCTs are thought to be the gold standard, we use examples of mitigation and adaptation policies to show that RCTs provide, at best, one piece of the evidential puzzle one needs to assemble for well-supported decisions regarding climate change policies.
This is a preview of subscription content, log in via an institution to check access.
Notes
- 1.
We use the expressions ‘anthropogenic global warming’ and ‘climate change’ interchangeably in this paper.
- 2.
Global warming is expected to have limited positive effects, in the short run and in some regions, for instance in the domain of timber productivity (IPCC 2007b, 289). It is also the task of policy makers to design policies for taking advantages of these positive effects.
- 3.
This distinction is reflected in the Fourth IPCC Assessment Report . This report treats of mitigation and adaptation in two distinct parts, though it contains a chapter on the relations between them (IPCC 2007b, chapter 18).
- 4.
They also want policies that have large benefit/cost ratios. We leave aside issues related to cost-benefit analysis itself in what follows, and focus on the preliminary step to any such analysis: the evaluation of the likelihood that a policy will yield the intended benefit.
- 5.
See http://www.thegef.org/gef/eo_office. Other funding agencies such the World Bank (http://ieg.worldbankgroup.org/), the International Monetary Fund (http://www.ieo-imf.org), or the US Food and Drug Administration (http://www.fao.org/evaluation/) also have their own evaluation offices. There are also organizations, such as the International Initiative for Impact Evaluation (3ie, http://www.3ieimpact.org/), whose sole role is to fund and carry out IEs. The multiplication of evaluation offices results in the multiplication of guidelines and methodologies for conducting IEs.
- 6.
It is widely assumed, and not just by the World Bank, that answering a causal question about the effect of a policy just is to answer some counterfactual question about what would have happened in the absence of the policy. Thus Duflo and Kremer, both members of the influential Jameel Poverty Action Lab at MIT, claim that “Any impact evaluation attempts to answer an essentially counterfactual question: how would individuals who participated in the program have fared in the absence of the program?” (Duflo and Kremer 2005, 3). And Prowse and Snilstveit, in a review of IEs of climate policies, claim that “IE is structured to answer the [counterfactual] question: how would participants’ welfare have altered if the intervention had not taken place?” (Prowse and Snilstveit 2010, 233).
- 7.
Who are sometimes called ‘randomistas’ as in, e.g., Ravallion et al. (2009).
- 8.
See, e.g., Rubin (2008).
- 9.
The terminology comes from clinical trials.
- 10.
It also enables one to answer the question ‘What would be the mean value of E for individuals (in the study population) not exposed to C were C present, all else being equal?’ by citing the mean value taken by E for individuals actually exposed to C. Note that we are here talking about mean values of E over the treatment and control groups respectively and over an extended run of repeated randomizations on the study population. RCTs enable one to estimate the mean causal effect of C on E in a given population, not the individual causal effect of C on E for any specific individual in this population.
- 11.
‘Ideal’ RCTs (ones for which balance of other causes is actually achieved) are, in the words of Cartwright Hardie (2012, §I.B.5.3), ‘self-validating’, i.e., their very design guarantees the satisfaction of the assumptions that must be satisfied in order for the causal conclusions they yield to be true.
- 12.
- 13.
We treat ‘mean’, ‘expectation’, and ‘expected value’ as synonyms here.
- 14.
The probabilistic independence of X i from b i guarantees that the size of the effect of C on E for i is causally unrelated to whether i is assigned to the treatment or the control group. And the probabilistic independence of X i from W i guarantees that whether i is assigned to the treatment or control group is causally unrelated to the causes of E that do not appear in (CP).
- 15.
For the full proof see e.g., Holland and Rubin (1987, 209–210). Essentially the same results as these hold for more complicated functional forms for (CP); we choose the linear form for ease of illustration.
- 16.
Though this does not mean that J-PAL members only work on RCTs, it does mean that all the IEs sponsored and conducted by J-PAL take the form of RCTs.
- 17.
There is a lot to be said about the standard view and why the labels ‘internal validity’ and ‘external validity’ are both vague and misleading. Given limitations of space, however, these issues cannot be discussed here. For more, see Cartwright and Hardie (2012, §I.B.6.3).
- 18.
The hedge ‘in principle’ is important. Poorly executed RCTs will not produce unbiased estimates of treatments effects.
- 19.
See Cartwright and Hardie (2012, op. cit.) for a concrete example of an appeal to similarity. See also http://blogs.worldbank.org/impactevaluations/impactevaluations/why-similarity-wrong-concept-external-validity
- 20.
All the conclusions we draw below apply mutatis mutandis when the relevant causal principles take more complex forms than that of (CP) (e.g., non-linear forms).
- 21.
You may be used to thinking of b i as the size of the effect of X i on Y i. Indeed, this is the way we described it above when introducing (CP). But because, as we explain below, causes are INUS conditions, the two descriptions are equivalent: The effect of C on E just is what happens to E when C is present along with all of its required support factors.
- 22.
Each term in an equation like (CP) represents a contribution to the effect. Mackie’s original theory does not mention ‘contributions’ because he only consider binary ‘yes-no’ variables. Our presentation is more general in that it encompasses both cases in which the cause and effect variables are binary, and more common cases in which they are not.
- 23.
As the ‘short circuit’ example makes evident, the distinction between policies and support factors is a pragmatic one. Both a policy and its support factors are causes, and so both are INUS conditions . Some factor is usually singled out as the policy because it is practical, ethically acceptable, or cost-efficient to manipulate it. Note also that we claim that all causes are INUS conditions , but not that all INUS conditions are causes.
- 24.
If this estimate is equal to 0, or very close to 0, then you cannot directly draw any conclusion about the causal role played by C in the study population because you do not know whether C is ineffective or, alternatively, its positive and its negative effects balance out. We leave this case aside here.
- 25.
See Heckman (1991) for a further critique of the limitations of RCTs when it comes to estimating parameters that are of interest for policy making.
- 26.
Apart from giving you a trustworthy estimate of the value of Exp[b i].
- 27.
Banerjee and Duflo, for instance, make the following claim: “A single experiment does not provide a final answer on whether a program would universally ‘work’. But we can conduct a series of experiments, differing in […] the kind of location in which they are conducted…” (Banerjee and Duflo 2012, 14). They add that “This allows us to […] verify the robustness of our conclusions (Does what works in Kenya also work in Madagascar?)…” (ibid).
- 28.
You may think this is an uncharitable reconstruction of the argument advanced by advocates of RCTs. But the claims they sometimes make, e.g., Banerjee and Duflo’s claim, quoted in note 27, regarding the need for several RCTs in order to establish that a policy works “universally”, seem to invite reconstructions that are far less charitable. One could thus see advocates of RCTs as advancing an argument of the form ‘If RCTs produce conclusive results in A, B, and C, then the policy works “universally”, and it will therefore work in D’. This construal seems less charitable in that it attributes to advocate of RCTs a claim (the conditional in the previous sentence) that’s highly likely to be false.
- 29.
In the case of mitigation-relevant PES program, the buyer of the ES often is an intergovernmental agency, e.g., the GEF, acting as a third party on behalf of users of the ES. When the GEF is the buyer of the ES, the users it represents are the citizens of states that are members of the UN.
- 30.
Of course, many PES programs that target biodiversity also results in the protection of carbon stocks and, conversely, many PES programs that target climate change mitigation also result in the conservation of biodiversity.
- 31.
- 32.
- 33.
The UN, for instance, is developing a program called ‘REDD+’ that relies on PES-type programs in order to reduce deforestation. Note that ‘REDD’ is an acronym for ‘Reduction of (carbon) Emissions from Deforestation and forest Degradation’.
- 34.
In the Oportunidades (originally PROGRESA) program, parents receive conditional payments for activities that improve human capital, e.g., enrolling their children to school. The idea is to reduce poverty both in the short term, via the cash payments, and in the long run, by improving human capital. The payments in this program, as well as in PES programs, are conditional in that they are made only if the service (e.g. an ES) is actually provided: They are not one-time payments that are made upfront.
- 35.
The project is supposed to last for four years, from April 2010 through April 2014.
- 36.
And it won’t tell you whether the same causal principle is at work in those parts of the study populations composed of landowners from the Hoima district and those parts composed of landowners the Kibaale districts.
- 37.
- 38.
And if the assumption that these factors are always required is dropped, then you also need evidence that these factors are indeed support factors needed for the PES program to produce the intended contribution to the effect in the location you are targeting.
- 39.
See http://www.vulnerabilityindex.net/ for the EVI and http://webra.cas.sc.edu/hvri/ for the US county-level SoVI. Note two difficulties with using these indices to evaluate the effects of adaptation policies. First, they are measures of vulnerability to environmental hazards in general, whether or not they are due to climate change. Second, there is no wide consensus as to how to measure overall vulnerability (at various geographical scales), and neither is there a consensus regarding how to measure an important component of vulnerability, namely adaptive capacity.
- 40.
- 41.
- 42.
RCTs conducted about weather insurance usually attempt to estimate the effects of such insurance on investment decisions (see e.g., Giné and Yang 2009) or to understand the causes of weather insurance take-up (see e.g., Cole et al. 2013). See de Nicola (2015) for a non-randomized evaluation of the effects of rainfall index insurance on the welfare of farmers and so on their adaptive capacity.
- 43.
See www.mwe.go.ug
References
Alix-Garcia, Jennifer, Alain De Janvry, Elisabeth Sadoulet, and Juan Manuel. 2009. Lessons Learned from Mexico’s Payment for Environmental Services Program. In Payment for Environmental Services in Agricultural Landscapes, 163–188. Rome: Springer.
Balshem, Howard, Mark Helfand, Holger J. Schünemann, Andrew D. Oxman, Regina Kunz, Jan Brozek, Gunn E. Vist, et al. 2011. GRADE Guidelines: 3. Rating the Quality of Evidence. Journal of Clinical Epidemiology 64 (4): 401–406. https://doi.org/10.1016/j.jclinepi.2010.07.015.
Banerjee, Abhijit, and Esther Duflo. 2012. Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty. New York: PublicAffairs.
Cartwright, Nancy. 2010. What Are Randomised Controlled Trials Good For? Philosophical Studies 147 (1): 59–70.
———. Forthcoming. Will Your Policy Work? Experiments vs. Models. In The Experimental Side of Modeling, ed. I.F. Peschard and B.C. van Frassen.
Cartwright, Nancy, and Jeremy Hardie. 2012. Evidence-Based Policy: A Practical Guide to Doing It Better. Oxford: Oxford University Press.
Coase, Ronald Harry. 1960. The Problem of Social Cost. The Journal of Law and Economics 3: 1–44.
Cole, Shawn, Xavier Giné, Jeremy Tobacman, Petia Topalova, Robert Townsend, and James Vickery. 2013. Barriers to Household Risk Management: Evidence from India. American Economic Journal: Applied Economics 5 (1): 104–135.
de Nicola, Francesca. 2015. The Impact of Weather Insurance on Consumption, Investment, and Welfare. Quantitative Economics 6 (3): 637–661.
der Werf, Van, R. Guido, Douglas C. Morton, Ruth S. DeFries, Jos G.J. Olivier, Prasad S. Kasibhatla, Robert B. Jackson, G. James Collatz, and James T. Randerson. 2009. CO2 Emissions from Forest Loss. Nature Geoscience 2: 737–738.
Duflo, Esther, and Michael Kremer. 2005. Use of Randomization in the Evaluation of Development Effectiveness. Evaluating Development Effectiveness 7: 205–231.
Engel, Stefanie, Stefano Pagiola, and Sven Wunder. 2008. Designing Payments for Environmental Services in Theory and Practice: An Overview of the Issues. Ecological Economics 65 (4): 663–674.
GEF. 2007. Adaptation to Climate Change through Effective Water Governance in Ecuador. Project Executive Summary.
———. 2010. Developing an Experimental Methodology for Testing the Effectiveness of Payments for Ecosystem Services to Enhance Conservation in Productive Landscapes in Uganda (Request for CEO Endorsement/Approval). Washington, DC: Global Environment Facility.
———. 2016. Developing an Experimental Methodology for Testing the Effectiveness of Payments for Ecosystem Services to Enhance Conservation in Productive Landscapes in Uganda. Washington, DC: Global Environmental Facility, June 4. https://www.thegef.org/project/developing-experimental-methodology-testing-effectiveness-payments-ecosystem-services
Giné, Xavier, and Dean Yang. 2009. Insurance, Credit, and Technology Adoption: Field Experimental Evidence from Malawi. Journal of Development Economics 89 (1): 1–11.
Heckman, James J. 1991. Randomization and Social Policy Evaluation. National Bureau of Economic Research Cambridge, MA. http://www.nber.org/papers/t0107
Holland, Paul W. 1986. Statistics and Causal Inference. Journal of the American Statistical Association 81 (396): 945–960.
Holland, Paul W., and Donald B. Rubin. 1987. Causal Inference in Retrospective Studies. ETS Research Report Series 1987 (1): 203–231. https://doi.org/10.1002/j.2330-8516.1987.tb00211.x.
IPCC. 2007a. Climate Change 2007: The Physical Science Basis. New York: Intergovernmental Panel on Climate Change.
———. 2007b. Impacts, Adaptation and Vulnerability. New York: Intergovernmental Panel on Climate Change.
———. 2012. Managing the Risks of Extreme Events and Disasters to Advance Climate Change Adaptation: Special Report of the Intergovernmental Panel on Climate Change. New York: Cambridge University Press.
Jayachandran, S. 2013a. Evaluating a Payments for Ecosystem Services Program in Uganda, April 22. http://www.climate-eval.org/?q=print/2235
Jayachandran, Seema. 2013b. Liquidity Constraints and Deforestation: The Limitations of Payments for Ecosystem Services. The American Economic Review 103 (3): 309–313.
Jones, Harry. 2009. The ‘Gold Standard’ is Not a Silver Bullet for Evaluation. Overseas Development Institute London. http://www.alnap.org/pool/files/3695.pdf
Mackie, J. 1965. Causes and Conditions. American Philosophical Quarterly 2: 245–264.
MEA. 2005. Ecosystems and Human Well-Being: Current State and Trends. Washington, DC: Millennium Ecosystem Assessment.
Muradian, R., E. Corbera, U. Pascual, N. Kosoy, and P.H. May. 2010. Reconciling Theory and Practice: An Alternative Conceptual Framework for Understanding Payments for Environmental Services. Ecological Economics 69 (6): 1202–1208.
Parker, Susan W., and Graciela M. Teruel. 2005. Randomization and Social Program Evaluation: The Case of Progress. The Annals of the American Academy of Political and Social Science 599 (1): 199–219. https://doi.org/10.1177/0002716205274515.
Pattanayak, Subhrendu K., Sven Wunder, and Paul J. Ferraro. 2010. Show Me the Money: Do Payments Supply Environmental Services in Developing Countries? Review of Environmental Economics and Policy 4 (2): 254–274. https://doi.org/10.1093/reep/req006.
Pirard, Romain, Raphaël Billé, and Thomas Sembrés. 2010. Questioning the Theory of Payments for Ecosystem Services (PES) in Light of Emerging Experience and Plausible Developments. Institut Pour Le Développement Durable et Les Relations Internationales (IDDRI), Analyses, 4, no. 2010/06/10, 5–22.
Prowse, Martin, and Birte Snilstveit. 2010. Impact Evaluation and Interventions to Address Climate Change: A Scoping Study. Journal of Development Effectiveness 2 (2): 228–262.
Ravallion, Martin, et al. 2009. Should the Randomistas Rule? Economists’ Voice 6 (2): 1–5.
Rubin, Donald B. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology 66 (5): 688–701.
———. 2008. Comment: The Design and Analysis of Gold Standard Randomized Experiments. Journal of the American Statistical Association 103: 1350–1353.
STAP. 2010. Payments for Environmental Services and the Global Environment Facility: A STAP Advisory Document. Washington, DC: Scientific and Technical Advisory Panel. https://www.thegef.org/sites/default/files/publications/STAP_PES_2010_1.pdf.
STAP. n.d. Payments for Environmental Services and the Global Environment Facility: A STAP Advisory Document. Washington, DC: Scientific and Technical Advisory Panel.
Acknowledgements
Both authors would like to thank the Templeton Foundation’s project ‘God’s Order, Man’s Order and the Order of Nature’, the UCSD Faculty Senate, and the AHRC project ‘Choices of evidence: tacit philosophical assumptions in debates on evidence-based practice in children’s welfare services’ for support for the research and writing of this chapter. Nancy Cartwright would in addition like to thank the Grantham Research Institute on Climate Change and the Environment at LSE.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 The Author(s)
About this chapter
Cite this chapter
Marcellesi, A., Cartwright, N. (2018). Modeling Mitigation and Adaptation Policies to Predict Their Effectiveness: The Limits of Randomized Controlled Trials. In: A. Lloyd, E., Winsberg, E. (eds) Climate Modelling. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-65058-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-319-65058-6_15
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-319-65057-9
Online ISBN: 978-3-319-65058-6
eBook Packages: Earth and Environmental ScienceEarth and Environmental Science (R0)