Keywords

We started the discussion on evaluation by highlighting the overarching theme of sustainable development that comprises growth with inclusion, environmental stewardship, and good governance. These themes are present in the development plans and discussions of countries, and, in varying measure, they are essential ingredients of societal visions. Sustainable Development Goals capture this desired direction with targets for 17 attributes to be achieved by countries by year 2030.

Evaluation of Sustainability

This book makes the case for a stronger pursuit of the Sustainable Development Goals (SDGs) by according economic evaluations their rightful role in development work. The degree to which these goals are being met often falls short of expectations. However, there are significant welfare gains from policies that more effectively help achieve them. In each of the chapters dealing with impact evaluation (IE), cost-benefit analysis (CBA), and objectives-based evaluation (OBE), we have seen illustrations of how the value of interventions might be enhanced.

One way to put greater energy and drive into achieving progress toward the SDGs is to have better and timely assessments of sustainability—much as the experience with the previous Millennium Development Goals (MDGs) showed (IED 2013). But our ability to evaluate how well SDGs are being achieved is patchy. Ways and means for assessing progress need to be pursued and continuously improved, as examples in Chaps. 2, 3, and 4 illustrate. There is room for developing capacity for undertaking such evaluations and for adequately funding the efforts across countries.

A key factor in ensuring sustainable development would be the political support across countries and at various levels of governance. And one way to garner political support is to embed the results of evaluation much more frontally in the policy agenda of countries and global financial institutions. Timely release of the findings and their transparent application in decision-making help, again, as the experience with MDGs demonstrated.

Bringing evaluation to bear on the goals of sustainable development has been one overriding objective of this book. While Chaps. 2, 3, and 4 did not evaluate the achievement of the SDG targets per se, the different approaches to evaluation picked up the goals of sustainable development, albeit with gaps. As indicated in Chap. 1, economic growth forms an integral part of the evaluation as the very matrix of measuring value addition, benefits, and costs, or welfare gains is often the change in output or GDP. The challenge is how to put inclusion, sustainability, and governance under an evaluative lens, side-by-side with economic growth.

While this broadens the scope of assessments, focus and rigor should not be compromised. To be effective, it is essential for the work to be well-focused, well-defined, and rigorous. The broader focus should allow the evaluation to triage actions and options toward sustainable development.

The interplay of evaluation and economics helps in making the decision of the choice of topics and the scope of the work (see Van den Berg et al. 2018). Presenting stronger ties between economics and evaluation has been a second objective of the book. We have seen how assessments of sustainable development can be done carefully and credibly by applying tried-and-true tools of IE, CBA, and OBE. Such work can span from the micro and project-level assessments to the macro and aggregative assessments. But in either case, the application of economic analysis and evidence can bolster evaluations.

Quantifying costs and benefits of interventions to reflect distributional considerations can be aided by economic analysis of growth impacts on changes in income distribution (Dabla-Norris et al. 2015; Ostry et al. 2014). The assessment of global spillovers can be assisted by featuring health and climate change externalities (Sommer 2016; IMF 2015). Objectivity of information can be enhanced by the complementary data often culled directly from sources, for example, weather data connecting knowledge on weather patterns, high-risk areas, and people at risk (Emmanouil and Nikolaos 2015).

In this final chapter, we go further to see how the frameworks in Chaps. 2, 3, and 4 can be extended to get more mileage on sustainability. We set out three aspects that can help make economic evaluation of sustainable development stronger. First, there is much to be gained by looking for and into the important linkages—both direct and indirect—that contribute to outcomes. For example, indirect and non-income aspects (Dennig 2017) are important in considerations of inclusive growth. Second, sustainable development challenges have a local component and a global part (Everett et al. 2010). Often the local effects are mostly intended while the global carry an unintended component. Third, evaluators can innovate the data being used: with the availability of big data from the internet and social media, a huge window of opportunity has been thrown open (Faghmous and Kumar 2014).

Direct and Indirect Impacts

Broadening the field for evaluation helps to identify linkages that trigger important positive or negative impacts—across areas of concern or over time. Human well-being, which underlies inclusive growth, is understood to be multidimensional, including aspects of not only income but also education, health, and life satisfaction. These attributes, which are seldom incorporated in assessments of growth, might have significant impacts. The same can be said for environmental protection, where increasing growth is not enough to generate sustainable outcomes and where lack of environmental care itself can stunt growth. The 17 goals under the SDG framework help put emphasis on these non-monetary attributes of well-being.

Well-Being and Inclusion

One limitation often found in evaluation studies is their sole focus on impacts of interventions that are immediately observable. Usually, economic evaluation primarily concentrates on direct effects on income or expenditure. However, going from outputs to outcomes and impacts (as shown in Fig. 1.2) requires evaluation of sustainable development to look beyond immediately observable outcomes and to broaden its lens to focus on outcomes and impacts. While income and expenditure are useful measurements in that they are objective and clear, they do not fully capture the essence of sustainable development as it pertains to well-being and inclusion.

Human well-being in the context of sustainable development incorporates human capital, subjective well-being, and equal opportunity among other things (Thomas et al. 2000; Sachs 2012). The evaluation questions and the goals against which sustainable development is evaluated need to incorporate these as well.

As an example, an evaluation of an education project should not only be about increasing access but also about augmenting human capital. The goals should include immediate outputs such as the construction of more schools and also longer-term outcomes such as building knowledge and skills. A look at previously excluded groups, in particular girls, is most important. While the SDGs in themselves are sensitive to these differences, evaluations are yet to catch up. Most targets under the goal on quality education (SDG 4) have a gender parity component.

Accounting for interactions and spillovers of policies and projects with subjective well-being might make evaluations more meaningful. As an example, a study on unreliable urban water supply in the Kathmandu Valley in Nepal examines impacts of household coping costs (including those for collecting, pumping, purchasing, storing, and treating water) on well-being, which captures both evaluative (life satisfaction) and hedonic (feeling and emotions) reactions (Chindarkar et al. 2018). Findings reveal that coping cost is positively correlated with life satisfaction.

This seemingly counterintuitive finding is explained by households’ perception of coping costs as investments in household health and ability to be resilient. An insight for policy-makers from this evaluation is that under conditions of policy inaction, as has been the case with worsening urban water supply in Kathmandu, households need to spend time and money on coping with unreliable water supply to sustain their well-being and develop resilience. Thus, conclusions from evaluations that consider subjective well-being as an outcome could be different and insightful.

The scope for using well-being as an outcome is broad. Measures of subjective well-being can be used to assess impacts of environmental problems such as air pollution and climate change as well as incorporated in CBA (Chen et al. 2019; Li et al. 2014; Rehdanz and Maddison 2005). Studies have also examined the impact of rising inequality on subjective well-being (Graham and Felton 2006; Jiang et al. 2012).

Gaps in opportunities stem from differences in access to education, health, and other basic services. Yet when it comes to assessing the impact of growth and other policies on inequality, outcomes are often restricted to monetary measures such as mean log deviation, Gini coefficient, and Theil index. The limitation of these measures is that by focusing on income distributions they look only at the observed outcome of unequal opportunities and not at the unequal distribution of opportunities themselves that underlie individual advantage or disadvantage.

One proposition to evaluate equality of opportunity itself is to examine differences based on “circumstances,” such as place of birth, gender, and parental characteristics, over which individuals have no influence (Roemer 1993). In recent years attempts have been made to develop indices that capture lack of opportunities. Important among these is the human opportunity index (HOI) developed by De Barros, Ferreira, Vega, and Chanduvi (2009).

The HOI focuses only on dependent children and measures inequality of opportunity in terms of access to education, health, sanitation, and other basic services. The rationale for this focus is that access to these basic services is exogenous to children and therefore constitutes a circumstance for them. However, studies evaluating the inequality inherent in policies such as those relating to school construction or provision of development finance have overlooked HOI to see how policies affect shifts in distribution of opportunities.

Environmental Protection

With increasing pressure on the use of natural resources and runaway climate change triggered by the build-up of emissions in the air, environmental stewardship needs to take center stage in evaluations. Underlying much of environmental care is the understanding that natural capital, along with physical and human capital, is an integral part of the framework on how economic growth is generated (Thomas et al. 2000). There are evaluations of individual aspects of environmental impacts, but assessments of how the environment impinges on overall sustainability are lacking.

In a framework of sustainable development, economic growth is generated by investments in physical and financial capital, human and social capital, and environmental and natural capital. Economic policies by and large have favored investment in physical and financial capital through various forms of subsidies. Human and social capital have received increasing investments over the decades, but evaluations need to pay more attention to the degree of underinvestment seen when taking into account the positive externalities being generated.

Environmental and natural capital are not generally invested in, rather there is much degradation and unsustainable use. There is room to evaluate how this gap affects growth and sustainable development. If nature is included as a capital asset in production activities, there is likely to be a concern over growth patterns that conflict with the achievement of sustainable economic development. It would be useful to assess how the accumulation of physical and human capital may not have compensated for the degradation of natural capital.

The broader evaluative framework would allow evaluators to make direct connections and assess spillovers and indirect impacts among investments in different forms of capital. For instance, greater provision of environmental services can have the direct and tangible benefits such as lesser air and water pollution, which in turn can generate broader gains for worker productivity and livelihood (Zivin and Neidell 2012).

Broader Goals in Asia

As an application of a broader framework, we might consider the developments in Asia. Economic growth remains the biggest driver of development aspirations, but the vital linkages of other attributes to growth are emerging.

The need for evaluation to factor in social inclusion and the environment come through prominently in the case of Asia. Income inequality has worsened over the last decade in China, India, and other countries that, taken together, account for 80 percent of the region’s population. Developing Asia’s Gini coefficient went from 0.39 in the mid-1990s to 0.46 in the late 2000s. Furthermore, developing Asia is the world’s leading emitter of greenhouse gases, accounting for nearly 40 percent of global emissions, twice its share of global GDP. Air pollution is now at dangerously high levels in many Asian cities, notably New Delhi and Beijing, and environmental degradation is worsening across the region.

Incorporating and addressing gender inequality is a crucial dimension of inclusion. It is estimated that close to 100 million women are “missing” in Asia owing to gender-discriminatory practices (ADB 2012). Women in Asia are found to be worse off compared to men across various dimensions including health, access to education, asset ownership, and political inclusion (ADB 2012). Sensitizing evaluations to gender equality by explicitly incorporating gender-sensitive indicators would be a huge step forward. Gender-sensitive indicators such as maternal health, time use, and distributive impacts can be explicitly incorporated in IE, CBA, and OBE. The SDG framework on gender equality (SDG 5) and other goals where gender parity is considered can help shed light on gender-development issues.

Evaluation would want to take on board research results showing the deleterious effect of poor governance on growth (Kaufmann et al. 2010). Asia presents a mixed picture in global measures of good governance. For example, Southeast Asian countries in general fare poorly in their control of corruption in governance surveys, and this can affect growth drivers, including foreign investment and credit ratings. In East Asia, the gaps are wide for voice and accountability—an indicator which captures perceptions of the extent to which citizens can participate in policy-making processes and the accountability of governments. South Asia ranks low in political stability.

An example of how policy and strategy can guide evaluations toward achieving these broader goals is Asian Development Bank’s (ADB) new 2030 strategy. This strategy stresses sustainable development beyond economic growth in terms of greater inclusion, resilience, and well-being (ADB 2018). The approaches to bring about such progress are to be “integrated and multi-disciplinary” in order to address the complex problems of “inequality, climate change and urbanization which cut across several sectors.” Development financing under this strategy is explicitly aimed at achieving well-being, inclusion, and climate mitigation and adaptation, and incorporates these as evaluation goals. The challenge is how to make these directions operational.

Local and Global Public Goods

Development priorities and challenges are increasingly taking on global dimensions. Local issues, like deforestation or slash-and-burn practices in one country, can affect neighboring countries (Thomas 2018). A case in point is Indonesia, where each year slash-and-burn agriculture causes massive emissions that hurt the health of populations not only within Indonesia but also in neighboring Malaysia, Singapore, and beyond. In this case as well as the case of massive air pollution and smog in Asia’s megacities, the local effects spillover to regional and even worldwide scales, aggravating global warming. Another example is the global financial crisis, which originated in a few centers in the developed world, but its social effects in terms of increased inequality and poverty rippled across the world.

Global efforts are called for as scientists make clear vast biodiversity losses and rapid climate change across the globe. The world has lost 60 percent of the animal life on the planet since 1970 (WWF 2018), and global warming is estimated to reach a critical level by as early as 2030 (IPCC 2018). Evaluations must move from a growth-only focus and pay considerably more attention to these urgent issues.

Governance too has global dimensions. Studies show that more open trade and globalization have brought net gains to countries in many instances (e.g., IMF Staff 2001; Dabla-Norris and Duval 2016). But there are also losers, and at times their interests, true or perceived, can dominate. The world has witnessed the United States government reneging on global agreements on emissions and international trade. This highlights the role of opposing interests and the fact that even where the aggregate gains are positive, the interests of particular groups that might lose become decisive inputs into policy.

Special efforts are needed to assess and share the findings about the gains for common goods from collective actions, especially where global public goods (GPGs) are involved. Important themes for future evaluations are the effectiveness of global funding mechanisms such as climate change funds, multilateral agreements such as regional economic partnerships and global climate agreements, and bilateral agreements such as transboundary water conventions.

Evaluating issues and policies pertaining to GPGs is complex, which probably explains why the evaluation techniques discussed in previous chapters—IE, CBA, and OBE—do not systematically incorporate GPGs. Complexities also pertain to funding evaluations of GPG interventions and the institutional setup required to conduct these evaluations.

Kanbur (2017) argues that by its very nature, the benefits of addressing transboundary issues are also transboundary. Since benefits accrue beyond individual countries, incentives, such as grants, are needed to motivate countries to collaborate on GPGs. By extension, financing evaluations of GPG interventions would also require setting up grants and collective deliberation on performance indicators. While each country should have its own platform for implementing actions addressing GPG issues, evaluation institutions and mechanisms are required at the global level. A way forward is to build in independent evaluations into the global mechanisms to assess transboundary benefits.

Attention would need to be given to spillover effects when evaluating GPG interventions. We have discussed how spillovers can be incorporated into IE and CBA in Chaps. 2 and 3. The same ideas can be extended to transboundary spillovers. To design an experimental or quasi-experimental IE of a GPG intervention, evaluators would first need to have good knowledge (based on theory or prior evidence) of why and how spillover effects occur.

The treatment and control groups in IE can then be identified in the relevant socioeconomic unit (group of regions or group of countries) within which the spillovers occur, and treatment effects can be adjusted to avoid biased estimates (Angelucci and Di Maro 2015). Similarly, in CBA the relevant socioeconomic unit of analysis will need to be identified and marginal social cost be adjusted based on whether the transboundary spillover is positive or negative, and consequently net social benefit would be altered.

A further complexity in evaluating GPG interventions is reliable data. Little is known about the spending by countries on GPGs. Some attempts are being made to estimate these outlays such as those by Birdsall and Diofasi (2015). However, this is just a start and better reporting practices and, more fundamentally, an agreement on what should count as spending on GPGs is needed. For instance, spending on HIV/AIDS prevention in Africa by the United States can be thought of primarily as financing for treating and preventing the disease within-country boundaries. However, given the large out-migration from Africa, HIV/AIDS prevention also has GPG characteristics.

A related challenge is that of suitable methodological tools. While entirely new tools are probably not required, what is required is pliability of the tools reviewed in this book in evaluating GPG interventions. For instance, an IE of development finance on global HIV/AIDS treatment and prevention should estimate and disaggregate the average treatment effect by within- and between-country treatment effects. More work is needed on refining the econometric tools for evaluating transboundary effects. A CBA of HIV/AIDS financing should account for the fact that net social benefits are not restricted to affected countries but also have implications for countries where people from affected countries migrate to. The same central criteria for OBE can be used but with specific attention given to transboundary effectiveness, efficiency, development impact, and sustainability.

Van den Berg (2011) cautions that evaluation of GPGs can show a “micro-macro paradox.” This term refers to a situation where local (or within-country) interventions might be successful, yet when assessed at the global level, the interventions do not translate into desirable outcomes. For example, individual countries might achieve emissions reductions through carbon taxes. However, at the global level there might be no observed change or even an increase in emissions if industrialized countries shift pollution-generating activities to less developed countries.

Similarly, an evaluation of development finance might find that it does achieve SDGs in individual countries; however, the global impact of development finance might be limited. In this case, the micro-macro paradox can partly be explained as a consequence of insufficient public funding available to meet global public costs such as for climate-induced disasters or forced migration. These paradoxes offer lessons on interventions that have different local, regional, or global effects.

Small and Big Data

Sound evaluations are invariably predicated on sound data. For the most part, these data have come from household, national, and international survey and estimations, made available to researchers in published and unpublished forms. Gaps are serious, particularly on many aspects of sustainable development. Greater attention to evaluation of sustainable development should motivate more investments in generating and sharing the underlying data.

The explosion of digital technology and the expanding amount of data now hold promise in enabling their use in research and evaluation. The application of big data is quickly expanding in business, government, and civil society. For example, various agencies of the United States’ government at the central and state levels are mining and analyzing data to mitigate fraud, enforce law, and monitor usage of resources. An example is the Department of Health and Human Services, which implemented a fraud prevention system identifying millions of dollars in improper payments to health-care providers (see, e.g., US GAO 2017).

Other examples include Australia, the Nordic countries, and the United Kingdom, where governments track citizens through the course of their lives. The data they collect contain information on birth outcomes, education outcomes, and health outcomes, which are then linked with socioeconomic information, creating a rich database that is ideal for policy evaluation. Large-scale administrative data are also being sourced from utility bills, public-transport smart cards, banking and credit card transactions, satellite images, and so on.

Application of big data to evaluations is mostly confined to identifying correlations and predicting trends (UN Global Pulse 2012). While this is quite different from counterfactual IE, correlations and trends generated from large volumes of data can still be useful as they may closely represent the population. Correlations can be used to identify systematic patterns and repeated behaviors, consequently unveiling stylized facts about inclusive growth, sustainability, and governance.

For instance, predictive analysis can help identify students at risk of dropping out. Monitoring student retention rates will make way for enhancing student academic performance and therefore satisfaction among students, teachers, and school administration. Data gathered on individual students’ learning styles can also assist teachers, who can adjust their teaching styles according to students’ needs.

The World Health Organization declared the Zika virus a global health emergency in 2016 and forecasted the spread of the virus. While there were no reliable tests and vaccines for the virus at the outset, utilizing data-driven infrastructure helped to identify trends and analyze clinical-test results, shortening the search for a cure. Health systems are using big-data technologies like Apache Hadoop to take real-time streams of data from monitors, machines, and wearables and combining it with electronic health records (Juric et al. 2017). Big-data technologies make it possible to apply intelligence to multiple electronic data feeds of clinical tests as they stream in.

As mayors struggle to make cities financially viable and sustainable, big data can be used to create “smart” cities. Every city has its own intricacies, and therefore no master design exists, but every smart city presents an opportunity for big data to govern public policies. For example, the city of Boston uses the crowdsourcing app Street Bump to collect data from citizens’ smartphones to allocate maintenance and repair crews, resulting in vast savings (Zie 2015). In San Francisco, smart meters provide digital reads of water flow to track citizens’ water usage, also producing sizable savings.

The use of big data is proving to be a valuable tool in disaster management. Advances in ground-based networks of radars as well as in satellite data are key to nearly continuous observation of global weather. Japan’s Meteorological Agency recently updated its Evaluation Alert System with much more detailed data to support evacuations, mapping the intensity of weather-related hazards and people with special needs. In Turkey, a new National Emergency Management Information System and an Uninterrupted and Secure Communication System Project link authorities during emergencies. Australia’s Emergency Alert enables territories to issue warnings through landline and mobile telephones linked to high-risk properties, working across telecommunication carrier networks.

Technologies that link sensor networks, large-scale data analysis, and communications systems can provide decision-makers with timely information to guide responses. Siemens implemented a levee monitoring system in the Netherlands using sensors to monitor water pressure, temperature, and shifting weather patterns to identify areas that are at risk of being breached and trigger alarms (Guardian 2013). IBM provided a digital command center that integrates real-time information on storm conditions, emergency-response assets, and areas at risk (Guardian 2013).

Pertaining to governance, law enforcement is another area benefiting from big data. The implementation of predictive policing is relatively new, and it is currently being tested and deployed in several cities across the United States. The method uses data from type, place, and time of previously committed crimes in order to assign probabilities of future crime incidents. In some places, there is evidence of a decline in crime as a result (Mohler et al. 2015).

Developing countries are benefiting from such real-time evaluations to track their progress toward achieving the SDGs. Case in point, a laboratory in Rwanda uses electronic sensors to assess the use of water filters and cook stoves. The UN Global Pulse runs a number of projects that use data from social media to monitor social and environmental issues. One project analyzes conversations on social media to understand public perceptions on sanitation, providing a baseline for change in public discourse on sanitation.

Similarly, other Global Pulse projects use Twitter to measure global engagement on climate change. Food security issues are also being assessed, as in Indonesia, where the correlation between actual food price fluctuations and perceptions about food inflation on Twitter were tracked (UN Global Pulse 2011). Comparison of the trends of actual food price fluctuations and tweets on food inflation shows that public perception about food inflation on social media closely tracked actual prices.

Evaluators have attempted to use big data for causal analysis by applying experimental and quasi-experimental tools to a large pool of observations. Ibarra, McKenzie, and Ortega (2017) use high-frequency financial data on over one hundred thousand credit card clients in Mexico to evaluate the impact of financial education on credit card usage and bill payment behavior. They find that while financial education increases the probability of paying bills on time and paying more than the minimum payment due, it does not reduce spending.

Big data can complement traditional IE, survey data, and official statistics by adding up-to-date information to provide a fuller picture of evaluations. However, there are several things to bear in mind when using big data for evaluation. While these data open up avenues for innovative evaluations, evaluators must exercise caution, particularly when it comes to privacy and personal data protection. When accessing and using these data, evaluators must be aware of country laws pertaining to data protection and undergo the required review process to get approval for conducting their IEs. Also, big data might contain inherent selection bias in countries where internet and smartphone penetration is low. In these cases, big data will only reflect the behaviors and opinions of those with access to technology. And finally, big data cannot fully replace, but only complement, rigorous evaluations.

Conclusion

In Chaps. 2, 3, and 4, we have seen applications of IE, CBA, and OBE in assessing performance and providing lessons for improvements. As countries, in varying degrees, now embrace the SDGs, it is crucial for evaluations to keep its eye on monitoring and tracking sustainable development. Aiming for sustainable development also helps to achieve a greater integration of work across sectoral and thematic boundaries, such as infrastructure and the environment or education and labor markets.

Our examples also suggest that there are gains in taking advantage of the interplay between evaluation and economics. For example, evaluations of economic growth and income distribution are much more impactful when they bring together findings from an economic theory of change and evaluation. The quality of the data, the rigor of analysis, and the timeliness of the findings all decide how useful the work is and how influential it is in shaping decisions and policy.

There are fruitful avenues for evaluation to capture social inclusion, environmental care, and good governance, in addition to economic growth. Incorporating regional and global effects beyond the local level is becoming increasingly essential. These effects are immensely important, for example, in income inequality and climate change. Innovative data may lend themselves to addressing these broader questions and help deliver better results.

Employing a broader development lens in individual evaluations has been a challenge. Broadening the agenda, even when it makes eminent sense, introduces complexities and difficulties, not least of them being the limits placed by the availability of methods and data. It is important that in broadening the scope of work, one does not lose sight of the priorities in terms of the outcomes and of the needed selectivity in terms of the most important linkages that matter.

In the end, the quality of the evaluation work determines the usefulness of the findings for policy-making. Broadening the evaluative lens strengthens the relevance of findings and improves the chances of capturing crucial indirect and unintended effects of interventions. At the same time, broadening of the field needs to ensure rigor, comparability, and some degree of replicability of the findings. In this respect, aligning evaluations with a commonly agreed set of goals and aspirations such as the SDGs, tracking progress, and drawing on lessons of experience will help.