Evaluation of Marine Spatial Planning: Valuing the Process, Knowing the Impacts

Abstract

A need for evaluating maritime spatial planning (MSP) has become more important as countries are preparing and implementing their MSP. Evaluations are opportunities to foster learning and to accumulate knowledge in order to improve the processes and effectiveness of MSP. Empirical evidence has shown that MSP processes are run in different ways and for different purposes. To address this diversity, this chapter presents a methodology for designing a flexible and context-specific evaluation of MSP. This chapter introduces a theory-based evaluation approach that aims to explain how and why the evaluated planning process or plan works or fails to work. It scrutinises how MSP produces intended and unintended effects, for whom and in what contexts and what mechanisms are triggered by the MSP.

1 Introduction

Marine/Maritime spatial planning (MSP) is an increasingly common approach to manage the use and protection of the resources, the ecosystems and the space of seas (Douvere 2008; Jay et al. 2013). An often-cited definition states that ‘Marine [or maritime] Spatial Planning is a public process of analysing and allocating the spatial and temporal distribution of human activities in marine areas to achieve ecological, economic, and social objectives that usually have been specified through a political process’ (UNESCO-IOC 2010). A recent European Union directive (EC 2014) states that MSP is ‘to promote the sustainable growth of maritime economies, the sustainable development of marine areas and the sustainable use of marine resources’ (Directive 2014/89/EU).

The earlier mentioned objectives for MSP are very ambitious. If achieved, the coastal states will gain thriving maritime economies, fewer conflicts at sea and improved environmental status of the marine ecosystems. But knowing whether the plan will help society to achieve all or any of the objectives requires specific attention. This is the evaluative question that is the focus of this chapter.

This chapter applies commonly used approaches in the evaluation of policies and spatial planning on land to the MSP. The emphasis of the chapter is on trying to know the effectiveness of MSP, which brings with it critical perspectives on MSP’s ability to deliver its objectives. The effectiveness of MSP is largely dependent also on the processes of preparing and implementing MSP. Therefore, characteristics of the process need to be addressed as well.

A distinction from strategic environmental assessment (SEA) helps to explain the consequences of focusing on evaluating effectiveness. In an SEA, which assesses likely environmental impacts of alternative versions of a plan, one should assume that the described alternative is realised. Then likely environmental impacts of each alternative are assessed and mitigation measures are suggested. In an evaluation of effectiveness that focuses on the chosen or proposed plan, one measures or assesses the extent that set targets will be met, any unintended impacts generated and possible obstacles to realising the plan and achieving the desired impacts. Whereas SEA aims to help design a plan that has the least negative environmental impacts, an evaluation of effectiveness aims at improving the implementation of the plan or suggesting improvements for the next versions of the plan. The following figure illustrates the difference (Fig. 18.1).

Fig. 18.1

Key questions in an evaluation of effectiveness and SEA. Linkages to indicators

This chapter is based on the work conducted in the Baltic SCOPE project (www.balticscope.eu) in 2015–2017. Within the Baltic SCOPE project the author developed an evaluation and monitoring framework for cross-border collaboration in MSP (Varjopuro 2017). The Baltic SCOPE project brought together MSP authorities from six Baltic Sea Region countries to enhance cross-border integration and coordination of MSP activities in the Baltic Sea. The countries were Sweden, Denmark, Germany, Poland, Latvia and Estonia. Other project partners were the intergovernmental organisations Helsinki Commission (HELCOM) and Vision and Strategies Around the Baltic Sea (VASAB) and the two research organisations NordRegio and the Finnish Environment Institute. For the purpose of preparing the evaluation and monitoring framework, the author and his colleagues followed the Baltic SCOPE process and conducted individual and group interviews to identify factors that influence the success of transboundary collaboration. The collected material also shed light on national MSP processes and practices. One admittedly obvious observation was that MSP is practised in very different ways and with very different objectives. A key conclusion for the prepared evaluation and monitoring framework was that presenting a standard evaluation protocol would not be useful. Instead, it has to be flexible and adaptable for different contexts and cases. The work on developing monitoring and evaluation approaches is now continuing in the Pan Baltic Scope project (http://www.panbalticscope.eu/). In Pan Baltic Scope the focus is on evaluation of national MSP. This chapter describes the methodological findings gained in these two projects.

The chapter starts by presenting approaches and concepts of evaluation of policies and spatial plans. This is followed by a presentation of the progress gained in evaluation of MSP. Section 5 introduces the theory-based evaluation approach for evaluating MSP, while the final Sect. 6 discusses practical considerations of organising evaluations of MSP.

2 Purposes of Evaluation

Evaluation assesses the merit and value of public policies or, as in this case, spatial plans. Evaluation often asks if the set targets are met, but evaluations can also address the processes of policy or plan formation as well as the processes of their implementation (Vedung 1997, 2006). Evert Vedung (2010, 263) justifies the usefulness of evaluating public policies:

If you carefully examine and assess the results of what you have done and the paths toward them, you will be better able to orient forward. Good intentions, increased funding and exciting visions are not enough; it is real results that count. The public sector must deliver. It must produce value for money.

The main objective of evaluation is thus to improve policies and plans and the processes of producing them. Evaluation should not be seen simply as a judgement of whether or not public authorities have been successful in designing or implementing the policies and plans. Furthermore, it is not enough to determine whether the objectives were or will be met or how satisfied different stakeholder groups are with the process or its outcomes. This is because to improve the policies and plan, it is important to understand why certain elements of the policy or plan work or do not work. In relation to stakeholders, it would also be important to elucidate which of the affected parties have benefitted the most and the least (EC 2013a).

Evaluations that shed light on outcomes as well as on processes of making and implementing policies and plans increase our understanding of various aspects of policies and plans. Such broad evaluations ‘provide opportunities to learn about the questions to ask, the goals to set and how to frame the issues as well as the instrumental learning about how to design or implement the policy’ (Mickwitz 2006, 18).

A primary purpose of the evaluation process should be to foster learning; but who then learns from the evaluations? The public authorities that commission the evaluation are obviously the ones that learn. This applies to individuals working in such organisations, but learning by an organisation should also be fostered. Evaluation can support both single-loop and double-loop learning (Mickwitz 2006), which are essential for adaptive management cycles (Armitage et al. 2008; Cundill et al. 2012; Pahl-Wostl et al. 2007). Through single-loop learning, one learns how to improve the effectiveness of spatial planning solutions, while double-loop learning helps one to develop the planning system as a whole.

The commissioning public body should not be considered the only one to learn. As pointed out in the quotation from Vedung (2010), public processes should deliver benefits to society at large. It is nowadays the norm that policies and plans are prepared in participatory processes, which considerably enlarge the group of those who could learn from the results of evaluations. It is also suggested that an evaluation itself should be participatory (Carneiro 2013; Hansen and Vedung 2010; Mickwitz 2006). Then persons who are engaged in evaluations can learn from participating in the evaluation process, and when evaluations are conducted as part of participatory planning or policy-making processes, the evaluation process and results reach the widest possible audience.

In addition to fostering learning, evaluation has other important functions. Checking for accountability of public policies is one. Accountability concerns the liability of those who are in charge of and conduct public tasks and spend public resources. Public resources should be used wisely, and it should be ensured that the goals regarding the quality of the process and results are achieved (Mickwitz 2006).

Evaluations can increase trust and the legitimacy of public authorities and processes, as they improve public knowledge and understanding of policies and plans. As evaluations may reveal flaws in processes and unachieved goals, it is important that evaluations contribute to the identification of corrective measures.

3 Alternative Evaluation Approaches

In order for evaluations to produce understandable and justifiable results, they should be done in systematic and rigorous ways (EC 2013a; Mickwitz 2006). This is a generic requirement, as there are many systematic and rigorous methods available to be used. A key issue is to choose the right methods for the purpose and scope of the evaluation. Obviously, resources dedicated to evaluation also determine the choice of methods to some extent.

There are several approaches and focuses for the evaluation of spatial planning. According to Terryn et al. (2016), the evaluation of spatial planning has often been based on a linear (or at least cyclical) understanding of planning processes. Consequently the evaluation methods have been structured in simple logical steps to be followed. Terryn et al. (2016, 1085) state that ‘most spatial developments do not evolve in a linear, circular or causal way, but rather present themselves more and more in a-linear, pragmatic and adaptive ways’. As this adds a certain level of uncertainty in determining the target of evaluation, they suggest that the evaluation should ideally be conducted as an integrated part of the planning process, as this would allow adjusting the methods to better fit the context of evaluation (Terryn et al. 2016).

The nonlinear and partly unpredictable character of spatial developments is an important point to be taken into account in the evaluation of spatial planning. However, there is also a need to make a distinction between the spatial developments and spatial planning. Spatial developments are outcomes of the combined effects of various processes, while spatial planning is a process that ideally translates the collective aspirations of society into decisions on how to use the area in question. In other words, spatial planning aims to influence, or at least contribute to, the spatial developments.

Even though spatial developments can be nonlinear and unpredictable, the levels of the complexity of the planning context vary—and some planning contexts are not at all complex. The level of complexity of the planning context should be taken into account in choosing the planning methods. Terryn et al. (2016) have studied how different planning approaches would fit different planning contexts. The following matrix presents how different evaluation approaches can fit different planning contexts (Fig. 18.2).

Fig. 18.2

Evaluation approaches in relation to the degree and reasons of complexity of the planning contexts (Terryn et al. 2016, 1087)

Circular evaluation (lower left in figure) is suitable for simple planning issues and situations when the main focus will be on how the intentions of planning meet the implementation. In such cases it is also well known who the key actors are, what the stakes are and what roles the institutional and non-institutional actors would have in the planning. In other words, the playing field is stable and known. An adaptive evaluation (upper left) approach is applicable when the planning issue itself is undefined and possibly changing, but the institutional and societal setting is relatively stable. Adaptive evaluations probe whether the final results meet the needs of changing contexts and various interests. Participative evaluation (lower right) is apt when the planning issue is simple, but there are uncertainties regarding the actors, stakes and possible roles of the different types of actors. In such a setting, the evaluation’s role as negotiation or dialogue is more important than objective attempts to measure the effectiveness of implementation of the plan. Participatory evaluations review the ability of interest groups to cooperate in a situation of changing playing fields. Finally, the co-evolutionary evaluation (upper right) approach is needed when both the planning issue and the playing field are not well known or they are in the process of transformation during the planning process or being transformed by the planning. Co-evolutionary evaluation asks if the planning itself is becoming more resilient and adaptive to be able to operate when both planning issues and the playing field are volatile. In such cases it would also be imperative that evaluation is continuous over the process to encourage learning-by-doing and co-evolution (Terryn et al. 2016, 1087–1088).

The importance of evaluations to be close to the evaluated process—or to co-evolve with it—has often been emphasised (Mickwitz 2006; Rae and Wong 2012; Terryn et al. 2016; Vedung 2010). Evaluations need to be sensitive to how the evaluated process unfolds. If need be, the evaluation approach itself should adapt—meaning, for instance, that new evaluation criteria can be learnt during the evaluation (Gomart and Hajer 2003).

Evaluations can have different targets and different timing in relation to the stage of decision-making or planning processes. The selected methodology should respect the nature and complexity of the object of the evaluation. These distinctions are discussed in the following sections.

3.1 Evaluation Can Target Impacts and Processes

Knowing the impacts of policies and plans is essential for evaluating the effectiveness of public policies. The evaluation will try to study to what extent the objectives of the policy or plan have been reached.

Spatial plans are typically aspirational: there are certain goals that are found to be valuable to reach. As pointed out earlier, spatial planning often operates in complex contexts where several factors have an influence at the same time. This happens especially in cases of strategic or general-level planning, which MSP often is. Only some of the factors that generate impacts follow directly from the spatial plan itself. Carneiro (2013) has observed that the current literature on MSP does not pay enough attention to the issue of multi-causality and has not sufficiently discussed the difficulty of isolating the contribution that MSP has or can have on observed changes in the use of sea areas. There is also another important limitation for spatial planning in reaching desired objectives: depending on the plan’s legal status, it may have only limited power to directly guide decision-making in other sectors. Then the effectiveness of a spatial plan depends on the other sectors’ willingness to follow that spatial plan (Faludi 2000).

How and to what extent are the observed changes attributable to the spatial plan? The notion of causality and attribution then becomes central in the evaluation of effectiveness (EC 2013a). In the evaluation of spatial planning that takes place in complex contexts such as MSP, attribution may become a considerable challenge (Carneiro 2013). When it may be very difficult, if not altogether impossible, to justify causality between the intervention and desired outcomes, that is the attribution, one can justify the argument of contribution with plausible evidence or a narrative that explains why the evaluated spatial plan can be seen as one of the causes of the observed change (Carneiro 2013; EC 2013a). Even if this cannot reduce uncertainty concerning the effects of plans, it can produce useful findings for improving performance of planning (EC 2013a).

The question of effectiveness is essential for the sake of accountability, but focusing only on the intended goals is often too limited. Identification of unintended consequences is widely recognised as an essential part of the evaluation of spatial planning, especially because spatial planning typically addresses and affects broad areas and a broad spectrum of human activities (Carneiro 2013; Faludi 2000; Mickwitz 2006; Terryn et al. 2016). Therefore, the narratives of MSP’s contributions should also pay attention to possible unintended consequences and side effects.

In addition to evaluating the effectiveness of policies and planning, it is also useful to focus on the processes. Evaluation of the process of implementation of a policy or plan not only helps to answer whether the results are met but also helps to understand why it is so. It can also help in explaining the observed results (Carneiro 2013). Furthermore, evaluation of the process of making a policy or a plan gives valuable information for improving the processes in the future, that is, double-loop learning (Mickwitz 2006). Some aspects of policy-making and spatial planning processes as well as processes of implementation have an important intrinsic value, which justifies paying attention to processes in the evaluation. The imperative of public participation in policy-making and spatial planning processes is an example of such intrinsic values. The requirements of transparency and accountability underline the need for focusing on processes (Carneiro 2013; Hansen and Vedung 2010; Mickwitz 2006).

Carneiro (2013) identified several possible foci of evaluations in relation to the planning cycle (Fig. 18.3).

Fig. 18.3

Different foci of evaluation in relation to steps of the spatial planning process (Carneiro 2013, 215)

3.2 Evaluation: Before, During or After the Intervention

Evaluations can be conducted while policies and plans are prepared. Such evaluations give valuable information when designing effective policies and plans. These so-called ex ante evaluations anticipate possible future impacts of planned policies. An ex ante evaluation should preferably produce results early enough in relation to the policy-making or planning process in order to have a valuable and timely contribution (EC 2013b). SEAs are also done while the policies and plans are prepared.

It is common that policies and plans are evaluated afterwards—ex post—or in the late stages of implementation to check whether and to what extent the set results are achieved. Ex post evaluations can also study unintended impacts of policies or plans.

Interim evaluations or mid-term reviews generate information that help to assess whether measures are being implemented as planned and whether it seems likely that they will produce the impacts that were anticipated. A more thorough interim evaluation can also help to assess whether the assumptions about a policy or plan’s effects were correct or not.

4 Evaluation of MSP

The importance of evaluating MSP and different approaches to an evaluation of MSP has already been discussed in early publications on MSP (Carneiro 2013; Day 2008; Douvere and Ehler 2011), but there have been fewer presentations on actual evaluation methods.

EU-funded projects such as TPEA, MASPNOSE, BaltSeaPlan, PlanBothnia and PartiSeaPate have all addressed evaluation of MSP. For instance, TPEA and MASPNOSE produced evaluation frameworks and also tested them to a certain extent during the projects, while PlanBothnia developed approaches to monitor the implementation of MSP. Evaluation approaches of MSP have also been developed in several academic papers (Carneiro 2013; Day 2008; Douvere and Ehler 2011; Fletcher et al. 2013; Kelly et al. 2014; Soma et al. 2014; Stelzenmüller et al. 2013; Vos et al. 2012).

There are also some publications that reviewed the evaluation of MSP (Carneiro 2013; TPEA 2014). The TPEA evaluation report identifies the diversity of evaluation approaches (e.g. the focus can be on ecological or planning aspects; the emphasis is on process or outcomes). And the report concludes that because of the diverse contexts in which MSP is practised, there cannot be a standardised protocol for evaluating MSP. Each evaluation has to be tailored to the context (TPEA 2014).

Even if questioning the usefulness of standardised evaluation approaches, the TPEA evaluation report (TPEA 2014) presents a few general principles:

• Evaluation of MSP should cover all stages of the MSP process, from preparation of planning to implementation.

• Evaluation should be based on a clear understanding of the focus and scope of the evaluation, which helps in defining clear objectives for the evaluation.

• Evaluations should cover context, process, outputs and outcomes.

• Evaluation criteria should be matched by suitable indicators.

• Stakeholder involvement is important for a successful evaluation.

Table 18.1 presents evaluation frameworks that were suggested by the TPEA evaluation report (2014) and Carneiro (2013). The frameworks have many obvious similarities, but they also introduce some unique features. The TPEA approach is more detailed regarding legal, administrative and institutional aspects, which are especially critical for the success of transboundary collaboration in planning—the focus of the TPEA evaluation framework. Carneiro’s (2013) evaluation framework emphasises the content of the plan itself without neglecting the importance of process evaluation. This framework is mainly meant for the evaluation of national MSP.

Table 18.1 Topics and criteria of two MSP evaluation frameworks (Carneiro 2013; TPEA 2014)

The United Nations Educational, Scientific and Cultural Organization (UNESCO)’s International Oceanographic Commission (IOC-UNESCO) has been promoting MSP and developing methodologies of MSP from very early on (see Ehler and Douvere 2009). IOC-UNESCO produced a guide to evaluate marine spatial plans in 2014 (Ehler 2014). The evaluation guide focuses mainly on outcome evaluation, but it also raises important questions regarding evaluation of the processes. The IOC-UNESCO guide covers the whole sequence of evaluation, from planning the evaluation via the actual evaluation to communicating the evaluation results. These are all also addressed in the TPEA evaluation report (TPEA 2014), but the IOC-UNESCO guide goes one important step further by discussing the use of evaluation results and taking corrective measures.

5 A Theory-Based Evaluation Framework for MSP

Introducing a theory-based evaluation approach to MSP is this chapter’s contribution to our common understanding of possibilities and methods for evaluating MSP. There are two main reasons for introducing this approach.

One reason is that as evaluation is a careful assessment (EC 2013a; Mickwitz 2006) that requires systematic and rigorous approaches to evaluation in order to produce understandable and justifiable results, systematic and rigorous evaluations are time-consuming. Therefore, those who commission evaluations need to consider the available methods of evaluation in relation to the expected use of the evaluation findings and the available resources (EC 2013b). Thus there is a need to adjust the evaluation approach to the context and knowledge requirements.

The other reason relates to the difficulty in isolating and identifying the actual effects and impacts of MSP from all other factors that influence maritime activities and marine ecosystems. Is MSP contributing to the changes that we can observe? (See Sect. 3.1).

A theory-based approach to evaluation is flexible in the sense that it must always be adjusted to the context and purposes of evaluation. Furthermore, constructing theories of change, which is the key for theory-based evaluation, is also a way to increase knowledge of the possible contributions of MSP.

Theory-based evaluations ask why an intervention—such as a spatial plan—produces intended and unintended effects, for whom and in what contexts and what mechanisms are triggered by the intervention. The goal is to know why an intervention works and whether it would work differently in different localities (Astbury and Leeuw 2010; Coryn et al. 2011; EC 2013a).

5.1 Theories of Change

A theory-based evaluation of spatial planning is based on describing plausible mechanisms through which the plan or the planning process can produce its impacts. The actual evaluation then collects evidence to test whether the implementation of the plan unfolded as anticipated (and why), whether the anticipated results were achieved and whether the implementation of the plan produced any unintended impacts (Coryn et al. 2011; Hansen and Vedung 2010; Mayne 2012; Weiss 1997). A theory-based evaluation does not usually produce numerical results as much as it produces narratives. Its results provide important insights into how spatial planning can work, and later why it worked as it did (EC 2013a).

The key element of theory-based evaluation is the theory of change.¹ The term theory-based reflects the understanding that all decisions and plans are based explicitly or implicitly on an idea—a theory—about how that decision or plan will be implemented and how it will produce results, that is, a theory of change. Theories of change are typically described as somewhat simplified, often linear models (Fig. 18.4). Obviously, spatial plans and the generation of their outcomes are not always, or even usually, as linear as depicted in the following figure (see Astbury and Leeuw 2010; Coryn et al. 2011; Hansen and Vedung 2010; Mayne 2012).

Fig. 18.4

A scheme of a theory of change (Coryn et al. 2011, 201)

An intervention consists of inputs, activities and outputs. Inputs are the required resources (e.g. human, financial, institutional). Activities are the actions taken to define and reach the objectives (e.g. data collection and spatial analyses, production of the plans and planning documents, workshops with stakeholders, consultation). Outputs are the immediate results of action (e.g. the planning decisions).

Impacts of the intervention can be grouped into initial, intermediate and long-term outcomes. Initial outcomes are changes in knowledge, skills and ability of key actors. Intermediate outcomes are typically behavioural changes (e.g. a decision to invest in sea areas designated in MSP). And long-term outcomes (sometimes simply called impacts) are a full or partial solution to the perceived problem that the plan was set to address (an outcome corresponding to one of the objectives for MSP) (Coryn et al. 2011, 202). Table 18.2 gives an example of this sequencing.

Table 18.2 An example of a plausible theory of change in MSP

5.2 Using Theories of Change in Evaluation

Construction of plausible theories of change is a key element of theory-based evaluation, but the actual evaluation focuses on the following if the anticipated steps will or can take place. It is essential to study why the plan produces or could produce the intended and unintended effects; for whom and in which contexts; what mechanisms are triggered by the plan or by the process of producing the plan; how various steps in the theory of change relate to each other and what factors influence the relations (Mayne 2012). These considerations are depicted in Fig. 18.5.

Fig. 18.5

Theory of change, considering factors that influence a logical sequence of events. Modified from Coryn et al. (2011) and Mayne (2012)

In elaborating the theories of change as step-wise developments, one risk is ‘to focus too much on input-output relationships, on linear chains of causality and on building tightly knit models of arrows and boxes’ (Weber 2006, 120), which is an important reminder to acknowledge the complexity and situatedness of the planning process. Astbury and Leeuw (2010, 375) suggest that ‘a more explicit focus on underlying generative mechanisms might help to counter […] toward oversimplified versions of program theory in the form of linear logical models’.

Theory-based evaluations often rely on participatory approaches to deal with different understandings and preferences. Such processes can help reach jointly agreed theories of change. But Hansen and Vedung (2010) have observed that due to substantive and multilevel complexities and political conflicts, this is not always possible.

It is not always advisable to reduce different perceptions on the intervention to only one theory of change. This is especially important in interventions that ‘involve several groups of actors in very different working situations and with very different expectation to the intervention’ (Hansen and Vedung 2010, 296). Implementation of MSP typically takes place in situations where success of the plan’s implementation is dependent on how the plan changes the behaviour of actors in various marine sectors (Faludi 2000). Approaches that elaborate on alternative, parallel theories of change are needed in complex and conflict-prone interventions that operate nationwide, multisite and multilevel (Hansen and Vedung 2010).

5.3 Testing of Theories of Change

Producing alternative theories of change in collaboration with key actors helps to identify possible impacts and challenges of implementation of MSP in a systematic and transparent way. The actual evaluation tests to what extent the actual cause of events followed or could follow the theories of change and whether or not the goals were reached. Importantly, the evaluation should try and explain why. Theory-based evaluation can be implemented in the following five steps (modified from Coryn et al. 2011, 205; EC 2013a, 56–57):

• 1. Formulate plausible theories of change to reflect different actors’ understandings.

• 2. Formulate and prioritise evaluation questions around a theory of change:

  • a. How can you know that the different steps of the theories of change (will) actually take place? What evidence do you need?

  • b. Choose relevant evaluation criteria and indicators.

• 3. Collect evidence relevant for answering the evaluation questions.

• 4. Analyse the evidence to test the theories of change:

  • a. Which links in the theory of change are strong? Is this conclusion based on strong logic or empirical evidence supporting the assumptions? Is this conclusion widely accepted by relevant actors? And similarly, which links are weak?

  • b. Does the observed pattern of outcomes and factors leading to them validate the theory of change? Do or did things unfold as anticipated?

  • c. Is it likely that any of the external significant factors had a noteworthy influence on the observed results?

  • d. What are the main weaknesses in the descriptions of the theories of change? Would additional data or information be useful?

• 5. Draw conclusions:

  • a. Identify breakdowns (links that did not exist) and respective corrective actions.

  • b. Identify side effects and unintended impacts (also identify who was affected).

  • c. Determine the effectiveness of implementation. (Were the objectives reached and to what extent?).

  • d. Describe and explain cause-effect associations between elements of the theories of change (why things unfolded as they did). Describe also why external factors influence (or influenced) the outcomes.

6 Organisation of Evaluation

Section 5 introduced the theory-based evaluation method. This final section focuses on pragmatic questions on how to organise and manage the evaluation process. There are certain essential steps and challenges that need to be solved. One of the most important issues is to define the scope of the evaluation. One must also decide who is in charge, who performs the evaluation and how many resources will be allocated to the evaluation. Finally, there is a need to decide on the roles stakeholders are given in the evaluation (EC 2013b).

6.1 Targeting the Evaluation: Scope and Purposes

Defining the scope of the evaluation is the most important first decision to be made. The evaluation should be given certain boundaries in terms of institutional, temporal, sectoral and geographical dimensions (EC 2013b). In defining the scope, one asks what exactly will be evaluated and when? It is also essential to consider the expected and possible uses of the evaluation results, which was also raised as an important factor in the IOC-UNESCO evaluation guidance document (Ehler 2014). It is important to understand what future decisions are likely to be informed by the evaluation results (EC 2013b).

For the success of the evaluation, the scope should be defined in a way that gives a clear focus and task for the evaluation. It may turn out, however, that a clear scope for the evaluation cannot be given. This may be the situation especially when a country is producing its first MSP. In such cases the scope of the evaluation needs to be somewhat flexible in the beginning but should be defined more precisely while the process unfolds. This would suggest that the evaluation should be conducted in close cooperation with the planners. However, a flexible scope for evaluation has to be considered against the availability of resources and time for conducting the evaluation. Evaluation questions and the scope should be realistic in relation to the resources, which often require clarity.

6.2 Financial and Institutional Resources for Evaluation

Evaluations should be conducted in systematic and rigorous ways to produce justifiable and relevant results, but the possibilities to live up to this standard can be limited by the availability of resources. However, the most important question for the planning and commissioning of evaluations is to identify the expected use(s) of evaluation findings and fit the resources accordingly.

Resources for evaluation include the financial resources, obviously, but there are other resources available as well. Actors to be involved in the evaluation can give valuable input and information for the evaluation. Also the end users of the evaluation findings are a key resource.

Who is in charge or who is the client for evaluation is critical for the usefulness and actual use of the evaluation findings. It is recommended that the person (or a function in administration) who commissions the evaluation should be in a high enough position to initiate corrective actions to the policies or plans that are being evaluated (EC 2013b). Regarding MSP, this could be the minister in charge of MSP, a representative of the ministry or a high-ranking officer in the spatial planning authority.

Evaluations are often commissioned by consultants who operate under a contract with a public organisation; they monitor or evaluate the policy or planning process being evaluated. If an external evaluator conducts the evaluation, it is important that the evaluation is conducted in close and frequent contact with the client. This ensures that the evaluation results are immediately available, and this will also give an opportunity to adjust the evaluation if new needs emerge or in case the evaluated process is reorganised. Information will then flow in both directions between the evaluator and the client.

In some cases, the public bodies have their own evaluation units. Then it would be advisable that the evaluation is conducted as an in-house service (EC 2013b). It is also possible that the officers who are running or supervising the spatial planning process conduct the evaluation in-house. Such arrangements may create challenges of partiality. In such an arrangement, it is advisable that some of the officers have experience in evaluation methods. If it is decided that the evaluation is conducted in-house and only limited expertise in evaluation methods is available, it is advisable to hire a consultant to facilitate the evaluation process.

The purpose and timing of the evaluation determines to some extent whether the evaluations should be conducted internally or externally. The European Commission’s (EC 2013b, 39) evaluation guidance advises that ‘[i]t may be preferable to rely more on internal resources for formative evaluation inputs or for ex ante exercises but depend more on external resources for the ex post evaluation’. Formative evaluations aim at improving the design and performance of policy-making or spatial planning processes usually while they are conducted. Ex ante evaluations have similar objectives, but they are conducted before the processes and also have predictive aims.

The character of issues dealt with in the spatial planning process should be taken into account when deciding whether the evaluation will be conducted internally or externally. If the process will address issues that are known to be controversial, an external evaluation will probably be better received. In such situations, an internal evaluation might appear to be simpler to conduct, but it must be acknowledged that an in-house evaluation of a controversial process will shed doubts on the reliability and impartiality of the evaluation.

An assessment of the financial resources needed for conducting or commissioning evaluations needs to be carefully considered against the anticipated purposes and expectations for the evaluation (EC 2013b). It is generally estimated that an evaluation of a rather routine policy or planning process would require a relatively small amount of money in proportion to the resources for the whole process—normally less than 1%. For evaluations of extensive and new types of policies or spatial planning processes, and if there are high-learning expectations and substantial investment in stakeholder participation, the costs are likely to be relatively high in proportion to the overall programme costs—up to 10% (EC 2013b). The EU Commission’s (DG REGIO) guidance document points out that ex ante evaluations usually have a rather limited time and limited possibility of acquiring data for the evaluation. Then the required resource is not that high in comparison to evaluations that come in later stages. Especially interim evaluations, if they have strong formative ambitions, may require a lot of evidence and extensive stakeholder engagement, which increases both the costs and time needed. Ex post evaluations do not necessarily require substantial resources, depending on the scope given for the evaluation. In conclusion, the most important factor that determines the required budget is the nature and scope of the evaluation (EC 2013b).

6.3 Stakeholder Engagement in Evaluations

It is common and even recommended that evaluations engage stakeholders at different stages (EC 2013b; Hansen and Vedung 2010). It is important to notice that the earlier discussion about whether to conduct the evaluation in-house or by external consultation is not related to the need for engaging the stakeholders—the evaluations conducted as an in-house service should also aim at engaging the stakeholders. In fact, due to the possible risk that in-house evaluations are perceived as not being transparent and neutral, it is even more important for such evaluations to be inclusive.

Stakeholder involvement is advisable for two broad reasons. First, stakeholders possess expertise, knowledge and information that can be an invaluable resource for the evaluation. For instance, in evaluations that are run within a limited time and with limited resources, well-organised stakeholder involvement can be decisive for the success of the evaluation. The second reason follows from an often-cited definition of a stakeholder: ‘a stakeholder is an individual or group influenced by—and with an ability to significantly impact (either directly or indirectly)—the topical area of interest’ (Glicken 2000, 1). Various individuals, communities and organisations are affected, positively or negatively, by the spatial plan that is being evaluated. Therefore, they have an interest in the evaluation results and outcomes. Stakeholders’ willingness is also important for successful implementation of the spatial plans, especially if the plans are nonbinding (Faludi 2000).

Stakeholders are a resource for evaluations in their capacity to provide information and insights that help design and implement the evaluation. It has been suggested that stakeholders should be involved at all stages of evaluation processes (Carneiro 2013; EC 2013b). In the early stages, their input can be valuable in defining the scope of the evaluation and in outlining the key evaluation questions.

It has been emphasised that theories of change should ideally be constructed together with the stakeholders, since in complex environments it is likely that there are well-justified alternative understandings of possible impacts and how they might be generated by the planning decisions (Hansen and Vedung 2010; Mayne 2012). Hansen and Vedung (2010) even point out that elucidation of different understandings of how interventions might play out and different valuations of the impacts is often one of the most important results of theory-based evaluations.

Finally, the stakeholders should be given an opportunity to comment on the evaluation results (Carneiro 2013). Participation of the stakeholders at different stages of evaluation aims at ensuring that there is ownership of the evaluation findings (EC 2013b).

Table 18.3 presents a summary of Sects. 5 and 6 as practical steps of the evaluation.

Table 18.3 Steps of an evaluation process