Multi-stakeholder Preference Analysis in Ex-ante Evaluation of Policy Options - Use Case: Ultra Low Emission Vehicles in UK

Abstract

While the simulation-based impact assessment of public policy proposals allows policy makers to identify the feasible policy options and verify their economic, social and environmental impacts, it does not provide the explicit evaluation of policy options. Multi-criteria decision analysis (MCDA) techniques can support an in-depth performance evaluation of policy options taking into account the preferences of decision makers and stakeholders. These preferences reflect acceptable trade-offs of performance among objectives. This study reviews multi-attribute decision-making (MADM) technique and presents a common policy appraisal format using main evaluation criteria linked to a set of measurable, context dependent attributes. We argue for a rank-based approach for eliciting preferences, select a novel method for attribute weight elicitation, and show how it can be integrated within a public policy multi-criteria evaluation framework. A use case for policymaking, ‘Ultra-Low Emission Vehicles (ULEV) Uptake in UK’, is used for demonstration of the proposed approach for policy decision analysis. This approach seeks to couple systems modelling and simulation of policy scenarios with MCDA, stakeholder analysis and preference elicitation. The outputs can further provide analytical insights in controversy/acceptability of policy options, and consequently guide further policy formulation and the design of better options.

1 Introduction

Public policy decision processes are often characterized by the presence of multiple and conflicting objectives and multiple stakeholders or decision makers who may have differing point-of-views. Using decision analysis practices to provide informed public policy decisions is not a novel approach per se. What might differ public policy decision analysis from a traditional decision analysis endeavor is that the former does not explicitly aim to reveal a clear recommendation of choice, but also to explore the problem and reveal conflicts, cf. (Quade 1982).

Prescriptive approaches for the study of such decision processes have been suggested within the field of systems thinking and in the field of multi-criteria decision analysis (MCDA). In (Larsson and Ibrahim 2015), we presented a work process with associated operational research (OR) modeling and analysis tools that supports a prescriptive analysis for policy that includes: the problem definition, ex-ante impact assessment and evaluation activities carried out at the policy formulation stage of the policymaking process. The proposed approach exploited the use of causal maps for problem structuring and scenario-based simulation for the design of policy options, together with decision analysis for evaluating generated scenarios. Having a set of feasible policy options identified, there are two main tasks remaining in structuring MCDA evaluation models; (i) representation of objectives in a structure, commonly a value tree, and (ii) the definition of attributes to measure the achievement of objectives (Franco and Montibeller 2010).

Multi-criteria evaluation can be organised with an objective to produce a single synthetic conclusion or to produce conclusions adapted to the preferences and priorities of several actors. Thus, applying MCDA should provide the relative global performance of each alternative taking preferences of decision makers and stakeholders into account. This is particularly useful when selecting one out of a finite set of feasible alternatives.

Since the rise of the graphical user interface, a flora of computer based decision analysis tools have emerged, aiming to exploit the strengths of graphic interaction with users and thereby enabling for users with less know-how of decision analysis to conduct decision modelling and evaluation, see, e.g., (Riabacke et al. 2014). A prerequisite to increase the practical aspects of decision analysis is to offer accessible tools, and of greatest importance for practical applicability of decision analysis methods seems to be how “easy” they are to use (Stewart 1992, Hülle et al. 2011).

This work is carried out within the contexts of the research project Sense4us¹, which is developing a web portal that integrates public policy decision support tools to linked open data search tools and social media analytics, in order to enable easy access to information sources and knowledge creation. The aim of this paper is the design of a preference elicitation method based on a multi-criteria evaluation model for policy decision analysis. The method should: (i) not require substantial formal decision analysis knowledge; (ii) not be too cognitively demanding by forcing people to express unrealistic precision or to state more than they are able to; (iii) not require too much time; (iv) make use of the information the decision-maker is actually able to supply, and (v) be supported through a graphical user interface (GUI) accessible from different client operating systems.

This paper is structured as follows. Section 1, introduces the research problem, context, challenges and tasks. Section 2 provides a review of the state-of-the-art research related to concept and application of decision analysis and preference elicitation for policy analysis. Section 3 presents the proposed criteria model for ex-ante evaluation of policy options. Section 4 presents the proposed rank-based preference elicitation method. Section 5, describes the public policy use case, as real example for demonstration of the use of the proposed policy decision analysis process. Finally, Sect. 6, provides the conclusions and possibilities for future work.

2 Related Work

Elicitation of preferences in policy making requires that there exists a basis for decision, typically in the form of more or less objective so-called “impact assessment reports” describing the impact of different alternatives on objectives but strive to be neutral with respect to decision maker preferences. The impact assessment approaches can be more or less formalized, and examples used together with decision analysis approaches within the context of policy making published in the literature include, e.g., life-cycle assessment (Miettinen and Hämäläinen 1997), quality adjusted life-years (Drake et al. 2009, Kivunike et al. 2014), life micro-level simulation and system dynamics (Hansson et al. 2008, Gou et al. 2001), scenario planning (Montibeller et al. 2006), and problem structuring using causal maps (Comes et al. 2011). One important feature of problem structuring approaches is to facilitate the context-setting activity and defining the environment of a decision problem to be modelled and subsequently evaluated, (see, e.g. Belton and Stewart (2002) for an introduction to how problem structuring relates to the decision analysis process).

Further, the use of decision analytic support in participatory/group decision processes involving several decision makers and/or stakeholders has received increasing attention during the last few decades. The decision makers can actually disagree on what the best alternative is but they need to select a common alternative having sufficient support from the group. Within such a context, a number of proposed decision analytic approaches and cases of its use have been published in the literature. For instance, Danielson et al. (2007) propose the analytical decision layer process, incorporating the stakeholders and decision maker views in an interaction layer and aggregating these preferences in a decision layer exploiting methods from interval decision analysis to accommodate for differing preferences such that all decision maker values are captured by each interval. Hansson et al. (2008) presents an MCDA approach for evaluating flood management strategies, putting weights on the stakeholders when aggregating their preferences. Schroeder and Lambert (2011) outlines a decision analysis process for comparison of transportation policies in the case of an emergency exploiting scenario planning in order to set the context.

Multi-Attribute Decision Making (MADM) techniques can be applied when the decision consists of selecting one out of a finite set of feasible alternatives, (Belton and Stewart 2002). The process of MADM is an analytical process that brings together three components; the decision objectives as measured by their associated attributes, the decision options or alternatives and the decision maker’s preferences toward the importance of the attributes as reflected by the attribute weights.

The determination of attribute weights is an important and time consuming process. Attribute weight elicitation methods, given the decision problem’s structural elements, include: the Balance Beam (BB), Lottery Technique (LT), Paired Comparison (PC), Point Allocation (PA), Pricing Out (PO), Ratio Weighting (RW) and Trade-off Weighting (TW), (for a comparative review see (Crain 2003)). These methods can be distinguished by the specific technique employed to compute the attribute weight and whether the technique is designed to determine an attribute importance weight or an attribute swing weight. Importance weights reflect the relative preference without regard to the decision problem at hand, whereas swing weights are sensitive to the values that the attributes possess for the particular decision problem. Crain (2003) emphasized the greater utility and correctness of eliciting swing weights in a MADM problem versus importance weights because of the sensitivity of the swing weight to the range of values taken on by the associated attribute.

A common practicability issue of applying decision analysis methods is that often too hard requirements are put on the users. It can be argued that much of the applicability issues in employing formal decision analysis in practical settings roots in the need to elicit beliefs and preferences from decision makers and stakeholders. One recurrent practicability issue reported in the literature is the need to obtain the decision-maker’s preferences in such a way that they can be formally represented. That leads to cognitively demanding methods less encouraging for decision makers to use. See, for instance (Riabacke et al. 2012, Riabacke et al. 2014). One increasingly popular way of relaxing the cognitive burden on decision makers is to rank criteria and/or alternatives and using cardinal rankings surrogate values can be generated. Recent studies of rank based approaches include (Sarabando and Dias 2010, Wang and Zionts 2015, Danielson and Ekenberg 2015a), showing that by allowing for the user to provide preference strength information, the equitability of the generated surrogate numbers will improve, (i.e. preferences of the decision maker are more properly reflected).

3 A Common Policy Appraisal Format

In MADM techniques, the decision process starts by structuring the problem as an attribute tree hierarchically ordering the decision makers’ aims at different abstraction levels. It is generally assumes that each criterion can be operationalised by a set of measurable attributes allowing for assessing the consequences arising from the implementation of any particular alternative. In the next step preferential information is elicited. The relative importance of criteria is captured in weights $w_{jl}$ for each criterion $j$ at each abstraction level $l$. At the lowest level of the value tree these objectives are translated into attributes, with each one of them evaluating a given characteristic of the decision options (for example, an objective ‘efficiency’ may be measured by the attribute ‘operating cost’). The performance of each decision option against each attribute is determined and weights are elicited from the decision-makers (Comes et al. 2011).

Figure 1, illustrates a multi-attribute value tree, with measurable attributes in gradient boxes, criteria and overall goal in white boxes. Dependence of attributes on the alternatives (represented as diamonds) is shown by dashed lines.

Fig. 1.

(Comes et al. 2011)

Multi-attribute value tree.

Given a set of decision options, $A = \{ a_{1} , \ldots ,a_{N} \}$ and $M$ evaluation criteria, we can represent a decision option $a_{k}$ with a vector of performance levels $(x_{k1} , x_{k2} , \ldots , x_{kM} )$. Conforming to multi-attribute value theory (MAVT), the global value of a decision option $a_{k}$ is given according to the additive value function:

$$ V\left( {a_{k} } \right) = \sum\nolimits_{i = 1}^{M} {w_{i} v_{i} (x_{ki} )} $$

(1)

where $v_{i} (x_{ki} )$ is a value function representing the value of alternative $a_{k}$ under criterion $i$, $w_{i}$ is the weight of criterion $i$, $0 \le w_{i} \le 1$ and $\varvec{\varSigma}w_{i} = 1$.

This additive model cannot capture the dependencies between criteria, (i.e., being substitutes or complements of each other). We refrain from eliciting such dependencies from decision makers, to achieve the use of use of the elicitation method.

We propose a criteria model for ex-ante evaluation of EU policy interventions, based on a set of main evaluation criteria² Effectiveness, Efficiency, Coherence, Relevance and Added value and the underlying measurable attributes. For each of the policy options, the observed changes on the targeted policy impact variables and the policy financial impacts are provided by the impact assessment results. Figure 2, illustrates the structure of the proposed common policy appraisal format as a multi-attribute value tree. The figure shows $k$ policy options to be evaluated.

Fig. 2.

Common policy appraisal format

Each of the attributes underlying the “effectiveness” evaluation criterion reflects how successful a policy option is in achieving the policy goals by comparing the observed change (OC) to the targeted change (TC) for each impact variable. The attributes underlying the “efficiency” criterion are concerned with the financial impacts of a policy option, whether financial costs or benefits (cost savings). The evaluation of the impact of the action is expressed as the costs involved on governmental departments given the achieved changes. In addition costs on businesses (including small businesses), or different stakeholder groups are expressed in terms of different attributes. The efficiency evaluations can be made in a quantitative or a qualitative way. The attributes underlying the “relevance” criterion are using qualitative descriptors (or value scales) to assess the relevance of a policy option to the policy objectives, to the problems/needs, to the technological advances and to the citizens. The attributes underlying the “coherence” criterion are using qualitative descriptors or value scales to assess the coherence of a policy option: (i) internally if it includes multiple policy instruments, (ii) with other similar interventions, (iii) with EU policy interventions and (iv) with international obligations. Finally the attributes underlying the “added-value” criterion are using qualitative descriptors or value scales to assess the added value of the policy option in terms of legal impacts, increased effectiveness, complementarities and the need for continuing this intervention.

4 Weight Elicitation Method

The proposed preference elicitation method to be used is a rank-based approach complemented with intuitive ways of generating a value function for policy decision evaluation. The value of each option under each criterion from the perspective of the decision maker is then captured in a so-called value function $v(x)$ such that $v: \varvec{X} \to [0,1]$ where $\varvec{X}$ is the range of the performance indicator. Further, if we assume positive preference direction (the more the better) and let $x_{min}$ denote the worst performance level and $x_{max}$ the best, then $v(x_{min} ) = 0$ and $v(x_{max} ) = 1$. Two simple ways of obtaining values on intermediate performance levels are suggested.

• (1) Either to assign them proportionally such that $v(x) = (x\,{-}\,x_{min} ) / (x_{max}\,{-}\,x_{min} )$ if the preference direction is positive, or $v(x) = (x_{max}\,{-}\,x) / (x_{max}\,{-}\,x_{min} )$ if the preference direction is negative. This is often labelled as “proportional scores” and is an intuitive way to generate a value function.

• (2) Use surrogate values derived from ranking statements according to the novel CAR method (Danielson and Ekenberg, 2015b).

For the CAR method, we use $>_{s(i)}$ to denote the strength of the rankings between criteria or alternatives, where $=_{0}$ means that they are equally ranked, $a_{i} >_{1} a_{j}$ means that option $i$ is better important than option $j$, and, $a_{j} >_{2} a_{k}$ means that option $j$ is “much better” than option $k$ and so forth. This can be represented such that each option is assigned a preference position $p(i) \in \{ 1, \ldots ,Q\}$ where lower position indicate stronger preference such that for two options whenever, $a_{i} >_{s(i)} a_{k}$ then $s\left( i \right) = |p(i) {-} p(k)|$ and the surrogate value is simply given from

$$ v_{i} = \frac{(Q - p(i) + 1)}{Q} $$

(2)

Given a slider with in total Q number of importance scale positions. Each criterion i has the position $p(i) \in \{ 1, \ldots ,Q\}$ on this importance scale where lower position indicate more importance, such that whenever $c_{i} >_{s(i)} c_{j} ,\,s(i) = \left| {p(i) - p(j)} \right|$, then the surrogate weight is given by:

$$ w_{i} = \frac{{\frac{1}{p(i)} + \frac{(Q - p(i) + 1)}{Q}}}{{\mathop \sum \nolimits_{j = 1}^{N} (\frac{1}{p(j)} + \frac{(Q - p(j) + 1)}{Q})}} $$

(3)

For example, Fig. 3 shows four criteria and a scale with eight steps where scale position $1$ is at the right end of the slider and scale $8$ at the left end of the slider. The ranking visualized is $c_{4} >_{2} \,c_{3} ,c_{3} >_{1} \,c_{1}$ and $c_{1} >_{4} c_{2}$, yielding surrogate weights $w_{1} = 0.21, w_{2} = 0.06, w_{3} = 0.26$, and $w_{4} = 0.47$.

Fig. 3.

Ranking of criteria

Further, following proportional scores value assignments for all three criteria but the Social criterion, for which we have the following ranking for options 1, 2, and 3, a ₁ > ₃ a ₃, a ₃ > ₂ a ₂ yielding values v ₃(x ₁₃) = 1, v ₃(x ₃₃) = 0.4, and v ₃(x ₂₃) = 0.

This can be illustrated using a multi-criteria evaluation matrix, constructed with as many columns as there are criteria and as many rows as there are decision options to be compared. Table 1 shows exemplary impact assessment results, each cell represents the performance $x_{ij}$ of option $i$ under criterion $j$ in the performance indicator used for that particular criterion. Note that: MCDA requires an evaluation of all the decision options for all the criteria, but does not require that all the evaluations take the same form. The technique can support a mix of quantitative criteria expressed by indicators, qualitative criteria expressed by descriptors, and intermediate criteria expressed by scores (e.g., a scale 0–10).

Table 1. Impact assessment matrix

In Table 2 below, we denote positive preference direction with ‘↗’ and negative direction with ‘↘’ together with S or R representing “proportional scores” or “ranking” respectively for the value function model. Unequal weights are assigned to criteria (note that the sum of the weights $w_{1} , \ldots , w_{n}$ equals $1$).

Table 2. Multi-criteria evaluation matrix

Thus, the preference elicitation methods proposed for enabling decision evaluation in the policy formulation process are:

• For decision option/alternative values: Cardinal ranking or proportional scores, complemented with direct rating.

• For criteria weights: Cardinal ranking, complemented with direct assessment.

The reason for complementing the ranking approach with direct rating is that the latter provides flexibility for exploiting other preference elicitation schemes for advanced users.

5 Use Case: Ultra Low Emission Vehicles (ULEVs) in UK

Department for Transport, UK, (2015),‘ Uptake of Ultra Low Emission Vehicles in the UK’. A Rapid Evidence Assessment, available on: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/464763/uptake-of-ulev-uk.pdf (Accessed 1/4/2016).

The British Government wants to increase take up of Ultra Low Emission Vehicles (ULEVs) throughout the UK, as part of its wider plans for reducing greenhouse gas emissions. Electric cars are deemed to reduce CO2 emissions, improve urban air quality, create new jobs, and provide other benefits to society. Amongst a set of initiatives to improve the uptake of electric cars, Plug-in Car Grant, Plugged-in Places, London congestion charge, while many others are being proposed, such as creating exclusive city “green-zones”, free parking for electric cars, investing in public recharging infrastructure, arranging hands-on trials for electric cars for citizens, and others. While all these policy interventions can be argued for with means-end rationality, they inevitably impact different stakeholders. Finally, when considering several policy options it’s not clear how to compare their acceptability.

Analysis of the national UK stakeholders for the use case is needed to provide answers to the following questions:

• 1. What are the key stakeholder groups and how are they related to the policy instruments?

• 2. How would a policy option affect the variety of stakeholders?

• 3. In what extent might the stakeholders be favouring or be against the transition or a specific policy option?

• 4. How to accommodate this information to prescriptive decision and policy analysis?

The identified stakeholders have been grouped in order to better articulate potential opponents, reflect the national (UK) level, and reduce the number of stakeholder groups to a manageable number. The list of selected stakeholder groups includes: government, citizens likely to switch to plug-in cars, citizens unlikely to switch to plug-in cars, electricity suppliers, plug-in car manufacturers, conventional cars manufacturers, renewable energy producer s, fossil fuel energy producers, power grid operators, petrol producers. “Citizens likely to buy plug-in cars” fully or partially encapsulate other stakeholder groups (citizens) with particular characteristics, such as early adopters, urban citizens, can drive, sufficient income to buy an EV, etc. Similarly, “Citizens unlikely to buy plug-in cars” can be characterised as risk-averse to new technologies, citizens with insufficient income to buy an EV, etc.

Preference elicitation implies (1) defining the direction of preference and (2) intensity of preference for each concept/stakeholder pair. The direction of preference refers to whether the higher value of a concept corresponds to a higher preference value (unidirectional, denoted by +1) or lower (oppositely directed, denoted by −1). The preference intensity is a weight showing relative importance of a concept. Each stakeholder group has been profiled in terms of their needs, goals and means for their achievement. Then, for each stakeholder assessing how increase or decrease in each concept would impact (positively, negatively, or no impact) stakeholders’ needs, goals and means. In this way, it was relatively straightforward to define preference directions or neutrality (no interest) with respect to the policy evaluation criteria. The selection of the concepts as evaluation criteria and/or attributes for each stakeholder group, has been based on the following principles: (1) the criteria set should represent balance and diversity of aspects for a given stakeholder group, (2) the criteria in the set should be relevant and representative, and (3) preferentially independent for a focal stakeholder group.

For instance, as a representative of the “Government” stakeholder group the Office for Low Emission Vehicles (OLEV) of the British government has been selected. The following attributes were considered for evaluation of the effectiveness of policy options by the government stakeholder group: ULEV industry economic value added, new jobs created, green-house gas (GHG)/CO2 emissions, urban air pollution and local electricity demand.

The proposed policy decision analysis process supports a visual hierarchy of criteria. Figure 4, shows an example for the resulting criteria hierarchy tree with exemplary criteria and attributes weights.

Fig. 4.

Visual hierarchy of the evaluation criteria for the use case

A relative global performance of the policy options for stakeholders can be calculated through a step-wise decision evaluation process to define: (i) Stakeholder groups and/or individual stakeholders; (ii) Evaluation criteria and/or the underlying attributes, each stakeholder stipulates its own set of criteria and attributes; (iii) Scenarios/Alternative policy options are common for all stakeholders; (iv) Scenario values: using cardinal ranking statements or surrogate value statements (a point, an interval or both) by each stakeholder; (v) Criteria weights: using cardinal ranking statements.

Figures 5 and 6 provide examples for ranking of criteria or scenario values utilizing a friendly GUI.

Fig. 5.

Example for criteria ranking using a GUI

Fig. 6.

Example for scenario values ranking using a GUI

6 Conclusions

The proposed common policy appraisal format provides a basis for computational decision analysis for evaluation of policy options. Further, identifying the stakeholders groups enables appraising a policy from the perspective of multiple stakeholders with different priorities and preferences. By allowing for preference modelling, we can gain insights into the level of disagreement on policy proposals as well as how a policy could be efficiently refined for the mutual benefits of two or more stakeholder groups or arriving at more than one competing policy options for further decision analysis. The preference elicitation method introduced in this study, uses rank-based elicitation complemented with proportional scores or direct rating in order to provide flexibility for advanced users. Future work aims to design an integrated decision support tool that couples systems modelling and simulation of public policy problems with MCDA techniques and preference elicitation methods. The implementation of the proposed decision analysis process using an online GUI, enables policy makers and stakeholders to provide preferential statements over their respective goals in a simple way.