1 Introduction

Toward the end of the Cold War, in the early 1980s, the RAND Corporation invested much effort toward using computer combat simulation models to inform national security decisions regarding weapons procurement and strategy. Designed to predict the course of future military conflicts, these models were obviously imperfect. Everyone recognized the aphorisms about the military’s propensity to plan for the last as opposed to the next war. But growing computational capabilities offered new opportunities to simulate complex military campaigns with unprecedented levels of fidelity and realism. The efforts to harness these capabilities provided a focal point for a growing debate on the value of using newly powerful but clearly imperfect analytic tools to inform real and consequential decisions.

The 1982 Falklands/Malvinas war provided a crystalizing moment. Argentine forces landed on the islands in early April and a British task force set sail a few days later for the two-month voyage to the South Atlantic. At that point, all knew the order of battle for the coming conflict. At RAND, those who distrusted the combat simulations used the opportunity to confront the model advocates, challenging them to predict how the conflict would unfold. Two months would provide ample time to conduct the analysis. But the advocates demurred. The models, they claimed, despite all their fidelity and realism, existed to provide “insight” not predictions. The skeptics wondered about the utility of insights gleaned from models whose predictions could not be trusted.

Decisionmakers often seek predictions about the future to inform policy choices. This seems natural because prediction is the bedrock of science, enabling researchers to test their hypotheses and demonstrate understanding of complicated systems. Decisionmakers also find predictions attractive because, when good ones are available, they unquestionably provide valuable input toward better choices. Society’s vast enterprise in the physical, biological, and social sciences continually improves its predictive capabilities in order to test and refine scientific hypotheses. Recent analytic innovations, such as big data and machine learning, enhance decision-relevant predictive capabilities. New processes, such as crowd sourcing, prediction markets, and super-forecasting (Tetlock and Gardner 2016), provide new and more reliable means to aggregate human judgments into probabilistic forecasts.

But, as argued throughout this book, the quest for predictions—and a reliance upon analytic methods that require them—can prove counter-productive and sometimes dangerous in a fast-changing, complex world. Prediction-focused analysis risks over-confidence in organizations’ decisionmaking and in their internal and external communications (Sarewitz and Pielke 2000). Prediction-focused policy debates can also fall victim to the strategic uses of uncertainty. Opponents may attack a proposed policy by casting doubt on the predictions used to justify it, rather than engaging with the merits of the policy itself, knowing that the policy may be more sound than the predictions (Herrick and Sarewitz 2000; Rayner 2000; Lempert and Popper 2005; Weaver et al. 2013).

A reliance on prediction can also skew the framing of a decision challenge. President Eisenhower (reportedly) advised “if a problem cannot be solved, enlarge it.” But science often reduces uncertainty by narrowing its focus, prioritizing questions that can be resolved by prediction, not necessarily on the most decision-relevant inquiries. So-called wicked problems (Rittel and Webber 1973) present this contrast most starkly. In addition to their irreducible uncertainty and nonlinear dynamics, wicked problems are not well-bounded, are framed differently by various stakeholders, and are not well-understood until after formulation of a solution. Using predictions to adjudicate such problems skews attention toward the proverbial lamp post, not the true location of the keys to a policy solution.

This chapter describes Robust Decision Making (RDM), a set of concepts, processes, and enabling tools designed to re-imagine the role of quantitative models and data in informing decisions when prediction is perilous (Lempert et al. 2003, 2006). Rather than regarding models as tools for prediction and the subsequent prescriptive ranking of decision options, models and data become vehicles for systematically exploring the consequences of assumptions; expanding the range of futures considered; crafting promising new responses to dangers and opportunities; and sifting through a multiplicity of scenarios, options, objectives, and problem framings to identify the most important tradeoffs confronting decisionmakers. That is, rather than making better predictions, quantitative models and data can be used to inform better decisions (Popper et al. 2005).

RDM rests on a simple concept (Lempert et al. 2013a). Rather than using computer models and data as predictive tools, the approach runs models myriad times to stress test proposed decisions against a wide range of plausible futures. Analysts then use visualization and statistical analysis of the resulting large database of model runs to help decisionmakers identify the key features that distinguish those futures in which their plans meet and miss their goals. This information helps decisionmakers identify, frame, evaluate, modify, and choose robust strategies—ones that meet multiple objectives over many scenarios.

RDM provides decision support under conditions of deep uncertainty. As described in Sect. 2.2, RDM builds on strong foundations of relevant theory and practice to provide an operational and newly capable synthesis through the use of today’s burgeoning information technology. The most commonly used analytic methods for predictive decision and risk analysis have their roots in the 1950s and 1960s, when relative computational poverty made a virtue of analytics recommending a single best answer based on a single best-estimate prediction. Today’s ubiquitous and inexpensive computation enables analytics better suited to more complex problems, many of them “wicked” and thus poorly served by the approximation that there exists such an optimal solution.

The British won the Falklands/Malvinas war, but lost a vessel to an unexpectedly effective anti-ship missile attack and sank an Argentine ship under disputed conditions. From an RDM perspective, the model skeptics at RAND asked the wrong question. Rather than judge the combat simulations by how well they could generate an accurate probability density function, the question should have been—can these quantitative models, and the processes for using them, help the British make better decisions regarding what forces were appropriate to bring and how they ought to be employed. While useful, even models that produced a perfect probability density function for precisely selected outcomes would not prove sufficient to answer such questions. Nor are they necessary.Footnote 1

2 RDM Foundations

How can quantitative, evidence-based analysis best inform our choices in today’s fast-paced and turbulent times? RDM and the other DMDU methods in this book aim to answer this question. In particular, RDM does so by providing a new synthesis of four key concepts: Decision Analysis, Assumption-Based Planning, scenarios, and Exploratory Modeling.

  • Decision Analysis

A large body of empirical research makes clear that people, acting as individuals or in groups, often make better decisions when using well-structured decision aids. The discipline of Decision Analysis (DA) comprises the theory, methodology, and practice that inform the design and use of such aids. RDM represents one type of quantitative DA method, drawing, for instance, on the field’s decision structuring frameworks, a consequentialist orientation in which alternative actions are evaluated in each of several alternative future states of the world, a focus on identifying tradeoffs among alternative decision options, and tools for comparing decision outcomes addressing multiple objectives.

As one key contribution, DA and related fields help answer the crucial question: What constitutes a good decision? No universal criterion exists. Seemingly reasonable decisions can turn out badly, but seemingly unreasonable decisions can turn out well. Good decisions tend to emerge from a process in which people are explicit about their goals, use the best available evidence to understand the potential consequences of their actions, carefully consider the tradeoffs, contemplate the decision from a wide range of views and vantages, and follow agreed-upon rules and norms that enhance the legitimacy of the process for all those concernedFootnote 2 (Jones et al. 2014).

While broad in principle, in practice the DA community often seeks to inform good decisions using an expected utility framework for characterizing uncertainty and comparing decision options (Morgan and Henrion 1990). This expected utility framework characterizes uncertainty with a single joint probability distribution over future states of the world. Such distributions often reflect Bayesian (i.e., subjective) rather than frequentist probability judgments. The framework then uses optimality criteria to rank alternative options. RDM, in contrast, regards uncertainties as deep and thus either eschews probabilities or uses sets of alternative distributions drawing on the concepts of imprecise probabilities (Walley 1991). RDM uses decision criteria based on robustness rather than optimality.

DA based on expected utility can be usefully termed “agree-on-assumptions” (Kalra et al. 2014) or “predict-then-act” (Lempert et al. 2004) approaches, because they begin by seeking agreement regarding the likelihood of future states of the world and then use this agreement to provide a prescriptive ranking of policy alternatives. In contrast, RDM and many of the approaches described in this book follow an “agree-on-decisions” approach, which inverts these steps.Footnote 3 They begin with one or more strategies under consideration, use models and data to stress test the strategies over a wide range of plausible paths into the future, and then use the information in the resulting database of runs to characterize vulnerabilities of the proposed strategies and to identify and evaluate potential response to those vulnerabilities. Such approaches seek to expand the range of futures and alternatives considered and, rather than provide a prescriptive ranking of options, often seek to illuminate tradeoffs among not-unreasonable choices. As summarized by Helgeson (2018), agree-on-assumptions approaches generally focus on identifying a normative best choice among a fixed menu of decision alternatives, while agree-on-decision approaches focus on supporting the search for an appropriate framing of complex decisions.

  • Assumption-Based Planning

As part of an “agree-on-decisions” approach, RDM draws on the related concepts of stress testing and red teaming. The former, which derives from engineering and finance (Borio et al. 2014), subjects a system to deliberately intense testing to determine its breaking points. The latter, often associated with best practice in US and other militaries’ planning (Zenko 2015), involves forming an independent group to identify means to defeat an organization’s plans. Both stress testing and red teaming aim to reduce the deleterious effects of over-confidence in existing systems and plans by improving understanding of how and why they may fail (Lempert 2007).

In particular, RDM draws on a specific form of this concept, a methodology called Assumption-Based Planning (ABP) (Dewar et al. 1993, 2002). Originally developed to help the US Army adjust its plans in the aftermath of the Cold War, ABP begins with a written version of an organization’s plans and then identifies load-bearing assumptions—that is, the explicit and implicit assumptions made while developing that plan that, if wrong, would cause the plan to fail. Planners can then judge which of these load-bearing assumptions are also vulnerable—that is, could potentially fail during the lifetime of the plan.

ABP links the identification of vulnerable, load-bearing assumptions to a simple framework for adaptive planning that is often used in RDM analyses. Essential components of an adaptive strategy include a planned sequence of actions, the potential to gain new information that might signal a need to change this planned sequence, and actions to be taken in response to this new information, i.e., contingent actions (Walker et al. 2001). After identifying the vulnerable, load-bearing assumptions, ABP considers shaping actions (those designed to make the assumptions less likely to fail), hedging actions (those that can be taken if assumptions begin to fail), and signposts (trends and events to monitor in order to detect whether any assumptions are failing).

  • Scenarios

RDM draws from scenario analysis the concept of a multiplicity of plausible futures as a means to characterize and communicate deep uncertainty (Lempert et al. 2003). Scenarios represent internally consistent descriptions of future events that often come in sets of two or more. Most simply, scenarios are projected futures that claim less confidence than probabilistic forecasts. More generally, a set of scenarios often seeks to represent different ways of looking at the world without an explicit ranking of relative likelihood (Wack 1985).

Scenarios are often developed and used in deliberative processes with stakeholders. Deemphasizing probabilistic ranking—focusing on a sense of possibility, rather than probability—helps stakeholders expand the range of futures they consider, allowing them to contemplate their choices from a wider range of views and vantages, thus helping participants consider uncomfortable or unexpected futures (Schoemaker 1993; Gong et al. 2017). The sense of possibility rather than probability can also help scenarios to communicate a wide range of futures to audiences not necessarily eager to have their vantage expanded. By representing different visions of the future without privileging among them, scenarios can offer a comfortable entry into an analysis. Each person can find an initially resonant scenario before contemplating ones that they find more dissonant.

RDM draws from scenario analysis the concept of organizing information about the future into a small number of distinct cases that help people engage with, explore, and communicate deep uncertainty. In particular, the Intuitive Logics school of scenario analysis (Schwartz 1996) uses qualitative methods to craft a small number of scenarios, distinguished by a small number of key uncertain determinants that differentiate alternative decision-relevant paths into the future. As described below, RDM uses quantitative “Scenario Discovery” algorithms to pursue the same ends. The resulting scenarios summarize the results of the ABP-style stress tests and can link to the development of adaptive strategies (Groves and Lempert 2007; Groves et al. 2014). Note that while the scenario literature traditionally distinguishes between probabilistic and non-probabilistic treatments, RDM often employs a third alternative—entertaining multiple views about the likelihood of the scenarios.

The scenario literature also describes a process for seeking robust strategies that include choosing a set of scenarios that include the most important uncertainties facing the users and then identifying strategies that perform well across all of them (van der Heijden 1996). This process provides an animating idea for RDM.

  • Exploratory Modeling

RDM integrates these concepts—DA, ABP, and scenarios—through Exploratory Modeling (EM). Bankes (1993) encapsulated the 1980s RAND debates on useful and predictive models by dividing computer simulations into two types: (1) consolidative models, which gather all known facts together into a single package that, once validated, can serve as a surrogate for the real world, and (2) exploratory models, which map a wide range of assumptions onto their consequences without privileging one set of assumptions over another. Exploratory models are useful when no single model can be validated because of missing data, inadequate or competing theories, or an irreducibly uncertain future.

Running a model many times is not profound. But as perhaps its key insight, EM notes that when used with an appropriate experimental design—that is, appropriate questions and a well-chosen set of cases designed to address those questions—the large database of results generated from non-predictive, exploratory models can prove surprisingly useful toward informing policy choices. Bankes describes several types of questions one may address with EM, including hypothesis generation, reasoning from special cases, and assessing properties of the entire ensemble (Weaver et al. 2013 and Box 2.1). RDM uses them all, but focuses in particular on robustness as a property of the entire ensemble. That is, identifying and evaluating robust strategies become key questions one can inform with EM.

EM provides RDM with a quantitative framework for stress testing and scenario analysis. While consolidative models most usefully support deductive reasoning, exploratory models serve best to support inductive reasoning—an iterative cycle of question and response. As described in the process section below (Sect. 2.3), RDM also aims to support a decision-analytic human-machine collaboration that draws upon what each partner does best.

RDM also exploits another EM advantage: the focus on the simple computational task of running models numerous times in the forward direction. This facilitates exploration of futures and strategies by reducing the requirements for analytic tractability on the models used in the analysis, relative to approaches that rely on optimization or Dynamic Programming. In addition, EM enables truly global sensitivity explorations, since it privileges no base case or single future as an anchor point.

Box 2.1: Key Elements of RDM

RDM meets its goals by proceeding in multiple iterations, as humans and computers alternatively test each other’s conclusions about futures and strategies. Four key elements govern these interactions (Lempert et al. 2003):

  • Consider a multiplicity of plausible futures. The ensemble of futures should be as diverse as possible to adequately stress test proposed policies. The ensemble can also facilitate group processes by including futures that correspond to different groups’ worldviews.

  • Seek robust, rather than optimal strategies. Robust strategies perform well, compared to the alternatives, over a wide range of plausible futures.

  • Employ adaptive strategies to achieve robustness. Adaptive strategies are designed to evolve over time in response to new information. Generally, such strategies reflect decisionmaking rules and in practice are often organized around near-term actions, signposts to monitor, and contingency actions to take in response to those signposts.

  • Use the computer to facilitate human deliberation over explorations, options, and tradeoffs, not as a device for recommending a particular ordering of strategies.

3 RDM Process

RDM explicitly follows a learning process called “deliberation with analysis” in which parties to a decision deliberate on their objectives and options; analysts generate decision-relevant information using system models; and the parties to the decision revisit their objectives, options, and problem framing influenced by this quantitative information (NRC 2009). Among learning processes, deliberation with analysis proves most appropriate for situations with diverse decisionmakers who face a changing decision environment and whose goals can evolve as they collaborate with others. Deliberation with analysis also supports continuous learning based on indicators and monitoring (NRC 2009, p. 74), a process important to the literature on adaptive policymaking (Swanson and Bhadwal 2009; Walker et al. 2010).

Step 1 As shown in Fig. 2.1, the RDM process starts with a decision framing exercise in which stakeholders define the key factors in the analysis: the decisionmakers’ objectives and criteria; the alternative actions they can take to pursue those objectives; the uncertainties that may affect the connection between actions and consequences; and the relationships, often instantiated in computer simulation models, between actions, uncertainties, and objectives. This information is often organized in a 2 × 2 matrix called “XLRM” (Lempert et al. 2003), for exogenous uncertainties (X), policy levers (L), relationships (R), and measures of performance (M).Footnote 4

Fig. 2.1
figure 1

Source Lempert et al. (2013a), with earlier versions in Lempert and Groves (2010); Lempert and Kalra (2011)

Steps in an RDM analysis.

Full size image

Step 2 As an “agree-on-decision” approach, RDM next uses simulation models to evaluate proposed strategies in each of many plausible paths into the future, which generate a large database of simulation model results. The proposed strategies can derive from a variety of sources. In some cases, an RDM analysis might start with one or more specific strategies drawn from the relevant public debate. For instance, an RDM analysis for a water agency might begin with that agency’s proposed plan for meeting its supply requirements (Groves et al. 2014) or water quality requirements (Fischbach et al. 2015). In other cases, optimization routines for one or more expected futures or decision criteria might yield the initial proposed strategies (Hall et al. 2012). Additionally, an analysis might begin with a wide span of simple strategies covering the logical spectrum and then refine, select, and modify to yield a small group of more sophisticated alternatives (Popper et al. 2009). Often, such as in the example in Sect. 2.5, an application uses a combination of these approaches.

Step 3 Analysts and decisionmakers next use visualization and data analytics on these databases to explore for and characterize vulnerabilities. Commonly, RDM analyses use statistical Scenario Discovery (SD) algorithms (see below) to identify and display for users the key factors that best distinguish futures in which proposed strategies meet or miss their goals. These clusters of futures are usefully considered policy-relevant scenarios that illuminate the vulnerabilities of the proposed policies. Because these scenarios are clearly, reproducibly, and unambiguously linked to a policy stress test, they can avoid the problems of bias and arbitrariness that sometimes afflict more qualitative scenario exercises (Lempert 2013; Parker et al. 2015).

Step 4 Analysts and decisionmakers may use these scenarios to display and evaluate the tradeoffs among strategies. For instance, one can plot the performance of one or more strategies as a function of the likelihood of the policy-relevant scenarios (e.g., see Fig. 2.2) to suggest the judgments about the future implied by choosing one strategy over another. Other analyses plot multi-objective tradeoff curves—for instance comparing reliability and cost (Groves et al. 2012)—for each of the policy-relevant scenarios to help decisionmakers decide how to best balance among their competing objectives.

Fig. 2.2
figure 2

Source for right panel, Lempert et al. (2003)

Two views of RDM as a deliberative process.

Full size image

Step 5 Analysts and decisionmakers could then use the scenarios and tradeoff analyses to identify and evaluate potentially more robust strategies—ones that provide better tradeoffs than the existing alternatives. These new alternatives generally incorporate additional policy levers, often the components of adaptive decision strategies: short-term actions, signposts, and contingent actions to be taken if the pre-designated signpost signals are observed. In some analyses, such adaptive strategies are crafted using expert judgment (e.g., see Lempert et al. 1996, 2000, 2003; Popper et al. 2009; Lempert and Groves 2010). In other analyses, optimization algorithms may help suggest the best combination of near-term actions, contingent actions, and signposts for the new adaptive strategies (Lempert et al. 2006; Lempert and Collins 2007; Kasprzyk et al. 2013; Herman et al. 2014). (This RDM step provides a natural point of connection with the Dynamic Adaptive Policy Pathways (DAPP) methods in Chaps. 4 and 9.)

RDM uses both absolute and relative performance measures to compare strategies in the vulnerability and tradeoff analyses. Absolute performance measures are useful when decisionmakers are focused on one or more outcomes, such as profit, energy produced, or lives saved. Absolute performance measures are also useful when decisionmakers are focused on some invariant standard—for instance, a regulatory requirement on reliability or environmental quality, a required threshold for an economic rate of return, or a requirement that benefits exceed costs. Relative performance measures are often useful when uncertainties create a wide range of outcomes, so decisionmakers seek strategies that perform well compared to alternatives over a wide range of futures. RDM often uses regret to represent relative performance.

At each of the RDM steps, information produced may suggest a re-framing of the decision challenge. The process produces key deliverables including (1) the scenarios that illuminate the vulnerabilities of the strategies, and (2) potential robust strategies and the tradeoffs among them.

The left panel of Fig. 2.2 shows the RDM steps in support of a process of deliberation with analysis. Stakeholders begin by deliberating over the initial decision framing. In the vulnerability and tradeoff analysis steps, stakeholders and analysts produce decision-relevant information products. Using these products, stakeholders deliberate over the choice of a robust strategy or return to problem framing, for instance seeking new alternatives or stress testing a proposed strategy over a wider range of futures. In practice, the process often moves back and forth between problem framing, generating the scenarios that illuminate vulnerabilities, identifying new alternatives based on those scenarios, and conducting a tradeoff analysis among the alternatives.

People teaming with computers—each doing what they do best—are more capable than computers or people alone.Footnote 5 RDM uses EM to support deliberation with analysis in a process of human/machine collaboration (Lempert et al. 2003). As shown in the right panel of Fig. 2.2, people use their creativity and understanding to pose questions or suggest solutions—for instance candidate robust strategies. Computers consider numerous combinations of strategies and futures to help users address their questions, search for initially unwelcome counter-examples to proposed solutions, and help people find new candidate robust strategies to propose (Lempert et al. 2002; Lempert and Popper 2005).

The RDM steps and deliberative processes are consistent with others in the DMDU literature. For instance, Multi-Objective RDM (MORDM) offers a similar iterative process but with the major advance of more articulation of the step of generating alternative strategies (Kasprzyk et al. 2013). The DAPP process (see Chaps. 4 and 9), also similarly, emphasizes monitoring and policy adjustment (Haasnoot et al. 2013). Among related literatures, many-objective visual analytics uses interactive visualizations to support problem framing and re-framing (Kollat and Reed 2007; Woodruff and Reed 2013), often with a posteriori elicitation of preferences (Maass et al. 1962; Cohon and Marks 1975).

Overall, the RDM process aims to provide quantitative decision support that helps meet the criteria for good decisions even in the presence of deep uncertainty and the other attributes of wicked problems. The process encourages participants to be explicit about their goals and consider the most important tradeoffs. The process uses scenario concepts linked to the idea of policy stress tests, along with computer-assisted exploration, to encourage and facilitate consideration of the decision from a wide range of views. It helps recognize the legitimacy of different interests, values, and expectations about the future by using models as exploratory, rather than predictive, tools within an evidence-based, multi-scenario, multi-objective decision-making process.

Box 2.2: What is a Robust Strategy?

A robust strategy is one that performs well, compared to the alternatives, over a wide range of plausible futures (Rosenhead et al. 1972; Lempert et al. 2003). Other definitions exist, including trading some optimal performance for less sensitivity to assumptions (Lempert and Collins 2007) and keeping options open (Rosenhead 1990). Choosing a specific quantitative criterion to judge robustness can, however, prove complicated because many robustness criteria exist, and in some cases they can yield a different ordering of strategies (Giuliani and Castelletti 2016). No robustness criterion is best in all circumstances and, as befits a decision support methodology designed to facilitate problem framing, RDM often includes the choice of a robustness criterion as part of its problem-framing step.

RDM and other deliberative “agree-on-decision” DMDU methods generally draw robustness criteria from the normative, “agree-on-assumptions” decision-analytic literature. This literature identifies four traditional criteria—called Wald, Hurwicz, Savage, and Laplace—for ranking choices without well-defined probability distributions over future states of the world (Luce and Raiffa 1957; Schneller and Sphicas 1983). These criteria envision a set of future states of the world \( f_{j} \) and a set of strategies \( s_{i} \), each with a known utility, \( u_{ij} \), in each state of the world. If the probability of each future, \( p_{j} \), were known, the best strategy would be that which yielded the maximum expected utility, \( {\text{Max}}_{i} \mathop \sum \nolimits_{j} p_{j} u_{ij} \). Lacking such probabilities, Wald selects the strategy that gives the best worse case, \( {\text{Max}}_{i} {\text{Min}}_{j} \left( {u_{ij} } \right) \), and Savage’s mini-max regret selects the strategy with the least regret, that is, which deviates least from the best one could choose with perfect information, \( {\text{Min}}_{i} {\text{Max}}_{j} \left[ {{\text{Max}}_{i} \left( {u_{ij} } \right) - u_{ij} } \right] \). Both Wald and Savage are conservative in that they attempt to avoid worst cases. In contrast, Hurwicz interpolates between selecting the strategy with the best case and the best worst case, \( {\text{Max}}_{i} \left[ {\alpha {\text{Max}}_{j} \left( {u_{ij} } \right) + \left( {1 - \alpha } \right){\text{Min}}_{j} \left( {u_{ij} } \right)} \right] \), where \( \alpha \) represents a level of confidence. Laplace’s criterion of insufficient reason assumes equal weighting over all the futures and then selects the strategy that maximizes expected utility. Starr (1962) subsequently proposed the domain criterion, which selects the strategy that has highest utility in the most futures.

The domain criterion comes closest to the idea of performing well over a wide range of futures and can also facilitate tradeoff analyses using scenario-themed visualizations. RDM analyses commonly use some form of the domain criterion, although RDM (see Table 7.2 in Cervigni et al. 2015) and other robustness approaches (e.g., Hansen and Sargent 2008) have also employed Savage’s mini-max regret. Traditionally, the domain criterion uses an absolute performance measure (e.g., the utility \( u_{ij} , \) or a benefit–cost ratio (Lempert 2014)), as its argument. RDM often extends the criterion by also using as arguments relative performance measures, e.g., the regret \( {\text{Max}}_{i} \left( {u_{ij} } \right) - u_{ij} \), or satisficing criteria for both absolute and relative performance measures. For instance, employed with a regret measure and satisficing criterion, the domain criterion might select the strategy that has a regret below some threshold value over the widest range of futures (Lempert and Collins 2007). In contrast to a strict application of Savage’s mini-max regret, such a criterion would retain the comparative quality of the regret measure without undue attention to worst cases.

RDM also uses the domain criterion over the space of probabilities rather than the space of futures; that is, the analysis considers a set of plausible probability distributions over the futures (Walley 1991), and the domain criterion selects the strategy with the highest expected utility or expected regret over the largest number of distributions in the set (see, e.g., Fig. 2.3 in this chapter). RDM often uses the domain criterion, or other robustness decision criteria, to focus on tradeoffs rather than on a normative ranking of strategies. For instance, Lempert and Collins (2007) employ a definition of robustness as trading some optimal performance for less sensitivity to assumptions, which it implements with a domain criterion using expected regret as its argument. The study displays the choice of a robust strategy as a function of the confidence one places in the best-estimate distribution. The Combined Strategy discussed in Sect. 2.5.3 is robust by both a domain criterion and by mini-max regret.

4 Tools

The concepts underlying RDM—scenario thinking, robustness decision criteria, stress testing proposed plans, and the use of exploratory models—have long pedigrees. But over the last decade, new computer and analytic capabilities have made it possible to combine them in practical decision analyses. In particular, RDM often relies on Scenario Discovery and visualization, robust multi-objective optimization, integrated packages for EM, and high-performance computing.

SD algorithms often implement the RDM vulnerability analysis step (Step 3) in Fig. 2.1. SD begins with a large database of runs in which each model run represents the performance of a strategy in one future. The SD cluster-finding algorithms then offer concise descriptions of those combinations of future conditions that best distinguish the cases in which the implementation plan does or does not meet its goals. The requisite classification algorithms—often Patient Rule Induction Method (PRIM) (Friedman and Fisher 1999) or Classification and Regression Tree (CART) (Breiman et al. 1984), combined with a principal component analysis (Dalal et al. 2013)—seek to balance between the competing goals of simplicity and accuracy in order to describe sets of strategy-stressing futures as concise, understandable, and decision-relevant scenarios (Groves and Lempert 2007; Bryant and Lempert 2010). Software to implement the PRIM algorithm is available, both in stand-alone routinesFootnote 6 and embedded in the EM software packages described below. Overall, SD replicates analytically the ideas of qualitative Intuitive Logics scenario analysis and provides information products that can prove compelling in stakeholder deliberations (Lempert 2013). RDM analyses also use computer visualization of the database of runs to support the vulnerability and tradeoff analyses. Tableau, a commercially available platform, has proven particularly useful in much RDM work (Groves et al. 2013; Cervigni et al. 2015).

The “new futures and strategies” step (Step 5) can employ a variety of methods. While some RDM analyses use only expert judgment to craft responses to potential vulnerabilities (e.g., Lempert and Groves 2010), many applications use some multi-objective robust optimization tool. Some such applications have used constrained optimization to trace out a range of potentially robust solutions, both in single objective (Lempert and Collins 2007) and multi-objective (Groves et al. 2012) cases. The latter instance involved a planning tool that allowed analysts to trace the Pareto optimal tradeoff curves in each scenario for any two objectives using constrained optimizations over the other objectives (Groves et al. 2012), and has been widely used. Other applications, such as the Colorado Basin Supply and Demand Study (Groves et al. 2013; also see Chap. 7), run large portfolio optimizations for many futures; identify the individual actions that occur in the optimal set for most, some, and few of the futures; and use this information to craft adaptive strategies that begin with the actions that occur in most of the futures’ optimal sets, and implement the others depending on which future comes to pass (Groves et al. 2013; Bloom 2015). MORDM tools provide a more general solution, using evolutionary algorithms to identify the regions of a Pareto surface over many objectives that are most robust to uncertainty (Kasprzyk et al. 2013). MORDM has been used to identify adaptive strategies modeled as controllers in the control theory sense (Quinn et al. 2017), through a process called direct policy search.

While many RDM analyses use stand-alone software to generate and analyze large numbers of model runs, several integrated EM packages exist that can greatly facilitate such analysis. These packages generally consider a simulation model as a mapping of inputs to outputs and then provide connectivity between the model and various analysis and visualization tools. For instance, EM packages allow the user to specify an experimental design over the inputs; run the resulting cases on one or more processors; store the results in a database; use a variety of visualization routines to examine the data; run SD and other statistical algorithms on the data; and use robust optimization routines to identify new, potentially more robust strategies. While they require study and practice to use, EM software can make it much easier to move seamlessly through the steps of an RDM analysis. Computer-Assisted Reasoning system (CARs) was the earliest such software package (Bankes et al. 2001). More recently, new open-source platforms have become available, such as the Exploratory Modeling and Analysis Workbench,Footnote 7 Open MORDM (Hadka et al. 2015), and Rhodium.Footnote 8

While many RDM analyses may be conveniently run on a laptop or desktop computer (e.g., Lempert et al. 2013b), recent studies have also used high-performance computation—either large-scale cluster computing (Zeff et al. 2014; Groves et al. 2016) or cloud-based computer services (Isley 2014; Cervigni et al. 2015)—to quickly and inexpensively conduct a very large number of runs.

5 Example: Carrots and Sticks for New Technology

An early RDM application provides an ideal example of the approach, the types of tools used, and how EM can draw together concepts from decision analysis, scenarios, and ABP. The example, called “carrots and sticks for new technology,” focused on determining the most robust combination of two policy instruments—carbon prices and technology subsidies—to reduce climate-altering greenhouse gas emissions (Robalino and Lempert 2000; Lempert 2002). We describe this study using the generic DMDU framework presented in Sect. 1.5 (the major subheadings below) linked to the relevant RDM steps shown in Fig. 2.1.

5.1 Frame the Analysis

  • Formulate Question (RDM Step 1)

It is well-understood that an economically ideal greenhouse gas emission reduction policy should include an economy-wide carbon price implemented through mechanisms such as a carbon tax or a cap-and-trade system. This early RDM study addressed the question of whether and under what conditions technology incentives, such as technology-specific tax credits or subsidies, also prove necessary and important as part of a greenhouse gas reduction strategy. Many national and regional jurisdictions worldwide employ such incentives in their climate policies because they prove politically popular and have a compelling logic, if for no other reason than significant technology innovation will prove crucial to limiting climate change. But technology incentives have a mixed record of success (Cohen and Noll 2002), and sometimes link to larger debates about the appropriate role of government (Wolf 1993). Standard economic analysis proves a poor platform to adjudicate such questions, because the extent to which technology incentives prove economically important may depend on coordination failures that occur in the presence of increasing returns to scale, imperfect information, and heterogeneous preferences—deeply uncertain factors not well-represented in standard economic models. Addressing the technology incentive question with RDM thus proved useful due to this deep uncertainty and because people’s views on technology incentives can be strongly affected by their worldviews.

  • Identify Alternatives (RDM Step 1)

The study was organized as a head-to-head comparison between strategies incorporating two types of “policy instruments” or “actions”: an economy-wide carbon tax and a technology price subsidy for low-carbon-emitting technologies. We considered four combinations: (1) neither taxes nor subsidies (No Action), (2) tax only (Taxes Only), (3) subsidies only (Subsidies Only), and (4) a combination of both taxes and subsidies (Combined). As described below, both the tax and subsidy were configured as adaptive strategies designed to evolve over time in response to new information.

  • Specify Objectives (RDM Step 1)

This study compared the strategies using two output measures: the present value of global economic output (pvGDP) and the mid-twenty-first-century level of greenhouse gas emissions. We focused on pvGDP to facilitate comparison of this work with other analyses in the climate change policy literature using more standard economic formulations. As described below, we calculated the regret for each strategy in each future and used a domain criterion for robustness (see Box 2.2), thus looking for strategies with low regret over a wide range of plausible futures.

  • Specify System Structure (RDM Step 1)

To compare these adaptive strategies, we employed an agent-based model of technology diffusion, linked to a simple macromodel of economic growth that focused on the social and economic factors that influence how economic actors choose to adopt, or not to adopt, new emission-reducing technologies. The agent-based representation proved useful, because it conveniently represents key factors potentially important to technology diffusion, such as the heterogeneity of technology preferences among economic actors and the flows of imperfect information that influence their decisions. Considering this tool as an exploratory, rather than predictive, model proved useful, because it allowed the study to make concrete and specific comparisons of price- and subsidy-based strategies even though, as described below, available theory and data allowed the model’s key outputs to vary by over an order of magnitude.

The model was rooted in the microeconomic understanding of the process of technology diffusion (Davies 1979). Each agent in our model represents a producer of a composite good, aggregated as total GDP, using energy as one key input. Each period the agents may switch their choice of energy-generation technology, choosing among high-, medium-, or low-emitting options. Agents choose technology to maximize their economic utility. The agents estimate utility based on their expectations regarding each technology’s cost and performance. Costs may or may not decline significantly due to increasing returns to scale as more agents choose to adopt. The agents have imperfect information and current and future technology cost and performance, but can gain information based on their own experience, if any, and by querying other agents who have used the technology. Thus, the model’s technology diffusion rates depend reflexively on themselves, since by adopting a technology each agent generates new information that may influence the adoption decisions of other potential users. The model also used simple, standard, but deeply uncertain relationships from the literature on the connections between greenhouse gas emissions and the economic impacts due to climate change (Nordhaus 1994).

5.2 Perform Exploratory Uncertainty Analysis

  • Specify Uncertainties or Disagreements (RDM Step 2)

The agent-based model had thirty input parameters representing the deeply uncertain factors, including the macroeconomic effects of potentially distortionary taxes and subsidies on economic growth, the microeconomic preferences that agents use to make technology adoption decisions, the future cost and performance of high- (e.g., coal), low- (e.g., gas), and non-emitting (e.g., solar) technologies, the way information about new technologies flows through agent networks, and the impacts of climate change.

We employed three sources of information to constrain our EM. First, the agent-based model embodied the theoretical economic understanding of technology diffusion. Second, we drew plausible ranges for each individual parameter using estimates from the microeconomics literature. Third, we required the model to reproduce macroeconomic data regarding the last fifty years of economic growth and market shares for different types of energy technology. We also constrained future technology diffusion rates in the model to be no faster than the fastest such rates observed in the past.

Consistent with these constraints, the model nonetheless was able to generate a vast range of plausible futures. To choose a representative sample of futures from this vast set, the study launched a genetic algorithm over the model inputs, searching for the most diverse set of model inputs consistent with the theoretical, macroeconomic, and microeconomic constraints (Miller 1998). This process yielded an ensemble of 1,611 plausible futures, with each future characterized by a specific set of values for each of the thirty uncertain model input parameters. Each member of the ensemble reproduced the observed history from 1950 to 2000, but differed by up to an order of magnitude in projected mid-twenty-first-century emissions.

  • Evaluate Strategies Against Futures (RDM Step 2)

The study represented its carbon tax and technology subsidies as adaptive strategies using a single set of parameters for each to describe their initial conditions, and how they would evolve over time in response to new information.

As with many carbon price proposals, the study’s carbon tax would start with an initial price per ton of CO2 at time zero and rise at a fixed annual rate, subject to two conditions meant to reflect political constraints. First, we assumed that the government would not let the tax rise faster than the increase in the observed social cost of carbon. Second, we assumed that if the economy dropped into recession (defined as a global growth rate below some threshold), the government would drop the carbon tax back to its original level. Three parameters defined the tax strategy: the initial tax rate, the annual rate of increase, and the minimum threshold for economic growth required not to repeal the tax.

The study’s technology subsidy reduces the cost to users of a technology to a fixed percentage of its unsubsidized cost. The subsidy stays in effect until the policy either succeeds in launching the technology (defined as its market share rising above some threshold value) or the policy fails (defined as the market share failing to reach a minimum threshold after a certain number of years). Meeting either of these conditions permanently terminates the subsidy. Four parameters defined the subsidy strategy: the subsidy level, the market share defining success, the market share defining failure, and the number of years before the subsidy can be judged a failure.

The study chose a single set of parameter values to define the tax and subsidy policies, each set chosen to optimize the pvGDP for the future represented by the average value for each of the thirty model parameters. The results of the study were relatively insensitive to this simplification.

We used the agent-based model to calculate the pvGDP and mid-century greenhouse gas emissions for each strategy in each of the 1,611 plausible futures. This ensemble of runs made it immediately clear that the Taxes Only and Combined strategies consistently perform better than the No Action and Subsidies Only strategies. The remainder of the analysis thus focused on the first two.

5.3 Choose Initial Actions and Contingent Actions

  • Illustrate Tradeoffs (RDM Steps 3 and 4 )

Lacking (not-yet-developed) SD algorithms and faced with too many dimensions of uncertainty for an exhaustive search, the study used importance sampling to find the five uncertain input parameters most strongly correlated with mid-century GHG emissions. Four of these key uncertainties related to the potential for coordination failures to slow technology diffusion—the rate of cost reductions for non-emitting technologies caused by increasing returns to scale, the rate at which agents learn from one another about the performance of new technologies, the agents’ risk aversion, and the agents’ price-performance preferences for new technologies—and one related to the damages from climate change.

We then examined the regret in pvGDP across the cases for the Taxes Only and Combined strategies as a function of all ten two-dimensional combinations of the five key uncertainties.Footnote 9 Each visualization told a similar story—that the Combined strategy cases had lower mean and variability in pvGDP regret than those for Taxes Only except in the corner of the uncertainty space with low potential for coordination failures and/or low impacts from climate change.

This analysis provides the study’s basic comparison among the strategies: Taxes Only performs best when the potential for coordination failures and the impacts from climate change are small, and Combined performs best otherwise. Both strategies perform better than No Action or Subsidies Only.

  • Select and Plan for Adaptation (RDM Step 4)

To help decisionmakers understand the conditions under which the Taxes Only and Combined strategies would each be favored over the other, we first collapsed the ten visualizations into a single two-dimensional graph by combining all four key uncertain parameters relating to potential coordination failures into a single variable. Figure 2.3 shows the sets of expectations in which pvGDP for the Taxes Only strategy exceeds that of Combined, and vice versa, as a function of the probability assigned to high rather than low values of the four uncertain parameters related to potential coordination failures, which we labeled “probability of a non-classical world,” and “probability of high damages” due to climate change. The region dominated by Combined is larger than that of Taxes Only for two reasons. First, Combined dominates Taxes Only over larger regions of the state space. Second, in the regions of the uncertainty space where Taxes Only is the better strategy, Combined’s regret is relatively small, while in the regions where Combined is better, Taxes Only’s regret is relatively large. Existing scientific understanding proves insufficient to define with certainty where the future lies in Fig. 2.3. Different parties to the decision may have different views. But the boundary between the two regions is consistent with increasing returns to scale much smaller than those observed for some energy technologies—such as natural gas turbines, wind, and solar—and the middle of the figure is consistent with relatively small levels of risk aversion, learning rates, and heterogeneity of preferences compared to those seen in various literatures. These results suggest that a combination of price instruments and technology subsidies may prove the most robust strategy over a wide range of plausible futures.

Fig. 2.3
figure 3

Scenario map comparing “Taxes Only” and “Combined” strategies

Full size image

  • Implementation, Monitoring, and Communication

This study addressed a high-level question of policy architecture—the best mix of policy instruments for decarbonization. While the study did not provide detailed implementation plans, it does suggest how a national or state/provincial government might pursue the study’s recommendations. The study envisions policymakers choosing a strategy which includes the rules by which the initial actions will be adapted over time (Swanson et al. 2007). The carbon price, presumably set by the legislature, would follow the social cost of carbon as periodically updated by executive agencies (National Academies of Sciences 2016) whenever the economy was not in recession. The legislature would also set the technology subsidy and terminate it when the subsidized technologies either succeeded or failed based on market share data gathered by executive agencies. The study did not examine pre-commitment issues—that is, how the current legislature could ensure that future legislatures would in fact follow the adaptive strategy. These interesting issues of political economy have, however, been recently explored using RDM methods (Isley et al. 2015).

5.4 Iterate and Re-Examine (RDM Steps 2, 3, and 5)

The study’s results are based on an examination of only six of the 30 dimensions of uncertainty in the model, representing a small subset of the full range of plausible futures. As a key final step, we tested the policy recommendations by launching a genetic search algorithm across the previously unexamined dimensions looking for additional futures that would provide counter-examples to our conclusions. This process represents the computer feedback loop in the right panel of Fig. 2.2. The genetic algorithm ran for most of the time the authors spent writing their manuscript and found no plausible counter-examples.

Overall, this study suggests that if decisionmakers hold even modest expectations that market imperfections are likely to inhibit the diffusion of new, emissions-reducing technologies or that the impacts of climate change will turn out to be serious, then strategies combining both carbon taxes and technology incentives may be a promising component of a robust strategy for reducing greenhouse gas emissions.

6 Recent Advances and Future Challenges

The “carrots and sticks for new technology” example of Sect. 2.5 includes all the steps of an RDM analysis shown in Fig. 2.1. It used optimization algorithms to define its alternative, adaptive strategies, and generated its futures using genetic algorithms to perform what has more recently been called scenario diversity analysis (Carlsen et al. 2016a, b). The study employed the process of human-machine collaboration shown in Fig. 2.2, in particular in the computer search for counter-examples to the human-derived patterns that constitute its policy conclusions.

Since this early example, the methods and tools for MORDM analyses have approached maturity, now reaching the point at which one can describe with some specificity how to conduct multi-scenario, MORDM for many wicked problems. For instance, a recent study, described in detail in Chap. 7, used RDM on a topic similar to the “carrots and sticks” example—examining how international finance institutions such as the Green Climate Fund (GCF) can best craft long-term investment strategies to speed decarbonization in the face of deep technological and climate uncertainty (Molina Perez 2016). This more recent study was made possible by powerful new SD algorithms and visualization tools.

Recent work for four North Carolina cities illustrates the power of MORDM, a combination of RDM with new evolutionary algorithms for multi-objective robust optimization (Herman et al. 2014, 2016; Zeff et al. 2014, 2016). The study helped the neighboring cities of Raleigh, Durham, Chapel Hill, and Cary link their short-term operational and long-term investment water plans by shifting the former from rule-based procedures to new dynamic risk-of-failure triggers, and the latter from static to adaptive policy pathways. The study also helped the four independent cities coordinate their plans in the presence of different objectives and deep uncertainty. Linking such MORDM with the Dynamic Adaptive Policy Pathways (DAPP) approach of Chaps. 4 and 9 represents an exciting direction for the DMDU community.

Recent work for the US Bureau of Reclamation and the parties to the Colorado Compact, also described in Chap. 7, showcases RDM’s ability to facilitate deliberation with analysis, helping contesting parties to agree on the vulnerabilities they face and adaptive strategies for addressing them (Groves et al. 2013; Bloom 2015).

Some important technical hurdles still remain before these capabilities fully mature. First, an approach is needed that provides full Pareto satisficing surfaces. Current MORDM analyses identify Pareto optimal surfaces for best-estimate cases and measure the robustness of alternative strategies, represented by different regions on the Pareto surface, to the deep uncertainties (Kasprzyk et al. 2013). In the future, MORDM could produce sets of strategies chosen specifically because their performance across multiple objectives was largely insensitive to the deep uncertainties. Recent work has taken steps toward providing such Pareto satisficing surfaces (Watson and Kasprzyk 2017), but more needs to be done. Furthermore, despite the availability of ubiquitous computation on the cloud, and through high-performance computation facilities, it still remains difficult in many cases to conduct a full MORDM analysis using realistic system models, which would require running many thousands of cases to perform the multi-objective robust optimization over each of many thousands of scenarios. Research is needed on what we might call adaptive sampling approaches to help navigate more efficiently through the set of needed runs. In addition, research could usefully provide guidance on when to use alternative robustness criteria, as well as the conditions under which RDM’s iterative analytic process is guaranteed to converge independent of the initial problem framing, or when path dependence may lead analyses to different answers (Kwakkel et al. 2016).

Finally, the cost of developing the needed system models often puts RDM analyses out of reach for many decisionmakers. Research on “RDM-lite”—means to quickly develop such models through approaches such as expert elicitation and participatory modeling—could greatly increase the use of these methods (see, e.g., O’Mahony et. al. 2018).

Evaluation plays a crucial role in the design and use of any successful decision support system (NRC 2009; Pidgeon and Fischhoff 2011; Wong-Parodi et al. 2016). Some evaluations of RDM tools, visualizations, and processes exist, both in the laboratory (Budescu et al. 2013; Parker et al. 2015; Gong et al. 2017) and through field experiments (Groves et al. 2008). Recent work has proposed frameworks for evaluating the impacts of RDM-based decision support in urban environments (Knopman and Lempert 2016). But much more such evaluation work is required to improve the practical application of RDM decision support (Bartels et. al., forthcoming).

More broadly, as DMDU methods reach technical maturity, they offer the opportunity to reshape the relationship between quantitative decision analytics and the way in which organizations use this information with their internal and external audiences and processes. The potential for such reshaping presents a rich menu of research needs to understand the organizational, anthropological, political, and ethical implications.

As one example, the concept of risk governance embeds risk management, which often has a narrow, more technocratic perspective, in a broader context that considers institutions, rules conventions, processes, and mechanisms through which humans acting as individuals and groups make choices affecting risk (Renn 2008). Recent work has explored how to embed RDM methods and tools in a risk governance framework (Knopman and Lempert 2016). For instance, RDM can help decisionmakers working within a multi-agent and multi-jurisdictional system organize their strategies into “tiers of transformation,” which derives from the ideas of triple-loop learning. Lower tiers represent actions the decisionmakers can address on their own, while the outer tiers represent large-scale, transformative system changes that only the decisionmakers can help to catalyze. Any understanding of how to implement and use such capabilities, and the extent to which they would prove useful, remains nascent.

Future work can also usefully situate the types of moral reasoning and social choice embodied in alternative approaches to decision support. In his treatise, the Idea of Justice, Amartya Sen (2009) describes two classes of moral reasoning—the transcendental and the relational. The former, represented by Sen’s teacher John Rawls (1971), seeks to inform ethical societal choices by first envisioning a common vision of a perfectly just world. People can then use that vision to inform their near-term choices. The latter, Sen’s preferred alternative, rests on the assumption that irreducible uncertainty about the consequences of our actions, and a diversity of priorities, goals, and values, are fundamental attributes of our world. Thus, no such transcendental vision of the type envisioned by Rawls is possible because the level of agreement and commonality of values it presupposes does not, and should not, exist in a diverse society in which people are free to pursue their lives according to their own, often very different, visions of what is good. In addition, Sen argues, even if a common transcendental vision were possible, it would prove insufficient to inform near-term choices, because human knowledge is too fallible and the uncertainties too deep to chart an unambiguous path to the ideal. But humans can obtain sufficient knowledge to craft near-term options and differentiate the better from the worse. Relational reasoning thus involves an iterative process of debating, choosing, learning, and revisiting choices, always trying to move in the direction of more justice in the face of imperfect knowledge and conflicting goals.

“Agree-on-assumptions” approaches to decision support reflect transcendental reasoning, while “agree-on-decisions” approaches reflect relational reasoning (Lempert et al. 2013c). Sen emphasizes the importance of deliberation in a relational process of social choice. His framework provides attributes for judging what constitutes an ethical process of deliberation with analysis. In particular, such deliberations work best when they recognize the inescapable plurality of competing views; facilitate re-examination and iterative assessments; demand clear explication of reasoning and logic; and recognize an “open impartiality” that accepts the legitimacy and importance of the views of others, both inside and outside the community of interest to the immediate policy discussion.

Recent work has pioneered methods for conducting ethical-epistemological analysis on the extent to which decision support products, methods, and systems meet such ethical criteria (Tuana 2013; Bessette et al. 2017; Mayer et al. 2017), but much more remains to be done. Today’s world presents numerous, complex decision challenges—from sustainability to national security—that require quantitative decision support to successfully address. But “agree-on-assumptions” methods often lure decisionmakers toward over-confidence and can make it difficult to engage and promote consensus among participants with diverse expectations and interests. Such methods—built on the assumption that the decision analytics aim to provide a normative ranking of decision options—have their foundations in a time of computational poverty and rest on a narrow understanding of how quantitative information can best inform decisions. Recent years have seen an explosion of computational capabilities and a much richer understanding of effective decision support products and processes. RDM—a multi-objective, multi-scenario “agree-on-decision” approach—exploits these new capabilities and understanding to facilitate deliberative processes in which decisionmakers explore, frame, and reach consensus on the “wicked” problems that today’s decisionmakers increasingly face.