Taking the reins: how regulatory decision-makers can stop being hijacked by uncertainty

Article

Abstract

Several decades after the mechanics of quantitative uncertainty analysis (QUA) for risk assessment and regulatory cost analysis were developed and refined, QUA still rarely reaches the minds of decision-makers. The most common justification for this situation is that “decision-makers want a number, not a set of statistical distributions.” This may be an accurate assessment of their druthers, but one obvious though perhaps impractical retort is to say that if decision-makers insist on misleading point estimates, then we need new and better decision-makers. This article offers a way out of this dilemma. Decision-makers do not have to understand (or even receive) all the information contained in a complete QUA, but they do have to drive the QUA. They need to instruct analysts how to approach the phenomena they analyze (parameter uncertainty, model uncertainty, interindividual variability, offsetting and second-order effects, and the monetary value of future uncertainty reductions), they need to insist that uncertainties in cost be treated a priori as exactly as important as uncertainties in risk, and—even more importantly—they need to instruct analysts which estimator(s) to seek, report, and explain. Here we offer 10 detailed principles to guide decision-makers into a new relationship with risk and cost analysts—10 observations about how “eyes wide open” point estimates can vastly outperform point estimates handed to the decision-maker without context, justification, or honesty about the value judgments they impose upon the decision. A decision-maker who explains “I chose Option A because its benefits of 2.345 exceed its costs of 1.234” can be replaced by a dollar-store calculator. We need decision-makers who can say “I chose Option A because the spectrum of benefits it likely offers, to these citizens, considering the range of costs it likely imposes, makes it a superior choice to any other.” QUA, performed carefully and following clear policy instructions, can empower decision-makers to earn their influential roles.

Keywords

Risk assessment Uncertainty Interindividual variability Science-policy 

References

  1. Bogen KT (1995a) Methods to approximate joint uncertainty and variability in risk. Risk Anal 15(3):411–419CrossRefGoogle Scholar
  2. Bogen KT (1995b) A note on compounded conservatism. Risk Anal 14:379–381CrossRefGoogle Scholar
  3. Burgman MA, Keith DA, Walshe TV (1999) Uncertainty in comparative risk analysis for threatened Australian plant species. Risk Anal 19(4):585–598Google Scholar
  4. Cooke RM (ed) (2009) Uncertainty modeling in dose response: bench testing environmental toxicity. Wiley, HobokenGoogle Scholar
  5. Cox LAT (2012) Confronting deep uncertainties in risk analysis. Risk Anal 32:1607CrossRefGoogle Scholar
  6. Cullen AC, Frey HC (1999). Probabilistic techniques in exposure assessment: a handbook for dealing with variability and uncertainty in models and inputs. Plenum Press, New YorkGoogle Scholar
  7. Cummings CL, Kuzma J (2017) Societal risk evaluation scheme (SRES): scenario-based multi-criteria evaluation of synthetic biology applications. PLoS ONE 12(1):e0168564CrossRefGoogle Scholar
  8. Dudley SE, Gray GM (2012) Improving the use of science to inform environmental regulation. In: Johnston J (ed) Institutions and incentives in regulatory science. Lexington Books, New YorkGoogle Scholar
  9. Finkel AM (1990a) Confronting uncertainty in risk management: a guide for decision-makers. Resources for the Future, Center for Risk Management monograph, Washington, DCGoogle Scholar
  10. Finkel AM (1990b) A simple formula for calculating the “mass density” of a lognormally-distributed characteristic: applications to risk analysis. Risk Anal 10(2):291–301CrossRefGoogle Scholar
  11. Finkel AM (1991) Edifying presentation of risk estimates: not as easy as it seems. J Policy Anal Manag 10(2):296–303CrossRefGoogle Scholar
  12. Finkel AM (1995) Towards less misleading comparisons of uncertain risks: the example of aflatoxin and alar. Environ Health Perspect 103(4):376–385CrossRefGoogle Scholar
  13. Finkel AM (2014) The cost of nothing trumps the value of everything: the failure of regulatory economics to keep pace with improvements in quantitative risk analysis. Mich J Environ Adm Law 4(1):91–156Google Scholar
  14. Finkel AM (2018) Demystifying evidence-based policy analysis by revealing hidden value-laden constraints. In: Governance of emerging technologies: aligning policy analysis with the public’s values. Hastings Center Special Report 48(S1):S21–S49Google Scholar
  15. Finkel AM, Evans JS (1987) Evaluating the benefits of uncertainty reduction in environmental health risk management. J Air Pollut Control Assoc 37(10):1164–1171Google Scholar
  16. Ginsberg G, Toal BF (2009) Quantitative approach for incorporating methylmercury risks and omega-3 fatty acid benefits in developing species-specific fish consumption advice. Environ Health Perspect 117(2):267–275CrossRefGoogle Scholar
  17. Gray GM, Cohen JT (2012) Rethink chemical risk assessment. Nature 489:27–28CrossRefGoogle Scholar
  18. Gray GM, Graham JD (1991) Risk assessment and clean air policy. J Policy Anal Manag 10(2):286–295CrossRefGoogle Scholar
  19. Groth E III (2017) Scientific foundations of fish-consumption advice for pregnant women: epidemiological evidence, benefit-risk modeling, and an integrated approach. Environ Res 152:386–406CrossRefGoogle Scholar
  20. Hammitt JK, Shlyakhter AI (1999) The expected value of information and the probability of surprise. Risk Anal 19:135–152Google Scholar
  21. Hammond JS, Keeney RL, Raiffa H (1998) Smart choices: a practical guide to making better decisions. Harvard Business Review Press, CambridgeGoogle Scholar
  22. Harrington W, Morgenstern RD, Nelson P (2000) On the accuracy of regulatory cost estimates. J Policy Anal Manag 19:297CrossRefGoogle Scholar
  23. Hattis DB, Anderson EL (1999) What should be the implications of uncertainty, variability, and inherent “biases”/“conservatism” for risk management decision-making? Risk Anal 19(1):95–107Google Scholar
  24. Krupnick A et al (2006) Not a sure thing: making regulatory choices under uncertainty. Resources for the Future, Washington DCGoogle Scholar
  25. Kruser JM et al (2017) “Best case/worst case”: training surgeons to use a novel communication tool for high-risk acute surgical problems. J Pain Symptom Manag 53(4):711–719CrossRefGoogle Scholar
  26. Laxminarayan R, Macauley MK (eds) (2012) The value of information: methodological frontiers and new applications in environment and health. Springer, DordrechtGoogle Scholar
  27. Limpert E, Stahel W (undated) Log-normal distribution: its widespread use and some basic hints for its application. Compendium of weblinks, available at https://stat.ethz.ch/~stahel/lognormal/
  28. Linkov I, Trump BD, Wender BA, Seager TP, Kennedy AJ, Keisler JM (2017) Integrate life-cycle assessment and risk analysis results, not methods. Nat Nanotechnol 12(8):740–743CrossRefGoogle Scholar
  29. Malloy T, Trump BD, Linkov I (2016) Risk-based and prevention-based governance for emerging materials. Environ Sci Technol 50(13):6822–6824CrossRefGoogle Scholar
  30. Morgan MG, Henrion M (1992) Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, CambridgeGoogle Scholar
  31. National Academy of Sciences (1994) Science and judgment in risk assessment. National Academy Press, WashingtonGoogle Scholar
  32. National Academy of Sciences (2008) Science and decisions: advancing risk assessment. National Academy Press, WashingtonGoogle Scholar
  33. National Academy of Sciences (2011) Review of the Environmental Protection Agency’s draft IRIS assessment of formaldehyde. National Academy Press, WashingtonGoogle Scholar
  34. Nichols AL, Zeckhauser RJ (1986) The perils of prudence: how conservative risk assessments distort regulation. Regulation 10:13–24Google Scholar
  35. Presidential/Congressional Commission on Risk Assessment and Management (1997) Risk assessment and risk management in regulatory decision-making. Available at https://cfpub.epa.gov/ncea/risk/recordisplay.cfm?deid=55006
  36. Robinson LA, Hammitt JK, Zeckhauser RJ (2016) Attention to distribution in US regulatory analyses. Rev Environ Econ Policy 10(2):308–328CrossRefGoogle Scholar
  37. Shapiro S, Fisher E, Wagner W (2012) The enlightenment of administrative law: looking inside the agency for legitimacy. Wake Forest Law Rev 47:463–502Google Scholar
  38. Sherman R (2016) Roger Goodell warns fans about risks of watching the Super Bowl. Article in SB Nation, Feb. 5, available at https://www.sbnation.com/2016/2/5/10924418/roger-goodell-concussions-super-bowl-nfl-couches-are-dangerous
  39. Shlyakhter AI (1994) An improved framework for uncertainty analysis: accounting for unsuspected errors. Risk Anal 14:441–447CrossRefGoogle Scholar
  40. Society for Risk Analysis (2016) Technical symposium on “Transparency and uncertainty analysis: benefits and pitfalls.” Abstracts available at http://birenheide.com/sra/2016AM/program/singlesession.php3?sessid=W1-E&order=3#3
  41. Stahl CH, Cimorelli AJ (2005) How much uncertainty is too much and how do we know? A case example of the assessment of ozone monitor network options. Risk Anal 25:1109CrossRefGoogle Scholar
  42. Stern A (2015) “Comments on Uncertainty Analysis,” (presentation to the U.S. EPA) available at https://www.epa.gov/sites/production/files/2015-01/documents/session3b_stern_talking_points.pdf
  43. Sunstein CR (2002) Risk and reason: safety, law, and the environment. Cambridge University Press, CambridgeGoogle Scholar
  44. Thompson KM, Graham JD (1996) Going beyond the single number: using probabilistic risk assessment to improve risk management. Hum Ecol Risk Assess 2:1008–1034CrossRefGoogle Scholar
  45. Trump B, Cummings C, Kuzma J, Linkov I (2017) A decision analytic model to guide early-stage government regulatory action: applications for synthetic biology. Regul Gov.  https://doi.org/10.1111/rego.12142 Google Scholar
  46. Weitzman ML (2011) Fat-tailed uncertainty in the economics of catastrophic climate change. Rev Environ Econ Policy 5(2):275–292CrossRefGoogle Scholar
  47. Wilson R, Crouch EAC, Zeise L (1985) Uncertainty in risk assessment. In: Hoel DG et al, (eds) Risk quantitation and regulatory policy. Cold Spring HarborGoogle Scholar
  48. Yokota F, Thompson KM (2004) The value of information in environmental health risk management decisions: past, present, future. Risk Anal 24:635–650CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Risk Management and Decision Processes Center, Wharton SchoolUniversity of PennsylvaniaPhiladelphiaUSA
  2. 2.University of Michigan School of Public HealthAnn ArborUSA
  3. 3.Department of Environmental and Occupational Health, Milken Institute School of Public HealthGeorge Washington UniversityWashingtonUSA

Personalised recommendations