Abstract
Validation is the hallmark of science. For expert judgment to contribute to science-based uncertainty quantification, it must become amenable to empirical validation. Using data in which experts quantify uncertainty on variables from their fields whose true values are known post hoc, this chapter explains how validation is performed in the Classical Model for structured expert judgment and reviews results for different combination methods.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Full documentation is not available at this writing and the information here is based on http://www.npr.org/sections/parallels/2014/04/02/297839429/-so-you-think-youre-smarter-than-a-cia-agent accessed 1/12/2017 and Ungar et al. (2012).
References
Aspinall WP, Loughlin SC, Michael FV, Miller AD, Norton GE, Rowley KC, Sparks RSJ, Young SR (2002) The montserrat volcano observatory: its evolution, organisation, role and activities. In: Druitt TH, Kokelaar BP (eds) The eruption of Soufrière Hills Volcano, Montserrat, from 1995 to 1999. Geological Society, London, pp 71–92
Aspinall W (2010) A route to more tractable expert advice. Nature 463:294–295
Aspinall WP, Cooke RM, Havelaar AH, Hoffmann S, Hald T (2015) Evaluation of a performance-based expert elicitation: WHO global attribution of foodborne diseases. PLoS One 11(3):e0149817. doi:10.1371/journal.pone.0149817. eCollection 2016
Bamber JL, Aspinall WJ, Cooke RM (2016) A commentary on “How to interpret expert judgment assessments of twenty-first century sea-level rise” by Hylke de Vries and Roderik SW van de Wal. Clim Chang 137(3–4):321–328. doi:10.1007/s10584-016-1672-7
Clemen RT (2008) Comment on Cooke’s classical method. Reliab Eng Syst Saf 93(5):760–765
Colson A, Cooke RM (2017) Cross validation for the classical model of structured expert judgment. Reliab Eng Syst Saf 173:109–120
Cooke RM (2017) Strictly proper scoring rules as weights. In: Dias LC, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment. Springer, New York. 2018 (on-line Appendix of this book)
Cooke RM (1991) Experts in uncertainty, opinion and subjective probability in science. Oxford University Press, Oxford, p 321
Cooke RM, Solomatine D (1992) EXCALIBUR—integrated system for processing expert judgments, user’s manual version 3.0. Delft University of Technology and SoLogic Delft, Delft
Cooke RM (2008) Response to Comments, Special issue on expert judgment. Reliab Eng Syst Saf 93:775–777. Available online 12 March 2007. Volume 93, Issue 5, May 2008
Cooke RM, Goossens LHJ (2008) TU Delft expert judgment data base, special issue on expert judgment. Reliab Eng Syst Saf 93:657–674. Available online 12 March 2007, Issue 5, May 2008
Cooke RM (2012a) Pitfalls of ROAT cross validation comment on effects of overconfidence and dependence on aggregated probability judgments. J Model Manag 7(1):20–22. issn: 1746-5664
Cooke RM (2012b) Uncertainty analysis comes to integrated assessment models for climate change…and conversely. Clim Chang 117(3):467–479. doi:10.1007/s10584-012-0634-y
Cooke RM, Wittmann ME, Lodge DM, Rothlisberger JD, Rutherford ES, Zhang H, Mason DM (2014) Out-of-sample validation for structured expert judgment of asian carp establishment in Lake Erie. Integr Environ Assess Manag 10(4):522–528. doi:10.1002/ieam.1559
Cooke RM (2015) Messaging climate change uncertainty with supplementary online material. Nat Clim Chang 5:8–10. doi:10.1038/nclimate2466
Cooke RM, Mendel M, Thijs W (1988) Calibration and information in expert resolution. Automatica 24(1):87–94
De Groot M, Fienberg S (1983) Effective scoring rules for probabilities forecasts. Manag Sci 29(4):447–454
Eggstaff JW, Mazzuchi TA, Sarkani S (2014) The effect of the number of seed variables on the performance of Cooke’s classical model. Reliab Eng Syst Saf 121:72–82. doi:10.1016/j.ress.2013.07.015
Flandoli F, Giorgi E, Aspinall WP, Neri A (2011) Comparison of a expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique. Reliab Eng Syst Saf 96:1292–1310. doi:10.1016/j.ress.2011.05.012
Genest C, Zidek J (1986) Combining probability distributions: a critique and an annotated bibliography. Stat Sci 1(1):114–148
Gillingham K, Nordhaus WD, Anthoff D, Blanford G, Bosetti V, Christensen P, McJeon H, Reilly J, Sztorc P (2015) Modeling uncertainty in climate change: a multi-model comparison. In Working paper 21,637 NBER working paper series http://www.nber.org/papers/w21637. National Bureau of Economic Research
Hald T, Aspinall W, Devleesschauwer B, Cooke RM, Corrigan T, Havelaar AH, Gibb H, Torgerson P, Kirk M, Angulo F, Lake R, Speybroeck N, Hoffmann S (2015) World Health Organization estimates of the relative contributions of food to the burden of disease due to selected foodborne hazards: a structured expert elicitation. PLoS One 11:e0145839. doi:10.1371/journal.pone.0145839
Harper FT, Hora SC, Young ML, Miller LA, Lui CH, McKay MD, Helton JC, Goossens LHJ, Cooke RM, Pasler-Sauer J, Kraan B, Jones JA (1995) Probabilistic accident consequence uncertainty study: Dispersion and deposition uncertainty assessment. Prepared for U.S. Nuclear Regulatory Commission and Commission of European Communities NUREG/CR-6244, EUR 15855 EN, SAND94–1453, Washington/USA, and Brussels-Luxembourg, November 1994, published January 1995. Volume I: main report, volume II: appendices A and B, volume III: appendices C, D, E, F, G, H
Hoffmann S, Aspinall W, Cooke RM, Cawthorne A, Corrigan T, Havelaar AH, Gibb H, Torgerson P, Kirk M, Angulo FJ, Lake R, Speyboeck N, Devleesschauwer B, Hald T, World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force (2016) Research synthesis methods in an age of globalized risks: lessons from the global burden of foodborne disease expert elicitation. Risk Anal 36(2):191–202
Laddaga R (1977) Lehrer and the consensus proposal. Synthese 36:473–477
Lichtendahl KC Jr, Grushka-Cockayne Y, Winkler RL (2013) Is it better to average probabilities or quantiles? Manag Sci 59(7):1594–1611. issn: 0025-1909. doi:10.1287/mnsc.1120.1667
Lin S-W, Cheng C-H (2008) Can Cooke’s model sift out better experts and produce well-calibrated aggregated probabilities? Department of Business Administration, Yuan Ze University, Chung-Li, Taiwan Proceedings of the 2008 IEEE, IEEM
Lin S-W, Cheng C-H (2009) The reliability of aggregated probability judgments obtained through Cooke’s classical model. J Model Manag 4(2):149–161
Lin S-W, Huang S-W (2012) Effects of overconfidence and dependence on aggregated probability judgments. J Model Manag 7(1):6–22
NAS (2017) Valuing climate damages: updating estimation of the social cost of carbon dioxide. The National Academies Press, Washington, DC. doi:10.17226/24651
Oppenheimer M, Little CM, Cooke RM (2016) Expert judgment and uncertainty quantification for climate change. Nat Clim Chang 6:445–451. doi:10.1038/NCLIMATE2959
Quigley J, Colson A, Aspinall W, Cooke RM (2017) Elicitation in the classical method. In: Dias LC, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment. Springer, New York. 2018 (Chapter 2 in this book)
Rasmussen NC, et al (1975) Reactor safety study. An assessment of accident risks in U. S. commercial nuclear power plants. WASH-1400 (NUREG-75/014). Rockville, MD, USA: Federal Government of the United States, U.S. Nuclear Regulatory Commission
Ungar L, Mellors B, Satopää V, Baron J, Tetlock P, Ramos J, Swift S (2012) The Good Judgment Project: A Large Scale Test of Different Methods of Combining Expert Predictions. 2012 AAAI Fall Symposium Series
Wadge G, Aspinall WP (2014) A review of volcanic hazard and risk assessments at the Soufrière Hills Volcano, Montserrat from 1997 to 2011, Ch. 24. In: Wadge G, Robertson REA, Voight B (eds) The eruption of Soufrière Hills Volcano, Montserrat, from 2000 to 2010: Geological Society Memoirs, vol 39. Geological Society, London, pp 439–456
Winkler RL (1969) Scoring rules and the evaluation of probability assessors. J Am Stat Assoc 64:1073–1078
Wittmann ME, Cooke RM, Rothlisberger JD, Rutherford ES, Zhang H, Mason D, Lodge DM (2014) Structured expert judgment to forecast species invasions: bighead and silver carp in Lake Erie. Conserv Biol 29(1):187–197. doi:10.1111/cobi.12369
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
3.1 Electronic Supplementary Material
(DOCX 34 kb)
(DOCX 34 kb)
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Cooke, R.M. (2018). Validation in the Classical Model. In: Dias, L., Morton, A., Quigley, J. (eds) Elicitation. International Series in Operations Research & Management Science, vol 261. Springer, Cham. https://doi.org/10.1007/978-3-319-65052-4_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-65052-4_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-65051-7
Online ISBN: 978-3-319-65052-4
eBook Packages: Business and ManagementBusiness and Management (R0)