Expert Judgment Quantification

  • Muhammad Amer
  • Tugrul Daim
Part of the Green Energy and Technology book series (GREEN)


Expert judgments are used when there are no objective data is available. It is critical to solicit these judgments accurately for decision makers. This chapter reviews methods and issues around the expert judgment quantification.


Analytic Hierarchy Process Expert Panel Expert Judgment Delphi Method Delphi Study 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Rohrbaugh J (1979) Improving the quality of group judgment: social judgment analysis and the Delphi technique. Organ Behav Hum Perform 24:73–92Google Scholar
  2. 2.
    Meyer MA, Booker JM (1991) Eliciting and analyzing expert judgment: a practical guide. Academic Press Limited, LondonGoogle Scholar
  3. 3.
    Keeney RL, von Winterfeldt D (1989) On the uses of expert judgment on complex technical problems. IEEE Trans Eng Manage 36:83–86Google Scholar
  4. 4.
    Durance P, Godet M (2010) Scenario building: uses and abuses. Technol Forecast Soc Chang 77:1488–1492Google Scholar
  5. 5.
    Bradfield R, Wright G, Burt G, Cairns G, Van Der Heijden K (2005) The origins and evolution of scenario techniques in long range business planning. Futures 37:795–812Google Scholar
  6. 6.
    Cooke RM (1991) Experts in uncertainty: opinion and subjective probability in science. Oxford University Press, New YorkGoogle Scholar
  7. 7.
    Simola K, Mengolini A, Bolado-Lavin R (2005) Formal expert judgment: an overview. European Commission, Directorate General Joint Research Centre (DG JRC), Institute for Energy. Accessed 18 March 2011
  8. 8.
    Börjeson L, Höjer M, Dreborg K-H, Ekvall T, Finnveden G (2006) Scenario types and techniques: towards a user’s guide. Futures 38:723–739Google Scholar
  9. 9.
    Amer M, Daim TU (2010) Application of technology roadmaps for renewable energy sector. Technol Forecast Soc Chang 77:1355–1370Google Scholar
  10. 10.
    Gerdsri N (2005) An analytical approach on building a technology development envelope (TDE) for roadmapping of emerging technologies. Ph.D. Dissertation, Systems Science: Engineering Management, Portland State University, PortlandGoogle Scholar
  11. 11.
    Gerdsri P (2009) A systematic approach to developing national technology policy and strategy for emerging technologies. Ph.D. Dissertation, Engineering and Technology Management (ETM) Department, Portland State University, PortlandGoogle Scholar
  12. 12.
    Schwartz P (1996) The Art of the Long View: Planning for the Future in an Uncertain World. Currency Doubleday, New YorkGoogle Scholar
  13. 13.
    Chermack TJ, Lynham SA, Ruona WEA (2001) A review of scenario planning literature. Futures Res Q 17:7–31Google Scholar
  14. 14.
    Kahn H, Wiener AJ (1967) The year 2000: a framework for speculation on the next thirty-three years. The Macmillan, New YorkGoogle Scholar
  15. 15.
    Varum CA, Melo C (2007) Strategic planning in an uncertain business environment: the diffusion of scenario planning. In: Conference Factores de Competitividade, Competitiveness Factors: A Portuguese Perspective, AveiroGoogle Scholar
  16. 16.
    Mietzner D, Reger G (2005) Advantages and disadvantages of scenario approaches for strategic foresight. Int J Technol Intell Planning 1:220–239Google Scholar
  17. 17.
    Varum CA, Melo C (2010) Directions in scenario planning literature: a review of the past decades. Futures 42:355–369Google Scholar
  18. 18.
    Van Der Heijden K (1996) Scenarios: the art of strategic conversation. Wiley, ChichesterGoogle Scholar
  19. 19.
    Rigby D, Bilodeau B (2007) Selecting management tools wisely. Harv Bus Rev 85:20–22Google Scholar
  20. 20.
    Porter AL, Roper AT, Mason TW, Rossini FA, Banks J (1991) Forecasting and management of technology, 1st edn. Wiley, New YorkGoogle Scholar
  21. 21.
    Saritas O, Oner MA (2004) Systemic analysis of UK foresight results: joint application of integrated management model and roadmapping. Technol Forecast Soc Chang 71:27–65Google Scholar
  22. 22.
    Chermack TJ, Lynham SA, van der Merwe L (2006) Exploring the relationship between scenario planning and perceptions of learning organization characteristics. Futures 38:767–777Google Scholar
  23. 23.
    Schoemaker PJH, van der Heijden CAJM (1992) Integrating scenarios into strategic planning at royal Dutch/Shell. Strategy Leadersh 20:41–46Google Scholar
  24. 24.
    Joseph CF (2000) Scenario planning. Technol Forecast Soc Chang 65:115–123Google Scholar
  25. 25.
    Hiltunen E (2009) Scenarios: process and outcome. J Futures Stud 13:151–152Google Scholar
  26. 26.
    Linneman RE, Klein HE (1979) The use of multiple scenarios by U.S. industrial companies. Long Range Plan 12:83–90Google Scholar
  27. 27.
    Linneman RE, Klein HE (1983) The use of multiple scenarios by U.S. industrial companies: a comparison study, 1977–1981. Long Range Plan 16:94–101Google Scholar
  28. 28.
    Amer M, Daim TU, Jetter A (2013) A review of scenario planning. Futures 46:23–40Google Scholar
  29. 29.
    Blomgren H, Jonsson P, Lagergren F (2011) Getting back to scenario planning: strategic action in the future of energy Europe. In: 8th international conference on the European energy market (EEM), Zagreb, pp 792–801Google Scholar
  30. 30.
    Silberglitt R, Hove A, Shulman P (2003) Analysis of the US energy scenarios: meta-scenarios, pathways, and policy implications. Technol Forecast Soc Chang 70:297–315Google Scholar
  31. 31.
    Czaplicka-Kolarz K, Stanczyk K, Kapusta K (2009) Technology foresight for a vision of energy sector development in Poland till 2030. Delphi survey as an element of technology foresighting. Technol Forecast Soc Chang 76:327–338Google Scholar
  32. 32.
    Di W, Rui N, Hai-ying S (2011) Scenario analysis of China’s primary energy demand and CO2 emissions based on IPAT model. Energy Procedia 5:365–369Google Scholar
  33. 33.
    Keles D, Möst D, Fichtner W (2011) The development of the German energy market until 2030—a critical survey of selected scenarios. Energy Policy 39:812–825Google Scholar
  34. 34.
    Zhou N, Lin J (2008) The reality and future scenarios of commercial building energy consumption in China. Energy Build 40:2121–2127Google Scholar
  35. 35.
    Winebrake JJ, Creswick BP (2003) The future of hydrogen fueling systems for transportation: an application of perspective-based scenario analysis using the analytic hierarchy process. Technol Forecast Soc Chang 70:359–384Google Scholar
  36. 36.
    Sørensen B, Hauge Petersen A, Juhl C, Ravn H, Søndergren C, Simonsen P, Jørgensen K, Henrik Nielsen L, Larsen HV, Erik Morthorst P, Schleisner L, Sørensen F, Engberg Pedersen T (2004) Hydrogen as an energy carrier: scenarios for future use of hydrogen in the Danish energy system. Int J Hydrogen Energy 29:23–32Google Scholar
  37. 37.
    Wietschel M, Hasenauer U, de Groot A (2006) Development of European hydrogen infrastructure scenarios–CO2 reduction potential and infrastructure investment. Energy Policy 34:1284–1298Google Scholar
  38. 38.
    Contaldi M, Gracceva F, Mattucci A (2008) Hydrogen perspectives in Italy: analysis of possible deployment scenarios. Int J Hydrogen Energy 33:1630–1642Google Scholar
  39. 39.
    Antoine B, Goran K, Neven D (2008) Energy scenarios for Malta. Int J Hydrogen Energy 33:4235–4246Google Scholar
  40. 40.
    Weisser D (2004) Costing electricity supply scenarios: a case study of promoting renewable energy technologies on Rodriguez, Mauritius. Renewable Energy 29:1319–1347Google Scholar
  41. 41.
    Chen T-Y, Yu OS, Hsu GJY, Hsu F-M, Sung W-N (2009) Renewable energy technology portfolio planning with scenario analysis: a case study for Taiwan. Energy Policy 37:2900–2906Google Scholar
  42. 42.
    Madlener R, Kowalski K, Stagl S (2007) New ways for the integrated appraisal of national energy scenarios: the case of renewable energy use in Austria. Energy Policy 35:6060–6074Google Scholar
  43. 43.
    Varho V, Tapio P (2005) Wind power in Finland up to the year 2025—‘soft’ scenarios based on expert views. Energy Policy 33:1930–1947Google Scholar
  44. 44.
    Morales JM, Mínguez R, Conejo AJ (2010) A methodology to generate statistically dependent wind speed scenarios. Appl Energy 87:843–855Google Scholar
  45. 45.
    Rowe G, Wright G (1999) The Delphi technique as a forecasting tool: issues and analysis. Int J Forecast 15:353–375Google Scholar
  46. 46.
    Raubitschek R (1988) Multiple scenario analysis and business planning. In: Lamb R, Shrivastava P (eds) Advances in strategic management, vol 5. JAI Press Inc., LondonGoogle Scholar
  47. 47.
    Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley, ReadingMATHGoogle Scholar
  48. 48.
    Martino JP (1983) Technological forecasting for decision making, 2nd edn. North-Holland, New YorkGoogle Scholar
  49. 49.
    Rowe G, Wright G, Bolger F (1991) Delphi: a reevaluation of research and theory. Technol Forecast Soc Chang 39:235–251Google Scholar
  50. 50.
    Erffmeyer RC, Erffmeyer ES, Lane IM (1986) The Delphi technique: an empirical evaluation of the optimal number of rounds. Group and Organ Manage 11:120–128Google Scholar
  51. 51.
    Lindeman CA (1975) Delphi survey of priorities in clinical nursing research. Nurs Res 24:434–441Google Scholar
  52. 52.
    Torrance EP (1957) Group decision-making and disagreement. Soc Forces 35:314–318Google Scholar
  53. 53.
    Powell C (2003) The Delphi technique: myths and realities. J Adv Nurs 41:376–382Google Scholar
  54. 54.
    Woudenberg F (1991) An evaluation of Delphi. Technol Forecast Soc Chang 40:131–150Google Scholar
  55. 55.
    Jairath N, Weinstein J (1994) The Delphi methodology: a useful administrative approach. Can J Nurs Adm 7:29–42Google Scholar
  56. 56.
    Linstone HA, Turoff M (2011) Delphi: a brief look backward and forward. Technol Forecast Soc Chang 78:1712–1719Google Scholar
  57. 57.
    Chaffin WW, Talley WK (1980) Individual stability in Delphi studies. Technol Forecast Soc Chang 16:67–73Google Scholar
  58. 58.
    Gordon T, Pease A (2006) RT Delphi: an efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Chang 73:321–333Google Scholar
  59. 59.
    Celiktas MS, Kocar G (2010) From potential forecast to foresight of Turkey’s renewable energy with Delphi approach. Energy 35:1973–1980Google Scholar
  60. 60.
    Steinert M (2009) A dissensus based online Delphi approach: an explorative research tool. Technol Forecast Soc Chang 76:291–300Google Scholar
  61. 61.
    Geist MR (2010) Using the Delphi method to engage stakeholders: a comparison of two studies. Eval Program Plann 33:147–154Google Scholar
  62. 62.
    Rauch W (1979) The decision Delphi. Technol Forecast Soc Chang 15:159–169Google Scholar
  63. 63.
    Turoff M (1970) The design of a policy Delphi. Technol Forecast Soc Chang 2:149–171Google Scholar
  64. 64.
    Delbecq AL, Van de Ven AH, Gustafson DH (1975) Group techniques for program planning. A guide to nominal group and Delphi processes. Scott, Foresman and Company, GlenviewGoogle Scholar
  65. 65.
    Murphy M, Black N, Lamping D, McKee CM, Sanderson CFB, Askham J (1998) Consensus development methods, and their use in clinical guideline development. Health Technol Assess 2:1–65Google Scholar
  66. 66.
    Pill J (1971) The Delphi method: substance, context, a critique and an annotated bibliography. Socio-Econ Plan Sci 5:57–71Google Scholar
  67. 67.
    Graham B, Regehr G, Wright JG (2003) Delphi as a method to establish consensus for diagnostic criteria. J Clin Epidemiol 56:1150–1156Google Scholar
  68. 68.
    Brockhoff K (1975) The performance of forecasting groups in computer dialogue and face to face discussions. In: Linstone HA, Turoff M (eds) The Delphi method: techniques and applications. Addison-Wesley, ReadingGoogle Scholar
  69. 69.
    Riggs WE (1983) The Delphi technique: an experimental evaluation. Technol Forecast Soc Chang 23:89–94Google Scholar
  70. 70.
    Larreche JC, Moinpour R (1983) Managerial judgment in marketing: the concept of expertise. J Mark Res 20:110–121Google Scholar
  71. 71.
    Coates JF (1975) In defense of Delphi: a review of Delphi assessment, expert opinion, forecasting, and group process by H. Sackman. Technol Forecast Soc Chang 7:193–194Google Scholar
  72. 72.
    Sackman H (1974) Delphi critique: expert opinion, forecasting, and group process. Lexington Books, LexingtonGoogle Scholar
  73. 73.
    Martino JP (2003) A review of selected recent advances in technological forecasting. Technol Forecast Soc Chang 70:719–733Google Scholar
  74. 74.
    Breiner S, Cuhls K, Grupp H (1994) Technology foresight using a Delphi approach: a Japanese German study. R&D Manage 24:141–153Google Scholar
  75. 75.
    Biloslavo R, Dolinek S (2008) Scenario planning for climate strategies development by integrating group Delphi, AHP and dynamic fuzzy cognitive maps. In: Portland International Center for Management of Engineering and Technology (PICMET), Cape Town, South Africa, pp 1103–1111Google Scholar
  76. 76.
    Huss WR, Honton EJ (1987) Scenario planning—what style should you use? Long Range Plan 20:21–29Google Scholar
  77. 77.
    Bengisu M, Nekhili R (2006) Forecasting emerging technologies with the aid of science and technology databases. Technol Forecast Soc Chang 73:835–844Google Scholar
  78. 78.
    Gupta UG, Clarke RE (1996) Theory and applications of the Delphi technique: a bibliography (1975–1994). Technol Forecast Soc Chang 53:185–211Google Scholar
  79. 79.
    Iniyan S, Sumathy K (2000) An optimal renewable energy model for various end-uses. Energy 25:563–575Google Scholar
  80. 80.
    Iniyan S, Sumathy K (2003) The application of a Delphi technique in the linear programming optimization of future renewable energy options for India. Biomass Bioenergy 24:39–50Google Scholar
  81. 81.
    Pätäri S (2010) Industry- and company-level factors influencing the development of the forest energy business—insights from a Delphi study. Technol Forecast Soc Chang 77:94–109Google Scholar
  82. 82.
    Rikkonen P, Tapio P (2009) Future prospects of alternative agro-based bioenergy use in Finland–Constructing scenarios with quantitative and qualitative Delphi data. Technol Forecast Soc Chang 76:978–990Google Scholar
  83. 83.
    Ronde P (2003) Delphi analysis of national specificities in selected innovative areas in Germany and France. Technol Forecast Soc Chang 70:419–448Google Scholar
  84. 84.
    Sharma DP, Nair PSC, Balasubramanian R (2003) Analytical search of problems and prospects of power sector through Delphi study: case study of Kerala state, India. Energy Policy 31:1245–1255Google Scholar
  85. 85.
    Suganthi L, Williams A (2000) Renewable energy in India—a modelling study for 2020–2021. Energy Policy 28:1095–1109Google Scholar
  86. 86.
    Utgikar VP, Scott JP (2006) Energy forecasting: predictions, reality and analysis of causes of error. Energy Policy 34:3087–3092Google Scholar
  87. 87.
    Blind K, Cuhls K, Grupp H (2001) Personal attitudes in the assessment of the future of science and technology: a factor analysis approach. Technol Forecast Soc Chang 68:131–149Google Scholar
  88. 88.
    Cuhls K, Kuwahara T (1994) Outlook of Japanese and German future technology: comparing technology forecast surveys. Phisica-Verlag, HeidelbergGoogle Scholar
  89. 89.
    Amer M, Daim T (2012) Technology and science policies in transitional economy: a case of Turkey. Sci Technol Soc 17:297–321Google Scholar
  90. 90.
    Gerdsri P Kocaoglu D (2009) A systematic approach to developing national technology policy and strategy for emerging technologies: a case study of nanotechnology for Thailand’s agriculture industry. In: Portland International Center for Management of Engineering and Technology (PICMET), Portland, pp 447–461Google Scholar
  91. 91.
    Chakravarti AK, Vasanta B, Krishnan ASA, Dubash RK (1998) Modified Delphi methodology for technology forecasting case study of electronics and information technology in India. Technol Forecast Soc Chang 58:155–165Google Scholar
  92. 92.
    Shin T (1998) Using Delphi for a long-range technology forecasting, and assessing directions of future R&D activities the Korean exercise. Technol Forecast Soc Chang 58:125–154Google Scholar
  93. 93.
    Tichy G (2004) The over-optimism among experts in assessment and foresight. Technol Forecast Soc Chang 71:341–363Google Scholar
  94. 94.
    Daim T, Yates D, Peng Y, Jimenez B (2009) Technology assessment for clean energy technologies: the case of the Pacific northwest. Technol Soc 31:232–243Google Scholar
  95. 95.
    Saaty TL (2003) Decision-making with the AHP: why is the principal eigenvector necessary. Eur J Oper Res 145:85–91MathSciNetMATHGoogle Scholar
  96. 96.
    Wang J–J, Jing Y–Y, Zhang C-F, Zhao J-H (2009) Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renew Sustain Energy Rev 13:2263–2278Google Scholar
  97. 97.
    Salo A, Gustafsson T, Ramanathan R (2003) Multicriteria methods for technology foresight. J Forecast 22:235–255Google Scholar
  98. 98.
    Elkarmi F, Mustafa I (1993) Increasing the utilization of solar energy technologies (SET) in Jordan analytic hierarchy process. Energy Policy 21:978–982Google Scholar
  99. 99.
    Forman EH, Gass SI (2001) The analytic hierarchy process—an exposition. Operations Research 49:469–486MathSciNetMATHGoogle Scholar
  100. 100.
    Saaty TL (1980) The analytic hierarchy process: planning, priority setting, resource allocation. McGraw Hill International, New YorkGoogle Scholar
  101. 101.
    Gerdsri N, Kocaoglu DF (2007) Applying the analytic hierarchy process (AHP) to build a strategic framework for technology roadmapping. Math Comput Model 46:1071–1080Google Scholar
  102. 102.
    Kocaoglu DF (1983) A participative approach to program evaluation. IEEE Trans Eng Manage 30:112–118Google Scholar
  103. 103.
    Kablan MM (2004) Decision support for energy conservation promotion: an analytic hierarchy process approach. Energy Policy 32:1151–1158Google Scholar
  104. 104.
    Ahsan MK, Bartlema J (2004) Monitoring healthcare performance by analytic hierarchy process: a developing-country perspective. Int Trans Oper Res 11:465–478MATHGoogle Scholar
  105. 105.
    Yates D, Jimenez BT, Peng Y (2007) Portland general electric (PGE): clean power generation wind project in biglow canyon boardman coal plant. In: PICMET, Portland, pp 2530–2549Google Scholar
  106. 106.
    Akash BA, Mamlook R, Mohsen MS (1999) Multi-criteria selection of electric power plants using analytical hierarchy process. Electric Power Syst Res 52:29–35Google Scholar
  107. 107.
    Lee SK, Yoon YJ, Kim JW (2007) A study on making a long-term improvement in the national energy efficiency and GHG control plans by the AHP approach. Energy Policy 35:2862–2868Google Scholar
  108. 108.
    Lee SK, Mogi G, Kim JW (2008) The competitiveness of Korea as a developer of hydrogen energy technology: the AHP approach. Energy Policy 36:1284–1291Google Scholar
  109. 109.
    Lee SK, Mogi G, Kim JW, Gim BJ (2008) A fuzzy analytic hierarchy process approach for assessing national competitiveness in the hydrogen technology sector. Int J Hydrogen Energy 33:6840–6848Google Scholar
  110. 110.
    Lee AHI, Chen HH, Kang H-Y (2008) Multi-criteria decision making on strategic selection of wind farms. Renewable Energy 34:120–126Google Scholar
  111. 111.
    Aras H, Erdogmus S, Koc E (2004) Multi-criteria selection for a wind observation station location using analytic hierarchy process. Renewable Energy 29:1383–1392Google Scholar
  112. 112.
    Jaber JO, Jaber QM, Sawalha SA, Mohsen MS (2008) Evaluation of conventional and renewable energy sources for space heating in the household sector. Renew Sustain Energy Rev 12:278–289Google Scholar
  113. 113.
    Lee TL, Lin HM, Jeng DS, Hsu TW (2008) Application of fuzzy analytic hierarchy process to assess the potential of offshore wind energy in Taiwan. In: Proceedings of the international offshore and polar engineering conference. Vancouver, pp 461–465Google Scholar
  114. 114.
    Pilavachi PA, Chatzipanagi AI, Spyropoulou AI (2009) Evaluation of hydrogen production methods using the analytic hierarchy process. Int J Hydrogen Energy 34:5294–5303Google Scholar
  115. 115.
    Tsoutsos T, Drandaki M, Frantzeskaki N, Iosifidis E, Kiosses I (2009) Sustainable energy planning by using multi-criteria analysis application in the island of Crete. Energy Policy 37:1587–1600Google Scholar
  116. 116.
    Wijayatunga PDC, Siriwardena K, Fernando WJLS, Shrestha RM, Attalage RA (2006) Strategies to overcome barriers for cleaner generation technologies in small developing power systems: Sri Lanka case study. Energy Convers Manage 47:1179–1191Google Scholar
  117. 117.
    Chatzimouratidis AI, Pilavachi PA (2008) Multicriteria evaluation of power plants impact on the living standard using the analytic hierarchy process. Energy Policy 36:1074–1089Google Scholar
  118. 118.
    Chatzimouratidis AI, Pilavachi PA (2009) Technological, economic and sustainability evaluation of power plants using the analytic hierarchy process. Energy Policy 37:778–787Google Scholar
  119. 119.
    Clarke RR (1997) Validation and legitimation of an analytic hierarchy approach to integrated resource planning for electric utilities. In: Energy conversion engineering conference (IECEC-97), Honolulu, pp 2197–2201Google Scholar
  120. 120.
    Afgan NH, Carvalho MG (2002) Multi-criteria assessment of new and renewable energy power plants. Energy 27:739–755Google Scholar
  121. 121.
    Amer M, Daim TU (2011) Selection of renewable energy technologies for a developing country: a case study of Pakistan. Energy Sustain Dev 15:420–435Google Scholar
  122. 122.
    Millett SM, Honton EJ (1991) A manager’s guide to technology forecasting and strategy analysis methods. Battelle Press, ColumbusGoogle Scholar
  123. 123.
    Rohrbaugh J (1981) Improving the quality of group judgment: social judgment analysis and the nominal group technique. Organ Behav Hum Perform 28:272–288Google Scholar
  124. 124.
    Herbert TT, Yost EB (1979) A comparison of decision quality under nominal and interacting consensus group formats: the case of the structured problem. Decis Sci 10:358–370Google Scholar
  125. 125.
    Islam R (2001) Modification of the nominal group technique by using the analytic hierarchy process. In: Multiple Criteria Decision Making in the New Millennium, vol 507, pp 294–303Google Scholar
  126. 126.
    Henderson NR (2009) Managing moderator stress: take a deep breath. You can do this! Mark Res 21:28–29Google Scholar
  127. 127.
    Sutton RI, Hargadon A (1996) Brainstorming groups in context: effectiveness in a product design firm. Adm Sci Q 41:685–718Google Scholar
  128. 128.
    Miles I, Keenan M (2002) Practical guide to regional foresight in the United Kingdom. Directorate-General for Research, European Commission. Accessed 14 March 2011
  129. 129.
    Keeney RL, von Winterfeldt D (1991) Eliciting probabilities from experts in complex technical problems. IEEE Trans Eng Manage 38:191–201Google Scholar
  130. 130.
    Cojazzi G, Fogli D, Grassini G, De Gelder P, Gryffroy D, Bolado R, Hofer E, Virolainen R, Coe IM, Bassanelli A, Puga J, Papazoglou I, Zuchuat O, Cazzoli E, Eyink J, Guida G, Pinola L, Pulkkinen U, Simola K, von Winterfeldt D, Valeri A (2001) Benchmark exercise on expert judgment techniques in PSA Level 2. Nucl Eng Des 209:211–221Google Scholar
  131. 131.
    Cojazzi G, Fogli D (2000) Benchmark exercise on expert judgment techniques in PSA level 2, extended final report.
  132. 132.
    Meyer MA, Booker JM (1991) Selecting and motivating the experts. In: Eliciting and analyzing expert judgment: a practical guide, pp 85–98, 1st edn. Academic Press Limited, LondonGoogle Scholar
  133. 133.
    Camerer CF, Johnson EJ (1991) The process-performance paradox in expert judgment. In: Ericsson KA, Smith J (eds) Toward a general theory of expertise: prospects and limits, 1st edn. Cambridge University Press, Cambridge, pp 195–217Google Scholar
  134. 134.
    McGraw KL, Harbison-Briggs K (1989) Knowledge acquisition: principles and guidelines. Prentice-Hall International, New JerseyGoogle Scholar
  135. 135.
    Landeta J (2006) Current validity of the Delphi method in social sciences. Technol Forecast Soc Chang 73:467–482Google Scholar
  136. 136.
    Jetter AJ (2006) Elicitation-extracting knowledge from experts. In: Jetter AJ, Kraaijenbrink J, Schroder H-H, Wijnhoven F (eds) Knowledge integration: the practice of knowledge management in small and medium enterprises, pp 65–76. Physica-Verlag, HeidelbergGoogle Scholar
  137. 137.
    Schaller RR (2004) Technology innovation in the semiconductor industry: a case study of the international technology roadmap for semiconductors (ITRS) Ph.D. Dissertation, School of Public Policy, George Mason University, FairfaxGoogle Scholar
  138. 138.
    Shanteau J (1992) The psychology of experts: an alternative view. In: Wright, G. Bolger F (eds) Expertise and decision support, Plenum Press, pp 11–23. New YorkGoogle Scholar
  139. 139.
    Ascher W (1978) Forecasting: An Appraisal for Policymakers and Planners. Johns Hopkins University Press, BaltimoreGoogle Scholar
  140. 140.
    Mitchell VW (1991) The Delphi technique: an exposition and application. Technol Anal Strat Manage 3:333–358Google Scholar
  141. 141.
    Okoli C, Pawlowsk SD (2004) The Delphi method as a research tool: an example, design considerations and applications. Inf Manage 42:15–29Google Scholar
  142. 142.
    Daim TU (1998) Technology evaluation and acquisition strategies and their implications in the U.S. electronics manufacturing industry. Ph.D. Dissertation, System Science: Engineering and Technology Management, Portland State University, PortlandGoogle Scholar
  143. 143.
    Hill KQ, Fowles J (1975) The methodological worth of the Delphi forecasting technique. Technol Forecast Soc Chang 7:179–192Google Scholar
  144. 144.
    Cooke RM, Goossens LHJ (2000) Procedures guide for structured expert judgement. Accessed 12 May 2011
  145. 145.
    Dajani JS, Sincoff MZ, Talley WK (1979) Stability and agreement criteria for the termination of Delphi studies. Technol Forecast Soc Chang 13:83–90Google Scholar
  146. 146.
    Kastein MR, Jacobs M, van der Hell RH, Luttik K, Touw-Otten FWMM (1993) Delphi, the issue of reliability: a qualitative Delphi study in primary health care in the Netherlands. Technol Forecast Soc Chang 44:315–323Google Scholar
  147. 147.
    Goodman LA, Kruskal WH (1959) Measures of association for cross classifications. II: further discussion and references. J Am Stat Assoc 54:123–163MATHGoogle Scholar
  148. 148.
    Mumpower JL, Stewart TR (1996) Expert judgement and expert disagreement. Thinking Reasoning 2:191–211Google Scholar
  149. 149.
    Bland JM, Altman DG (1997) Statistics notes: Cronbach’s alpha. Br Med J 314:572Google Scholar
  150. 150.
    Terrados J, Almonacid G, Pérez-Higueras P (2009) Proposal for a combined methodology for renewable energy planning. Application to a Spanish region. Renew Sustain Energy Rev 13:2022–2030Google Scholar
  151. 151.
    Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86:420–428Google Scholar
  152. 152.
    Wuensch KL (2007) Inter-rater agreement.
  153. 153.
    Coates JF (1975) In defense of Delphi: a review of Delphi assessment: expert opinion, forecasting, and group process by H. Sackman. Technol Forecast Soc Chang 7:193–194Google Scholar
  154. 154.
    Kandasamy WBV, Smarandache F (2003) Fuzzy cognitive maps and neutrosophic cognitive maps. Indian institute of technology, ChennaiMATHGoogle Scholar
  155. 155.
    Taber R (1991) Knowledge processing with fuzzy cognitive maps. Expert Syst Appl 2:83–87Google Scholar
  156. 156.
    Taber W, Siegel M (1987) Estimation of experts’ weights using fuzzy cognitive maps. In: IEEE International Conference on Neural Networks, pp 319–326Google Scholar
  157. 157.
    Taber R, Yager RR, Helgason CM (2007) Quantization effects on the equilibrium behavior of combined fuzzy cognitive maps. Int J Intell Syst 22:181MATHGoogle Scholar
  158. 158.
    Hans-Horst S, Jetter AJM (2003) Integrating market and technological knowledge in the fuzzy front end: an FCM-based action support system. Int J Technol Manage 26:517–539Google Scholar
  159. 159.
    Jetter AJM (2003) Educating the guess: strategies, concepts, and tools for the fuzzy front end of product development. In: Portland International Center for Management of Engineering and Technology (PICMET), Portland, pp 261–273Google Scholar
  160. 160.
    Amer M, Jetter AJ, Daim TU (2011) Development of fuzzy cognitive map (FCM) based scenarios for wind energy. Int J Energy Sect Manage 5:564–584Google Scholar
  161. 161.
    Babbie ER (2007) The practice of social research, 12th edn. Wadsworth, BelmontGoogle Scholar
  162. 162.
    Ono R, Wedemeyer DJ (1994) Assessing the validity of the Delphi technique. Futures 26:289–304Google Scholar
  163. 163.
    Martino JP (1970) The consistency of Delphi forecasts. Futurist 4:63–64Google Scholar
  164. 164.
    Keeney S, Hasson F, McKenna HP (2001) A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud 38:195–200Google Scholar
  165. 165.
    Engels TCE, Powell Kennedy H (2007) Enhancing a Delphi study on family-focused prevention. Technol Forecast Soc Chang 74:433–451Google Scholar
  166. 166.
    van Zolingen SJ, Klaassen CA (2003) Selection processes in a Delphi study about key qualifications in senior secondary vocational education. Technol Forecast Soc Chang 70:317–340Google Scholar
  167. 167.
    Jolson MA, Rossow GL (1971) The Delphi process in marketing decision making. J Mark Res 8:443–448Google Scholar
  168. 168.
    Brown BB, Helmer O (1964) Improving the reliability of estimates obtained from a consensus of experts, pp 1–13.
  169. 169.
    Dalkey NC, Brown BB, Cochran SW (1970) The Delphi Method IV: effect of percentile feedback and feed-in of relevant facts.
  170. 170.
    Landeta J, Matey J, Ruíz V, Galter J (2008) Results of a Delphi survey in drawing up the input-output tables for Catalonia. Technol Forecast Soc Chang 75:32–56Google Scholar
  171. 171.
    Chambers JC, Mullick SK, Smith DD (1971) How to choose the right forecasting technique. Harv Bus Rev 49:45–74Google Scholar
  172. 172.
    Levary RR, Han D (1995) Choosing a technological forecasting method, Indus Manag 37:14–18Google Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  1. 1.Portland State UniversityPortlandUSA

Personalised recommendations