Skip to main content

Integrated Framework for Evaluation of National Foresight Studies

  • Chapter
  • First Online:
Book cover Science, Technology and Innovation Policy for the Future

Abstract

With the growing importance of Foresight in strategic decision-making, there is parallel growth in interest in ways in which we might evaluate Foresight activities and outputs, and thus learn how to improve their effectiveness. But general principles for such evaluation have not yet been formulated. The purpose of this chapter is to identify the basic criteria and methods used in evaluating national Foresight exercises. Reviewing a range of evaluation efforts, it presents some interesting results for specific tasks, themes examined, methods and findings. A key motivation for these evaluations was to provide feedback to national Foresight organizations and identify areas for further development. Evaluation is used to guide Foresight and to learn lessons that can inform future projects (for example, lessons concerning simplifying implementation, involving business and social organizations, recognizing the objectives of participants and the need for better compliance with the methodology). We suggest a general methodology for assessing national Foresight programmes, together with a framework that can be used to improve the comparability of results of Foresight evaluation initiatives in different countries, and even to promote the standardization and upgrading of evaluation procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Behavioural additionality is the difference in actors’ behaviour resulting from the Foresight intervention (Georghiou et al. 2004b).

  2. 2.

    It is a framework of the adjusted integrated Foresight management model.

  3. 3.

    The main reasons for that choice were the success of the evaluation procedures and the openness of information. The latter is a crucial requirement: evaluation results are sometimes classified. For example, evaluation reports on “FUTUR” and “Vision 2023” were not published, although the sufficient minimum of information was presented in several papers (e.g. Cuhls and Georghiou 2004; Dursun et al. 2011).

  4. 4.

    Bundesministerium für Bildung und Forschung, BMBF.

  5. 5.

    For main themes were analysed: Create Open Access to Tomorrow’s World of Learning, Living in the Networked World: Individual and Secure, Healthy and Vital throughout Life through Prevention, Understanding Thought Processes (Cuhls and Georghiou 2004).

  6. 6.

    There is a wide range of indicators for the last two topics evaluation: specific criteria are developed and applied to meet a particular project’s needs (e.g. Johnston 2012; Chan and Daim 2012; Miles 2012; Kappel 2001). In-depth analysis of Foresight results and impact is beyond the bounds of the research: it’s an issue for further development.

  7. 7.

    Capability to influence on the situation in national innovation system (Meissner and Cervantes 2008).

References

  • Alsan A, Öner MA (2004) Comparison of national Foresight studies by integrated Foresight management model. Futures 36(8):889–902

    Article  Google Scholar 

  • Amanatidou E, Guy K (2008) Interpreting Foresight process impacts: steps towards the development of a framework conceptualising the dynamics of ‘Foresight systems’. Technol Forecast Soc Change 75(4):539–557

    Article  Google Scholar 

  • Bezold C (2010) Lessons from using scenarios for strategic Foresight. Technol Forecast Soc Change 77:1513–1518

    Article  Google Scholar 

  • Bohanec M, Rajkovič V, Semolič B, Pogačnik A (1995) Knowledge-based portfolio analysis for project evaluation. Inf Manage 28:293–302

    Article  Google Scholar 

  • Calof J (2011) Evaluating future technology assessment – Canadian Case Study. Paper presented at the 4th international Seville conference on future-oriented technology analysis (FTA), Seville, 12–13 May. http://ftp.jrc.es/EURdoc/JRC64647.pdf

  • Calof J, Smith JE (2008) Critical success factors for government led Foresight. Paper presented at the third international Seville seminar on future oriented technology analysis: impacts and implications for policy and decision-making, Seville, 16–17 October

    Google Scholar 

  • Chan L, Daim T (2012) Exploring the impact of technology Foresight studies on innovation: case of BRIC countries. Futures 44(6):618–630

    Google Scholar 

  • Clean Air Partnership (CAP) (2010) Evaluation toolkit: a guide for environmental NGOs. http://www.cleanairpartnership.org/building_evaluation_capacity_private

  • Cuhls K (2003) From forecasting to Foresight processes—new participative Foresight activities in Germany. J Forecast 22:93–111

    Article  Google Scholar 

  • Cuhls K, Georghiou L (2004) Evaluating a participative Foresight process: “Futur—the German research dialogue”. Res Eval 13(3):143–153

    Article  Google Scholar 

  • Destatte P (2007) Evaluation of Foresight: how to take long term impacts into consideration? FOR-LEARN Mutual Learning Workshop-Evaluation of Foresight, Brussels

    Google Scholar 

  • Dursun O, Türe TE, Daim TU (2011) Post-evaluation of Foresight studies: Turkish case. Int J Foresight Innov Policy 7(4):311–337

    Article  Google Scholar 

  • Eilat H, Golany B, Shtub A (2008) R&D project evaluation: an integrated DEA and balanced scorecard approach. Omega 36:895–912

    Article  Google Scholar 

  • Georghiou L (2003) Evaluating Foresight and lessons for its future impact. Paper presented at the second international conference on technology Foresight, Tokyo, 27–28 February

    Google Scholar 

  • Georghiou L, Keenan M (2006) Evaluation of national Foresight activities: assessing rationale, process and impact. Technol Forecast Soc Change 73(7):761–777

    Article  Google Scholar 

  • Georghiou L, Acheson H, Cassingena Harper J, Clar G, Klusacek K (2004a) Evaluation of the Hungarian Technology Foresight Programme. http://www.nih.gov.hu/english/technology-foresight/evaluation-of-the-080519

  • Georghiou L, Clarysse B, Steurs G, Bilsen V, Larosse J (2004b) Making the difference: the evaluation of behavioural additionality of R&D subsidies. IWT Studies 48:7–20

    Google Scholar 

  • Georghiou L, Keenan M, Miles I, Cameron H (2006) An evaluation of the UK Foresight Programme: final report, University of Manchester

    Google Scholar 

  • German Ministry for Education and Research, BMBF (2002) Futur lead visions complete document, Berlin

    Google Scholar 

  • Giesecke S (2008) Futur – the German research dialogue. In: Giesecke S, Crehan P, Elkins S (eds) The European Foresight Monitoring Network. Collection of EFMN Briefs – Part 1. Office for Official Publications of the European Communities, Luxembourg. http://ec.europa.eu/research/social-sciences/pdf/efmn-report_en.pdf

  • Grun RE (2006) Monitoring and evaluating projects: a step-by-step primer on monitoring, Benchmarking and impact evaluation. The World Bank, Washington, DC

    Google Scholar 

  • Habegger B (2010) Strategic Foresight in public policy: reviewing the experiences of the UK, Singapore, and the Netherlands. Futures 42(1):49–58

    Article  Google Scholar 

  • Havas A, Schartinger D, Weber A (2010) The impact of Foresight on innovation policy-making: recent experiences and future perspectives. Res Eval 19(2):91–104

    Article  Google Scholar 

  • IFAD (2009) Evaluation manual: methodology and processes. http://www.ifad.org/evaluation/process_methodology/doc/manual.pdf

  • JICA (2004) Guideline for project evaluation: practical methods for project evaluation. http://www.jica.go.jp/english/operations/evaluation/tech_and_grant/guides/pdf/guideline01-01.pdf

  • Johnston R (2012) Developing the capacity to assess the impact of Foresight. Foresight 14(1):56–68

    Article  Google Scholar 

  • Kappel TA (2001) Perspectives on roadmaps: how organizations talk about the future. J Product Innov Manage 18:39–50

    Article  Google Scholar 

  • Kováts et al (2000) The Hungarian Technology Foresight Programme, report by the Steering Group, Budapest

    Google Scholar 

  • Meissner D, Cervantes M (2008) Results and impact of national Foresight-studies. Paper presented at the third international Seville seminar on future-oriented technology analysis: impacts and implications for policy and decision-making, Seville, 16–17 October

    Google Scholar 

  • Miles I (with contributions from Mike Keenan) (2003) Ten years of Foresight in the UK. Paper presented at NISTEP’s second international conference on technology Foresight, Tokyo

    Google Scholar 

  • Miles I (2002) Appraisal of alternative methods and procedures for producing regional Foresight, report prepared by CRIC for the European Commission’s DG research funded STRATA – ETAN Expert Group Action, Manchester. http://ec.europa.eu/research/social-sciences/pdf/appraisalof-alternative-methods_en.pdf

  • Miles I (2012) Dynamic Foresight evaluation. Foresight 14(1):69–81

    Article  Google Scholar 

  • Phillips JJ, Bothell TW, Snead GL (2002) The project management scorecard: measuring the success of project management solutions. Butterworth-Heinemann, New York

    Google Scholar 

  • Popper R (2008) Foresight methodology. In: Georghiou L, Cassingena J, Keenan M, Miles I, Popper R (eds) The handbook of technology Foresight. Edward Elgar, Cheltenham

    Google Scholar 

  • Popper R (2009) Mapping Foresight: revealing how Europe and other world regions navigate into the future. Publications Office of the European Union, European Commission, EFMN, Luxembourg

    Google Scholar 

  • Popper R, Georghiou L, Keenan M, Miles I (2010) Evaluating Foresight: fully-fledged evaluation of Colombian Technology Foresight Programme. Universidad del Valle, Santiago de Cali

    Google Scholar 

  • Project Management Institute (PMI) (1996) A guide to the project management body of knowledge (PMBOK Guide). Project Management Institute, Newtown Square

    Google Scholar 

  • Rader M (2003) Hungary – Hungarian Foresight Programme (TEP) 1997/99. FISTERA report WP 1 – review and analysis of national Foresight D1.1 HU – case study

    Google Scholar 

  • Remer DS, Nieto AP (1995a) A compendium and comparison of 25 project evaluation techniques. Part 1: net present value and rate of return methods. Int J Prod Econ 42(1):79–96

    Article  Google Scholar 

  • Remer DS, Nieto AP (1995b) A compendium and comparison of 25 project evaluation techniques. Part 2: ratio, payback and accounting Methods. Int J Prod Econ 42(2):101–129

    Article  Google Scholar 

  • Ricker KT et al (1998) Water quality project evaluation: a handbook for objectives-based evaluation of water quality projects. http://ohioline.osu.edu/b868

  • Rijkens-Klomp N, Duin van der P (2011) Evaluating local public Foresight studies from a user perspective. Paper presented at the 4th international Seville conference on future-oriented technology analysis (FTA), Seville, 12–13 May. http://ftp.jrc.es/EURdoc/JRC64647.pdf

  • Rollwagen I, Hofmann J, Schneider S (2008) Improving the business impact of Foresight. Technol Anal Strateg Manage 20(3):337–349

    Article  Google Scholar 

  • Saritas O, Taymaz E, Tumer T (2007) Vision 2023: Turkey’s national technology Foresight program: a contextualist analysis and discussion. Technol Forecast Soc Change 74:1374–1393

    Article  Google Scholar 

  • Treasury HM (2003) The green book: appraisal and evaluation in central government. TSO, London

    Google Scholar 

  • Westat JF (2002) The 2002 user-friendly handbook for project evaluation, NSF publication. http://www.nsf.gov/pubs/2002/nsf02057/start.htm

  • Yoda T (2011) Perceptions of domain experts on impact of Foresight on policy making: the case of Japan. Technol Forecast Soc Change 78(3):431–447

    Article  Google Scholar 

  • Zarinpoush F (2006) Project evaluation guide for nonprofit organizations: fundamental methods and steps for conducting project evaluation. http://library.imaginecanada.ca/files/nonprofitscan/en/csc/projectguide_final.pdf

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Sokolova, A., Makarova, E. (2013). Integrated Framework for Evaluation of National Foresight Studies. In: Meissner, D., Gokhberg, L., Sokolov, A. (eds) Science, Technology and Innovation Policy for the Future. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31827-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-31827-6_2

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-31826-9

  • Online ISBN: 978-3-642-31827-6

  • eBook Packages: Business and EconomicsEconomics and Finance (R0)

Publish with us

Policies and ethics