Skip to main content
SpringerLink
Log in

Evaluating Integrated Assessment Tools for Policy Support

  • Chapter
  • First Online:
Environmental and Agricultural Modelling

Abstract

Integrated Assessment Modelling tools are complex tools requiring specific evaluation methodologies. Based on the example of the SEAMLESS-Integrated framework, we show how the conceptual, technical and system evaluation steps of the different components (procedures, quantitative models, graphic user interfaces) were performed by a multidisciplinary team. To make the not-yet-available tool real, mock-up and test cases were mobilized throughout the development process in order to integrate final end-users in the evaluation process. The main lessons from the project are that the evaluation required: (i) the use of prototypes to advance properly in the design and testing (spiral methodology); (ii) the use of case studies to stick to the end-users requirements; (iii) a proper timing of development and delivery in order to keep on schedule and leave time to the evaluation process; (iv) a multidisciplinary team of evaluators as tools are of diverse types; and (v) that it is difficult to keep independence between testers, end-users and modellers in order to guaranty transparency in the development and evaluation process.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Adelman, L. (1992). Evaluation decision support and expert systems. New York: Wiley.

    Google Scholar 

  • Béguin, P., & Cerf, M. (2004). Formes et enjeux de l’analyse de l’activité pour la conception de systèmes de travail. @ctivités, 1, 54-71.

    Google Scholar 

  • Belhouchette, H., Therond, O., Adenaeuer, M., Kuiper, M., Bigot, G., AlkanOlsson, J., Wery, J., et al. (2007). Documentation of baseline and policy options to be assessed with Prototypes 2 and 3, D6.2.3.3. SEAMLESS Integrated Project, EU 6th Framework Programme, contract no. 010036-2, www.SEAMLESS-IP.org, 112 p.

  • Belhouchette, H., Wery, J., Therond, O., Bergez, J.E., & van Ittersum, M. (2006a). Design of scenarios for integrated impact assessment of interactions between EU policies and agro-ecological technologies using SEAMLESS-IF. Proceedings of the IX ESA Congress (Pt. III), Warsaw (Poland), pp. 11-20.

    Google Scholar 

  • Belhouchette, J., Wery, J., Therond, O., Duru, M., Bigot, G., et al. (2006b). The major characteristics of environmental policies and agro-ecological technologies to be studied in Test case 2. SEAMLESS Rep. No.13, SEAMLESS Integrated Project, EU 6th Framework Programme, contract no. 010036-2, www.SEAMLESS-IP.org, 87 p.

  • Boehm, B. W. (1981). Software engineering economics. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Boehm, B.W. (1988). A spiral model of software development and enhancement. Computer, May, 61-72.

    Google Scholar 

  • Börjeson, L., Hojer, M., Dreborg, K. H., Ekvall, T., & Finnveden, G. (2006). Scenario types and techniques: Towards a user’s guide. Futures, 38, 723-739.

    Article  Google Scholar 

  • Brunner, N., & Starkl, M. (2004). Decision aid systems for evaluating sustainability: A critical survey. Environmental Impact Assessment Review, 24, 441-469.

    Article  Google Scholar 

  • Darses, F. (2002). A cognitive analysis of collective decision-making in the participatory design process. Participatory Design Conference, Malmö, Sweden.

    Google Scholar 

  • Darses, F., Détienne, F., & Visser, W. (2004). Les activités de conception et leur assistance. In P. Falzon (Ed.), Ergonomie (pp. 545-563). Paris: PUF.

    Google Scholar 

  • Dieste, O., Genero, M., Juristo, N., Mate, J. L., & Moreno, A. M. (2003). A conceptual model completely independent of the implementation paradigm. Journal of Systems and Software, 68, 183-198.

    Article  Google Scholar 

  • Finlay, P. N., & Wilson, J. M. (1991). Validation for decision support systems: Recent developments and findings. Systems Practice, 4, 599-610.

    Article  Google Scholar 

  • Gabbert, S., Van Ittersum, M., Ewert, F., Kroeze, C., Stalpers, S. & Alkan-Olsson, J., (2009). Uncertainty information in Integrated Assessment: The users’ perspective. Regional Environmental Change, in press.

    Google Scholar 

  • Harris, G. (2002). Integrated assessment and modelling: An essential way of doing science. Environmental Modelling and Software, 17, 201-207.

    Article  Google Scholar 

  • Jakeman, A. J., Letcher, R. A., & Norton, J. P. (2006). Ten iterative steps in development and evaluation of environmental models. Environmental Modelling and Software, 21, 602-614.

    Article  Google Scholar 

  • Langvad, A. M., & Noe, E. (2006). (Re-)innovating tools for decision-support in the light of farmers’ various strategies. In H. Langeveld & N. Röling (Eds.), Changing European farming systems for a better future - new visions for rural areas (pp. 335-339). The Netherlands: Wageningen.

    Google Scholar 

  • Lopez, M. (2003). Application of an evaluation framework for analyzing the architecture tradeoff analysis methodSM. Journal of Systems and Software, 68, 233-241.

    Article  Google Scholar 

  • Louhichi, K., Belhouchette, H., Flichman, G., Therond, O., & Wery, J. (2008). Application of FSSIM in two test case regions to assess agro-environmental policies at farm and regional level, PD6.3.2.2. SEAMLESS Integrated Project, EU 6th Framework Programme, contract no. 010036-2, www.SEAMLESS-IP.org, 67 p.

  • Mosqueira-Rey, E., & Moret-Bonillo, V. (2000). Validation of intelligent systems: A critical study and a tool. Expert Systems with Applications, 18, 1-16.

    Article  Google Scholar 

  • Oreskes, N., Shrader-Frechette, K., & Belitz, K. (1994). Verification, validation, and confirmation of numerical models in the earth sciences. Science, 263, 641-646.

    Article  PubMed  CAS  Google Scholar 

  • Parker, P., Letcher, R. A., Jakeman, A. J., Beck, M. B., Harris, G., Argent, R. M., et al. (2002). Progress in integrated assessment and modelling. Environmental Modelling and Software, 17(3), 209-217.

    Article  Google Scholar 

  • Pastre, P. (2005). Apprendre par la simulation: de l’analyse du travail aux apprentissages professionnels. Octares, Collection Formation, 363 p.

    Google Scholar 

  • Rotmans, J. (1998). Methods for IA: The challenges and opportunities ahead. Environmental Modeling and Assessment, 3, 155-179.

    Article  Google Scholar 

  • Rykiel, E. J., Jr. (1996). Testing ecological models: The meaning of validation. Ecological Modelling, 90, 229-244.

    Article  Google Scholar 

  • SEC (2005). Impact Assessment Guidelines (791), from http://ec.europa.eu/governance/impact/docs/SEC2005_791_IA_guidelines_main.pdf

  • Sharma, M., & Norton, B. G. (2005). A policy decision tool for integrated environmental assessment. Environmental Science and Policy, 8, 356-366.

    Article  Google Scholar 

  • Sinclair, T. R., & Seligman, N. G. (2000). Criteria for publishing papers on crop modeling. Field Crops Research, 68, 165-172.

    Article  Google Scholar 

  • Sojda, R. S. (2007). Empirical evaluation of decision support systems: Needs, definitions, potential methods, and an example pertaining to waterfowl management. Environmental Modelling and Software, 22, 269-277.

    Article  Google Scholar 

  • Thérond, O., Belhouchette, O., Janssen, S., Louhichi, K., Ewert, F., Bergez, J.E., Wery, J., Heckelei, T., Alkan Olsson, J., Leenhardt, D., & van Ittersum, M. (2009). Methodology to translate policy assessment problems into scenarios: The example of the SEAMLESS Integrated Framework. Environmental Science and Policy, 12(5), 619-630.

    Google Scholar 

  • Tol, R. S. J., & Vellinga, P. (1998). The European Forum on Integrated Environmental Assessment. Environmental Modelling and Assessment, 3, 181-191.

    Article  Google Scholar 

  • Toth, F. L. (2003). State of the art and future challenges for integrated environmental assessment. Integrated Assessment, 4, 250-264.

    Article  Google Scholar 

  • van Ittersum, M., Ewert, F., Heckelei, T., Wery, J., Alkan Olsson, J., Andersen, E., et al. (2008). Integrated assessment of agricultural systems - a component-based framework for the European Union (SEAMLESS). Agricultural Systems, 96, 150-165.

    Article  Google Scholar 

  • van Notten, P. W. F., Rotmans, J., van Asselt, M. B. A., & Rothman, D. S. (2003). An updated scenario typology. Futures, 35, 423-443.

    Article  Google Scholar 

Download references

Acknowledgements

The authors wish to thank all the participants of SEAMLESS-IF involved in testing as well as the students that have worked on the testing specific components of SEAMLESS-IF.

Author information

Authors and Affiliations

  1. INRA, UMR 1248 AGIR, BP 52627, F-31326, Castanet Tolosan, France

    Jacques-Eric Bergez & Olivier Thérond

  2. LEI Wageningen UR, 29703, 2502, LS, The Hague, The Netherlands

    Marijke Kuiper

  3. Dynamics and Functions of Rural Areas, Cemagref, 24 avenue des Landais, BP 50085, 63172, Aubière Cedex, France

    Marie Taverne

  4. INRA - UMR SYSTEM (Systèmes de Culture Tropicaux et Méditerranéens), Montpellier, France

    Hatem Belhouchette & Jacques Wery

Authors
  1. Jacques-Eric Bergez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marijke Kuiper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Olivier Thérond
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marie Taverne
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hatem Belhouchette
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jacques Wery
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jacques-Eric Bergez .

Editor information

Editors and Affiliations

  1. Research Institute, Agricultural Economics, s-Gravenhage, Netherlands

    Floor M. Brouwer

  2. Dept. Plant Sciences, Wageningen University, Wageningen, Netherlands

    Martin K. Ittersum

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer Science+Business Media B.V.

About this chapter

Cite this chapter

Bergez, JE., Kuiper, M., Thérond, O., Taverne, M., Belhouchette, H., Wery, J. (2010). Evaluating Integrated Assessment Tools for Policy Support. In: Brouwer, F., Ittersum, M. (eds) Environmental and Agricultural Modelling. Springer, Dordrecht. https://doi.org/10.1007/978-90-481-3619-3_10

Download citation

Publish with us

Policies and ethics

Search

Navigation