Skip to main content
SpringerLink
Log in

Verification Metrics for Hydrological Ensemble Forecasts

  • Reference work entry
  • First Online:
Book cover Handbook of Hydrometeorological Ensemble Forecasting

Abstract

This chapter reviews the most commonly used verification metrics for measuring the performance of hydrological ensemble forecasts. It links metrics to the different attributes of forecast quality and discusses the links between verification variables, metrics, and applications in a broad perspective. It provides an overview of the use of these metrics in forecast evaluation studies and general insights into what forecasters, practitioners, and end-users should consider when applying verification measures in the practice of hydrological ensemble forecasting.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 599.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 799.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • M. Abaza, F. Anctil, V. Fortin, R. Turcotte, A comparison of the Canadian global and regional meteorological ensemble prediction systems for short-term hydrological forecasting. Mon. Weather Rev. 141, 3462–3472 (2013). Corrigendum. Mon. Weather Rev. 142, 2561–2562

    Article  Google Scholar 

  • M. Abaza, F. Anctil, V. Fortin, R. Turcotte, Sequential streamflow assimilation for short-term hydrological ensemble forecasting. J. Hydrol. 519, 2692–2706 (2014)

    Article  Google Scholar 

  • L. Alfieri, F. Pappenberger, F. Wetterhall, T. Haiden, D. Richardson, P. Salamon, Evaluation of ensemble streamflow predictions in Europe. J. Hydrol. 517, 913–922 (2014)

    Article  Google Scholar 

  • C. Alvarez-Garreton, D. Ryu, A.W. Western, C.-H. Su, W.T. Crow, D.E. Robertson, C. Leahy, Improving operational flood ensemble prediction by the assimilation of satellite soil moisture: comparison between lumped and semi-distributed schemes. Hydrol. Earth Syst. Sci. 19, 1659–1676 (2015)

    Article  Google Scholar 

  • F. Anctil, C. Michel, C. Perrin, V. Andréassian, A soil moisture index as an auxiliary ANN input for stream flow forecasting. J. Hydrol. 286, 155–167 (2004)

    Article  Google Scholar 

  • J.L. Anderson, A method for producing and evaluating probabilistic forecasts from ensemble model integrations. J. Clim. 9, 1518–1530 (1996)

    Article  Google Scholar 

  • D. Anghileri, N. Voisin, A.F. Castelletti, F. Pianosi, B. Nijssen, D.P. Lettenmaier, Value of long-term streamflow forecast to reservoir operations for water supply in snow-dominated catchments. Water Resour. Res. 52(6), 4209–4225 (2016). https://doi.org/10.1002/2015WR017864

    Article  Google Scholar 

  • F. Atger, Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Process. Geophys. 8, 401–417 (2001)

    Article  Google Scholar 

  • L. Baringhaus, C. Franz, On a new multivariate two-sample test. J. Multivar. Anal. 88(1), 190–206 (2004)

    Article  Google Scholar 

  • J.C. Bartholmes, J. Thielen, M.H. Ramos, S. Gentilini, The European Flood Alert System EFAS – part 2: statistical skill assessment of probabilistic and deterministic operational forecasts. Hydrol. Earth Syst. Sci. 13(2), 141–153 (2009)

    Article  Google Scholar 

  • M.A. Boucher, J.P. Laliberté, F. Anctil, An experiment on the evolution of an ensemble of neural networks for streamflow forecasting. Hydrol. Earth Syst. Sci. 14, 603–612 (2010)

    Article  Google Scholar 

  • M.-A. Boucher, D. Tremblay, L. Delorme, L. Perreault, F. Anctil, Hydro-economic assessment of hydrological forecasting systems. J. Hydrol. 416, 133–144 (2012). https://doi.org/10.1016/j.jhydrol.2011.11.042

    Article  Google Scholar 

  • F. Bourgin, M.-H. Ramos, G. Thirel, V. Andréassian, Investigating the interactions between data assimilation and post-processing in hydrological ensemble forecasting. J. Hydrol. 519, 2775–2784 (2014)

    Article  Google Scholar 

  • A.A. Bradley, T. Hashino, S.S. Schwartz, Distributions-oriented verification of probability forecasts for small data samples. Weather Forecast. 18, 903–917 (2003)

    Article  Google Scholar 

  • A.A. Bradley, J. Demargne, J.J. Franz, Attributes of forecast quality, in Handbook of Hydrometeorological Ensemble Forecasting, ed. by Q. Duan, F. Pappenberger, J. Thielen, A. Wood, H. Cloke, J. Schaake (Springer, Berlin/Heidelberg, 2016), 46p. https://doi.org/10.1007/978-3-642-40457-3_2-1

    Google Scholar 

  • D. Brochero, F. Anctil, C. Gagné, Simplifying a hydrological ensemble prediction system with a backward greedy selection of members, part I: optimization criteria. Hydrol. Earth Syst. Sci. 15, 3307–3325 (2011)

    Article  Google Scholar 

  • J.D. Brown, J. Demargne, D.J. Seo, Y. Liu, The Ensemble Verification System (EVS): a software tool for verifying ensemble forecasts of hydrometeorological and hydrologic variables at discrete locations. Environ. Model. Softw. 25(7), 854–872 (2010). https://doi.org/10.1016/j.envsoft.2010.01.009

    Article  Google Scholar 

  • J.D. Brown, M. He, S. Regonda, L. Wu, H. Lee, D.-J. Seo, Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 2. Streamflow verification. J. Hydrol. 519, 2847–2868 (2014)

    Article  Google Scholar 

  • T.M. Carpenter, K.P. Georgakakos, Impacts of parametric and radar rainfall uncertainty on the ensemble streamflow simulations of a distributed hydrologic model. J. Hydrol. 298, 202–221 (2004)

    Article  Google Scholar 

  • B. Casati, L.J. Wilson, D.B. Stephenson, Forecast verification: current status and future directions. Meteorol. Appl. 15(1), 3–18 (2008)

    Article  Google Scholar 

  • M.P. Clark, A.G. Slater, Probabilistic quantitative precipitation estimation in complex terrain. J. Hydrometeorol. 7, 3–22 (2006)

    Article  Google Scholar 

  • H. Cloke, F. Pappenberger, Evaluating forecasts of extreme events for hydrological applications: an approach for screening unfamiliar performance measures. Meteorol. Appl. 15, 181–197 (2008)

    Article  Google Scholar 

  • C. Corradini, F. Melone, L. Ubertini, A semi-distributed adaptive model for real-time flood forecasting. Water Resour. Bull. 22, 1031–1038 (1986)

    Article  Google Scholar 

  • L. Crochemore, M.-H. Ramos, F. Pappenberger, Bias correcting precipitation forecasts to improve the skill of seasonal streamflow forecasts. Hydrol. Earth Syst. Sci. 20, 3601–3618 (2016). https://doi.org/10.5194/hess-20-3601-2016

    Article  Google Scholar 

  • J. Demargne, J.D. Brown, Y. Liu, D.-J. Seo, L. Wu, Z. Toth, Y. Zhu, Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11(2), 114–122 (2010)

    Article  Google Scholar 

  • D. Demeritt, S. Nobert, H.L. Cloke, F. Pappenberger, The European Flood Alert System and the communication, perception, and use of ensemble predictions for operational flood risk management. Hydrol. Process. 27, 147–157 (2013). https://doi.org/10.1002/hyp.9419

    Article  Google Scholar 

  • K. Engeland, I. Steinsland, Probabilistic postprocessing models for flow forecasts for a system of catchments and several lead times. Water Resour. Res. 50, 182–197 (2014). https://doi.org/10.1002/2012WR012757

    Article  Google Scholar 

  • F.M. Fan, W. Collischonn, A. Meller, L.C.M. Botelho, Ensemble streamflow forecasting experiments in a tropical basin: the São Francisco River case study. J. Hydrol. 519, 2906–2919 (2014)

    Article  Google Scholar 

  • V. Fortin, M. Abaza, A. Anctil, R. Turcotte, Why should ensemble spread match the RMSE of the ensemble mean? J. Hydrometeorol. 15, 1708–1713 (2014)

    Article  Google Scholar 

  • K.J. Franz, T.S. Hogue, M. Barik, Assessment of SWE data assimilation for ensemble streamflow predictions. J. Hydrol. 519(Part D), 2737–2746 (2014)

    Article  Google Scholar 

  • T. Gneiting, A.E. Raftery, A.H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev. 133, 1098–1118 (2005). https://doi.org/10.1175/MWR2904.1

    Article  Google Scholar 

  • T. Gneiting, A.E. Raftery, Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102, 359–378 (2007)

    Article  Google Scholar 

  • T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat. Methodol. 69, 243–268 (2007)

    Article  Google Scholar 

  • I.J. Good, Rational decisions. J. R. Stat. Soc. 14, 107–114 (1952)

    Google Scholar 

  • T. Hamill, Interpretation of rank histograms for verifying ensemble forecasts. Mon. Weather Rev. 129, 550–560 (2001)

    Article  Google Scholar 

  • T.M. Hamill, S.J. Colucci, Verification of Eta RSM short-range ensemble forecasts. Mon. Weather Rev. 125, 1312–1327 (1997)

    Article  Google Scholar 

  • T. Hashino, A.A. Bradley, S.S. Schwartz, Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol. Earth Syst. Sci. 11, 939–950 (2007)

    Article  Google Scholar 

  • H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast. 15, 559–570 (2000)

    Article  Google Scholar 

  • H. Holt, J. Pullen, C. Bishop, Urban and ocean ensembles for improved meteorological and dispersion modeling of the coastal zone. Tellus 61A, 232–249 (2009)

    Article  Google Scholar 

  • I.T. Jolliffe, D.B. Stephenson, Forecast Verification: A practitioner’s Guide in Atmospheric Science, 2nd edn. (Wiley, New York, 2012). https://doi.org/10.1002/9781119960003

    Book  Google Scholar 

  • Y.-O. Kim, H. Eum, E.G. Lee, I.H. Ko, Optimizing operational policies of a Korean multireservoir system using sampling stochastic dynamic programming with ensemble streamflow prediction. J. Water Resour. Plan. Manag. 133, 4–14 (2007). https://doi.org/10.1061/(ASCE)0733-9496(2007)133:1(4)

    Article  Google Scholar 

  • P.K. Kitanidis, R.L. Bras, Real-time forecasting with a conceptual hydrologic model. 2. Applications and results. Water Resour. Res. 16(6), 1034–1044 (1980)

    Article  Google Scholar 

  • F. Laio, S. Tamea, Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol. Earth Syst. Sci. 11(4), 1267–1277 (2007)

    Article  Google Scholar 

  • K. Liechti, M. Zappa, F. Fundel, U. Germann, The potential of radar-based ensemble forecasts for flash-flood early warning in the southern Swiss Alps. Hydrol. Earth Syst. Sci. 17, 3853–3869 (2013)

    Article  Google Scholar 

  • Y. Liu, J.D. Brown, J. Demargne, D.-J. Seo, A wavelet-based approach to assessing timing errors in hydrologic predictions. J. Hydrol. 397(3–4), 210–224 (2011)

    Article  Google Scholar 

  • S.J. Mason, A model for assessment of weather forecast. Aust. Meteorol. Mag. 30, 291–303 (1982)

    Google Scholar 

  • S. Matte, M.-A. Boucher, V. Boucher, T.-C. Fortier Filion, Moving beyond the cost–loss ratio: economic assessment of streamflow forecasts for a risk-averse decision maker. Hydrol. Earth Syst. Sci. 21, 2967–2986 (2017)

    Article  Google Scholar 

  • D.N. Moriasi, J.G. Arnold, M.W. Van Liew, R.L. Bingner, R.D. Harmel, T.L. Veith, Model evaluation guidelines for systematic quantification of accuracy in watershed simulations. Trans. ASABE 50, 885–900 (2007)

    Article  Google Scholar 

  • A.H. Murphy, A new vector partition of the probability score. J. Appl. Meteorol. 12(4), 595–600 (1973)

    Article  Google Scholar 

  • A. Murphy, What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather Forecast. 8(2), 281–293 (1993)

    Article  Google Scholar 

  • T. Palmer, R. Buizza, R. Hagedorn, A. Lawrence, M. Leutbecher, L. Smith, Ensemble prediction: a pedagogical perspective. ECMWF Newsl. 106, 10–17 (2005). ECMWF, Reading

    Google Scholar 

  • F. Pappenberger, K. Scipal, R. Buizza, Hydrological aspects of meteorological verification. Atmos. Sci. Lett. 9, 43–52 (2008)

    Article  Google Scholar 

  • F. Pappenberger, M.H. Ramos, H.L. Cloke, F. Wetterhall, L. Alfieri, K. Bogner, A. Mueller, P. Salamon, How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction. J. Hydrol. 522, 697–713 (2015)

    Article  Google Scholar 

  • W.W. Peterson, T.G. Birdsall, W.C. Fox, The theory of signal detectability. Trans. IRE Prof. Group Inf. Theory 2–4, 171–212 (1954)

    Article  Google Scholar 

  • M.H. Ramos, J. Bartholmes, J. Thielen-del Pozo, Development of decision support products based on ensemble forecasts in the European Flood Alert System. Atmos. Sci. Lett. 8, 113–119 (2007). https://doi.org/10.1002/asl.161

    Article  Google Scholar 

  • M.H. Ramos, T. Mathevet, J. Thielen, F. Pappenberger, Communicating uncertainty in hydro-meteorological forecasts: mission impossible? Meteorol. Appl. 17, 223–235 (2010)

    Article  Google Scholar 

  • A. Randrianasolo, M.H. Ramos, G. Thirel, V. Andreassian, E. Martin, Comparing the scores of hydrological ensemble forecasts issued by two different hydrological models. Atmos. Sci. Lett. 11(2), 100–107 (2010)

    Article  Google Scholar 

  • B. Renard, D. Kavetski, G. Kuczera, M. Thyer, S.W. Franks, Understanding predictive uncertainty in hydrologic modeling: the challenge of identifying input and structural errors. Water Resour. Res. 46, W05521 (2010)

    Article  Google Scholar 

  • D.S. Richardson, Skill and relative economic value of ECMWF ensemble prediction system. Q. J. R. Meteorol. Soc. 126, 649–667 (2000)

    Article  Google Scholar 

  • E. Roulin, Skill and relative economic value of medium-range hydrological ensemble predictions. Hydrol. Earth Syst. Sci. 11, 725–737 (2007)

    Article  Google Scholar 

  • E. Roulin, S. Vannitsem, Post-processing of medium-range probabilistic hydrological forecasting: impact of forcing, initial conditions and model errors. Hydrol. Process. 29(6), 1434–1449 (2015)

    Article  Google Scholar 

  • M.S. Roulston, L.A. Smith, Evaluating probabilistic forecasts using information theory. Mon. Weather Rev. 130, 1653–1660 (2002)

    Article  Google Scholar 

  • M.S. Roulston, L.A. Smith, Combining dynamical and statistical ensembles. Tellus A 55, 16–30 (2003). https://doi.org/10.1034/j.1600-0870.2003.201378.x

    Article  Google Scholar 

  • D.L. Shrestha, D.E. Robertson, J.C. Bennett, Q.J. Wang, Improving precipitation forecasts by generating ensembles through postprocessing. Mon. Weather Rev. 143, 3642–3663 (2015)

    Article  Google Scholar 

  • G. Székely, M. Rizzo, A new test for multivariate normality. J. Multivar. Anal. 1(93), 58–80 (2005)

    Article  Google Scholar 

  • O. Talagrand, R. Vautard, B. Strauss, Evaluation of probabilistic prediction systems, in Workshop on Predictability, ed. by for Medium-Range Weather Forecasts, E. C., Shinfield Park, Reading (1997), pp. 1–25

    Google Scholar 

  • A. Thiboult, F. Anctil, M.-A. Boucher, Accounting for three sources of uncertainty in ensemble hydrological forecasting. Hydrol. Earth Syst. Sci. 20, 1809–1825 (2016)

    Article  Google Scholar 

  • A. Thiboult, F. Anctil, M.H. Ramos, How does the quantification of uncertainties affect the quality and value of flood early warning systems? J. Hydrol. 551, 365–373 (2017). https://doi.org/10.1016/j.jhydrol.2017.05.014

    Article  Google Scholar 

  • P. Trambauer, M. Werner, H.C. Winsemius, S. Maskey, E. Dutra, S. Uhlenbrook, Hydrological drought forecasting and skill assessment for the Limpopo River basin, southern Africa. Hydrol. Earth Syst. Sci. 19, 1695–1711 (2015). https://doi.org/10.5194/hess-19-1695-2015

    Article  Google Scholar 

  • J. Van den Bergh, E. Roulin, Hydrological ensemble prediction and verification for the Meuse and Scheldt basins. Atmos. Sci. Lett. 11, 64–71 (2010). https://doi.org/10.1002/asl.250

    Article  Google Scholar 

  • J.-A. Velázquez, F. Anctil, C. Perrin, Performance and reliability of multimodel hydrological ensemble simulations based on seventeen lumped models and a thousand catchments. Hydrol. Earth Syst. Sci. 14, 2303–2317 (2010)

    Article  Google Scholar 

  • J.S. Verkade, M.G.F. Werner, Estimating the benefits of single value and probability forecasting for flood warning. Hydrol. Earth Syst. Sci. 15, 3751–3765 (2011)

    Article  Google Scholar 

  • J.S. Verkade, J.D. Brown, P. Reggiani, A.H. Weerts, Post-processing ECMWF precipitation and temperature ensemble reforecasts for operational hydrologic forecasting at various spatial scales. J. Hydrol. 501, 73–91 (2013)

    Article  Google Scholar 

  • N. Voisin, F. Pappenberger, D.P. Lettenmaier, R. Buizza, J.C. Schaake, Application of a medium-range global hydrologic probabilistic forecast scheme to the Ohio River Basin. Weather Forecast. 26, 425–446 (2011)

    Article  Google Scholar 

  • A.S. Weigend, S. Shi, Predicting daily probability distributions of S&P500 returns. J. Forecast. 19, 375–392 (2000)

    Article  Google Scholar 

  • S.V. Weijs, R. van Nooijen, N. van de Giesen, Kullback–Leibler divergence as a forecast skill score with classic reliability–resolution–uncertainty decomposition. Mon. Weather Rev. 138, 3387–3399 (2010)

    Article  Google Scholar 

  • K. Werner, J.S. Verkade, T.C. Pagano, Application of hydrological forecast verification information, in Handbook of Hydrometeorological Ensemble Forecasting, ed. by Q. Duan, F. Pappenberger, J. Thielen, A. Wood, H. Cloke, J. Schaake (Springer, Berlin/Heidelberg, 2016), 22p. https://doi.org/10.1007/978-3-642-40457-3_7-1

    Google Scholar 

  • D.S. Wilks, Statistical Methods in the Atmospheric Sciences: An Introduction (Academic, 2011), Amsterdam, 676p

    Google Scholar 

  • A.W. Wood, A. Kumar, D.P. Lettenmaier, A retrospective assessment of National Centers for Environmental Prediction climate model–based ensemble hydrologic forecasting in the western United States. J. Geophys. Res. Atmos. 110, D04105 (2005). https://doi.org/10.1029/2004JD004508

    Article  Google Scholar 

  • X. Yuan, J. Roundy, E. Wood, J. Sheffield, Seasonal forecasting of global hydrologic extremes: system development and evaluation over GEWEX basins. Bull. Am. Meteorol. Soc., 1895–1912 (2015). https://doi.org/10.1175/BAMS-D-14-00003.1

    Article  Google Scholar 

  • I. Zalachori, M.H. Ramos, R. Garçon, T. Mathevet, J. Gailhard, Statistical processing of forecasts for hydrological ensemble prediction: a comparative study of different bias correction strategies. Adv. Sci. Res. 8, 135–141 (2012)

    Article  Google Scholar 

  • M. Zappa, F. Fundel, S. Jaun, A ‘Peak-Box’ approach for supporting interpretation and verification of operational ensemble peak-flow forecasts. Hydrol. Process. 27(1), 117–131 (2013). https://doi.org/10.1002/hyp.9521

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Département de génie civil et de génie des eaux, Université Laval, Québec, QC, Canada

    François Anctil

  2. IRSTEA, National Research Institute of Science and Technology for Environment and Agriculture, UR HBAN, Antony, France

    Maria-Helena Ramos

Authors
  1. François Anctil
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria-Helena Ramos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to François Anctil .

Editor information

Editors and Affiliations

  1. Faculty of Geographical Science, Beijing Normal University, Beijing, China

    Qingyun Duan

  2. European Centre for Medium-Range Weather Forecasts, ECMWF, Reading, UK

    Florian Pappenberger

  3. National Center for Atmospheric Research, Boulder, CO, USA

    Andy Wood

  4. Department of Meteorology, Reading University, Reading, UK

    Hannah L. Cloke

  5. U.S. National Weather Service (retired), Annapolis, MD, USA

    John C. Schaake

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer-Verlag GmbH Germany, part of Springer Nature

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Anctil, F., Ramos, MH. (2019). Verification Metrics for Hydrological Ensemble Forecasts. In: Duan, Q., Pappenberger, F., Wood, A., Cloke, H., Schaake, J. (eds) Handbook of Hydrometeorological Ensemble Forecasting. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39925-1_3

Download citation

Publish with us

Policies and ethics

Search

Navigation