Abstract
Forecast verification is a process used to assess the quality of hydrometeorological ensemble forecasts. This chapter describes the many aspects of forecast quality using a distributions-oriented approach. Using the joint distribution of forecasts and observations, or one of its factorizations into a conditional and marginal distribution, the aspects of forecast quality are defined. Hypothetical ensemble forecasts are then used to illustrate aspects of forecast quality. The hypothetical ensemble forecasts are used to construct single-valued forecasts, probability forecasts for an event, and ensemble probability distribution forecasts. Their forecast quality is then diagnosed using visual comparisons and numerical comparisons of forecast quality measures. The examples illustrate that a single aspect of forecast quality is insufficient and that many aspects are needed to understand the nature of the forecasts. Some practical considerations in the application of the framework to ensemble forecast verification are discussed.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids. Weather Forecast. 18, 918–932 (2003)
C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, A. Speranza, Verification of precipitation forecasts from two limited-area models over Italy and comparison with ECMWF forecasts using a resampling technique. Weather Forecast. 20, 276–300 (2005)
F. Atger, Spatial and interannual variability of the reliability of ensemble-based probabilistic forecasts: Consequences for calibration. Monthly Weather Review. 131(8), 1509–1523 (2003)
A.A. Bradley, S.S. Schwartz, Summary verification measures and their interpretation for ensemble forecasts. Mon. Weather Rev. 139, 3075–3089 (2011)
A.A. Bradley, T. Hashino, S.S. Schwartz, Distributions-oriented verification of probability forecasts for small data samples. Weather Forecast. 18, 903–917 (2003)
A.A. Bradley, S.S. Schwartz, T. Hashino, Distributions-oriented verification of ensemble streamflow predictions. J. Hydrometeorol. 5, 532–545 (2004)
A.A. Bradley, S.S. Schwartz, T. Hashino, Sampling uncertainty and confidence intervals for the Brier score and Brier skill score. Weather Forecast. 23, 992–1006 (2008)
G.W. Brier, Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78, 1–3 (1950)
G.W. Brier, R.A. Allen, Verification of weather forecasts, in Compendium of Meteorology, ed. by T.F. Malone (American Meteorological Society, Boston, 1951), pp. 841–848
J.D. Brown, D.J. Seo, A nonparametric postprocessor for bias correction of hydrometeorological and hydrologic ensemble forecasts. J. Hydrometeorol. 11, 642–665 (2010)
J.D. Brown, L. Wu, M. He, S. Regonda, H. Lee, D.-J. Seo, Verification of temperature, precipitation, and streamflow forecasts from the NOAA/NWS Hydrologic Ensemble Forecast Service (HEFS): 1. Experimental design and forcing verification. J. Hydrol. 519, 2869–2889 (2014)
J. Demargne, J. Brown, Y.Q. Liu, D.J. Seo, L.M. Wu, Z. Toth, Y.J. Zhu, Diagnostic verification of hydrometeorological and hydrologic ensembles. Atmos. Sci. Lett. 11, 114–122 (2010)
F.J. Doblas-Reyes, V. Pavan, D.B. Stephenson, The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation. Climate Dynam. 21, 501–514 (2003)
E.E. Ebert, L.J. Wilson, B.G. Brown, P. Nurmi, H.E. Brooks, J. Bally, M. Jaeneke, Verification of nowcasts from the WWRP Sydney 2000 forecast demonstration project. Weather Forecast. 19, 73–96 (2004)
B. Efron, Nonparametric estimates of standard error: the jacknife, the bootstrap and other methods. Biometrika 68, 589–599 (1981)
C.A. Ferro, Comparing probabilistic forecasting systems with the Brier score. Weather Forecast. 22, 1076–1088 (2007)
K.J. Franz, H.C. Hartmann, S. Sorooshian, R. Bales, Verification of national weather service ensemble streamflow predictions for water supply forecasting in the Colorado River basin. J. Hydrometeorol. 4, 1105–1118 (2003)
T. Gneiting, A.E. Raftery, A.H. Westveld, T. Goldman, Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon. Weather Rev. 133, 1098–1118 (2005)
T. Gneiting, F. Balabdaoui, A.E. Raftery, Probabilistic forecasts, calibration and sharpness. J. R. Stat. Soc. Ser. B Stat Methodol. 69, 243–268 (2007)
T. Gorgas and M. Dorninger, Quantifying verification uncertainty by reference data variation. Meteorologishe Zeitschrift. 21(3), 259–277 (2012)
T.M. Hamill, Hypothesis tests for evaluating numerical precipitation forecasts. Weather Forecast. 14, 155–167 (1999)
T.M. Hamill, J. Juras, Measuring forecast skill: is it real skill or is it the varying climatology? Q. J. Roy. Meteorol. Soc. 132, 2905–2923 (2006)
T. Hashino, A.A. Bradley, S.S. Schwartz, Evaluation of bias-correction methods for ensemble streamflow volume forecasts. Hydrol. Earth Syst. Sci. 11, 939–950 (2007)
H. Hersbach, Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast. 15, 559–570 (2000)
S. Jaun, B. Ahrens, Evaluation of a probabilistic hydrometeorological forecast system. Hydrol. Earth Syst. Sci. 13, 1031–1043 (2009)
I. Jolliffe, Uncertainty and inference for verification measures. Weather Forecast. 22, 637–650 (2007)
I.T. Jolliffe, D.B. Stephenson, Introduction, in Forecast Verification, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 1–9
T.L. Kane, B.G. Brown, Confidence intervals for some verification measures – a survey of several methods, in Preprints, 15th Conference on Probability and Statistics in the Atmospheric Sciences (American Meteorologic Society, Asheville, 2000)
R. Krzysztofowicz, The case for probabilistic forecasting in hydrology. J. Hydrol. 249, 2–9 (2001)
S.J. Mason, N.E. Graham, Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: statistical significance and interpretation. Q. J. Roy. Meteorol. Soc. 128, 2145–2166 (2002)
S.J. Mason, G.M. Mimmack, The use of bootstrap confidence intervals for the correlation coefficient in climatology. Theor. Appl. Climatol. 45, 229–233 (1992)
J.E. Matheson, R.E. Winkler, Scoring rules for continuous probability distributions. Manag. Sci. 22, 1087–1095 (1976)
A.H. Murphy, Skill scores based on the mean-square error and their relationships to the correlation-coefficient. Mon. Weather Rev. 116, 2417–2425 (1988)
A.H. Murphy, What is a good forecast – an essay on the nature of goodness in weather forecasting. Weather Forecast. 8, 281–293 (1993)
A.H. Murphy, Forecast verification, in Economic Value of Weather and Climate Forecasts, ed. by R.W. Katz, A.H. Murphy (Cambridge University Press, Cambridge, 1997), pp. 19–74
A.H. Murphy, R.L. Winkler, A general framework for forecast verification. Mon. Weather Rev. 115, 1330–1338 (1987)
A.H. Murphy, R.L. Winkler, Diagnostic verification of probability forecasts. Int. J. Forecast. 7, 435–455 (1992)
D.N. Politis, J.P. Romano, The stationary bootstrap. J. Am. Stat. Assoc. 89, 1303–1313 (1994)
J.M. Potts, Basic concepts, in Forecast Verification, ed. by I.T. Jolliffe, D.B. Stephenson (Wiley, Chichester, 2011), pp. 11–29
D.J. Seo, H.D. Herr, J.C. Schaake, A statistical post-processor for accounting of hydrologic uncertainty in short-range ensemble streamflow prediction. Hydrol. Earth Syst. Sci. Discuss. 3, 1987–2035 (2006)
D.B. Stephenson, Use of the “odds ratio” for diagnosing forecast skill. Weather Forecast. 15, 221–232 (2000)
J.E. Thornes, D.B. Stephenson, How to judge the quality and value of weather forecast products. Meteorol. Appl. 8, 307–314 (2001)
D.A. Unger, A method to estimate the continuous ranked probability score, in Nineth Conference on Probability and Statistics in Atmospheric Sciences (American Meteorological Society, Virginia Beach, 1985), pp. 206–213
E. Welles, S. Sorooshian, G. Carter, B. Olsen, Hydrologic verification – a call for action and collaboration. Bull. Am. Meteorol. Soc. 88, 503 (2007)
D.S. Wilks, Statistical significance of long-range “optimal climate normal” temperature and precipitation forecasts. J. Climate 9, 827–839 (1996)
D.S. Wilks, Sampling distributions of the Brier score and Brier skill score under serial dependence. Q. J. Roy. Meteorol. Soc. 136, 2109–2118 (2010)
D.S. Wilks, Statistical Methods in the Atmospheric Sciences, 3rd edn. (Academic, Amsterdam, 2011), p. 704
H. Zhang, T. Casey, Verification of categorical probability forecasts. Weather Forecast. 15, 80–89 (2000)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer-Verlag GmbH Germany, part of Springer Nature
About this entry
Cite this entry
Bradley, A.A., Demargne, J., Franz, K.J. (2019). Attributes of Forecast Quality. In: Duan, Q., Pappenberger, F., Wood, A., Cloke, H., Schaake, J. (eds) Handbook of Hydrometeorological Ensemble Forecasting. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-39925-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-642-39925-1_2
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-39924-4
Online ISBN: 978-3-642-39925-1
eBook Packages: Earth and Environmental ScienceReference Module Physical and Materials ScienceReference Module Earth and Environmental Sciences