Advertisement

Verification of Meteorological Forecasts for Hydrological Applications

  • Eric GillelandEmail author
  • Florian Pappenberger
  • Barbara Brown
  • Elizabeth Ebert
  • David Richardson
Reference work entry

Abstract

This chapter illustrates how verification is conducted with operational meteorological ensemble forecasts. It focuses on the main aspects of importance to hydrological applications, such as verification of point and spatial precipitation forecasts, verification of temperature forecasts, verification of extreme meteorological events, and feature-based verification.

Keywords

Forecast verification Model evaluation Spatial statistics Spatial forecast verification Extreme-value verification 

References

  1. C. Accadia, S. Mariani, M. Casaioli, A. Lavagnini, Sensitivity of precipitation forecast skill scores to bilinear interpolation and a simple nearest-neighbor average method on high-resolution verification grids. Weather Forecast. 18, 918–932 (2003)CrossRefGoogle Scholar
  2. A. AghaKouchak, N. Nasrollahi, J. Li, B. Imam, S. Sorooshian, Geometrical characterization of precipitation patterns. J. Hydrometeorol. 12(2), 274–285 (2010).  https://doi.org/10.1175/2010JHM1298.1CrossRefGoogle Scholar
  3. G.D. Alexander, J.A. Weinman, J.L. Schols, The use of digital warping of microwave integrated vapor imagery to improve forecasts of marine extratropical cyclones. Mon. Weather Rev. 126, 1469–1496 (1998)CrossRefGoogle Scholar
  4. L. Alfieri, D. Velasco, J. Thielen, Flash flood detection through a multi-stage probabilistic warning system for heavy precipitation events. Adv. Geosci. 29, 69–75 (2011).  https://doi.org/10.5194/adgeo-29-69-2011. www.adv-geosci.net/29/69/2011/CrossRefGoogle Scholar
  5. L. Alfieri, P. Burek, E. Dutra, B. Krzeminski, D. Muraro, J. Thielen, F. Pappenberger, GloFAS – global ensemble streamflow forecasting and flood early warning. Hydrol. Earth Syst. Sci. 13, 141–153 (2012)Google Scholar
  6. T. Anderson, D. Darling, Asymptotic theory of certain goodness of fit criteria based on stochastic processes. Ann. Math. Stat. 23, 193–212 (1952)CrossRefGoogle Scholar
  7. F. Atger, Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlinear Processes Geophys. 8, 401–417 (2001)CrossRefGoogle Scholar
  8. J.C. Bartholmes, J. Thielen, M.H. Ramos, S. Gentilini, The European Flood Alert System EFAS – part 2: statistical skill assessment of probabilistic and deterministic operational forecasts. Hydrol. Earth Syst. Sci. 13, 141–153 (2009)CrossRefGoogle Scholar
  9. W.M. Briggs, R.A. Levine, Wavelets and field forecast verification. Mon. Weather Rev. 125, 1329–1341 (1997)CrossRefGoogle Scholar
  10. B.G. Brown, E. Gilleland, E.E. Ebert, Forecasts of spatial fields, in Forecast Verification, ed. by I. Jolliffe, D.B. Stephenson. The Atrium, Southern Gate, Chichester, UK (2011), pp. 95–117CrossRefGoogle Scholar
  11. R. Bullock, Development and implementation of MODE time domain object-based verification, in 24th Conference on Weather and Forecasting, Seattle, 24–27 Jan 2011. American Meteorological SocietyGoogle Scholar
  12. N. Bussieres, W. Hogg, The objective analysis of daily rainfall by distance weighting schemes on a mesoscale grid. Atmosphere–Ocean 27, 521–541 (1989)CrossRefGoogle Scholar
  13. B. Casati, New developments of the intensity-scale technique within the spatial verification methods inter-comparison project. Weather Forecast. 25(1), 113–143 (2010).  https://doi.org/10.1175/2009WAF2222257.1CrossRefGoogle Scholar
  14. B. Casati, G. Ross, D.B. Stephenson, A new intensity-scale approach for the verification of spatial precipitation forecasts. Meteorol. Appl. 11, 141–154 (2004)CrossRefGoogle Scholar
  15. G.J. Ciach, W.F. Krajewski, On the estimation of radar rainfall error variance. Adv. Water Resour. 22, 585–595 (1999)CrossRefGoogle Scholar
  16. R. Cifelli, V. Chandrasekar, Dual-polarization radar rainfall estimation, in Rainfall: State of the Science, ed. by Y.M. Tistek, M. Gebremichael. American Geophysical Union, Washington, DC (2010), pp. 105–125. doi:10.1029/2010GM000930Google Scholar
  17. H.L. Cloke, C. Jeffers, F. Wetterhall, T. Byrne, J. Lowe, F. Pappenberger, Climate impacts on river flow: projections for the Medway catchment, UK, with UKCP09 and CATCHMOD. Hydrol. Process. (2010).  https://doi.org/10.1002/hyp.776CrossRefGoogle Scholar
  18. S.G. Coles, An Introduction to Statistical Modeling of Extreme Values (Springer, London, 2001)CrossRefGoogle Scholar
  19. C. Davis, B. Brown, R.G. Bullock, Object-based verification of precipitation forecasts. Part I: methodology and application to mesoscale rain areas. Mon. Weather Rev. 134(7), 1772–1784 (2006a).  https://doi.org/10.1175/MWR3145.1CrossRefGoogle Scholar
  20. C. Davis, B. Brown, R.G. Bullock, Object-based verification of precipitation forecasts. Part II: application to convective rain systems. Mon. Weather Rev. 134(7), 1785–1795 (2006b).  https://doi.org/10.1175/MWR3146.1CrossRefGoogle Scholar
  21. C. Davis, B. Brown, R.G. Bullock, J. Halley Gotway, The method for object-based diagnostic evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC spring program. Weather Forecast. 24(5), 1252–1267 (2009).  https://doi.org/10.1175/2009WAF2222241.1CrossRefGoogle Scholar
  22. F.X. Diebold, R.S. Mariano, Comparing predictive accuracy. J. Bus. Econ. Stat. 13, 253–263 (1995)Google Scholar
  23. L. Duc, K. Saito, H. Seko, Spatial-temporal fractions verification for high-resolution ensemble forecasts. Tellus A 65, 18171 (2013).  https://doi.org/10.3402/tellusa.v65i0.18171CrossRefGoogle Scholar
  24. J.D. Duda, X. Wang, F. Kong, M. Xue, Using varied microphysics to account for uncertainty in warm-season QPF in a convection-allowing ensemble. Mon. Weather Rev. 142, 2198–2219 (2014)CrossRefGoogle Scholar
  25. E. Dutra, M. Diamantakis, I. Tsonevsky, E. Zsoter, F. Wetterhall, T. Stockdale, D. Richardson, F. Pappenberger, The extreme forecast index at the seasonal scale. Atmos. Sci. Lett. 14(4), 256–262 (2013)CrossRefGoogle Scholar
  26. E.E. Ebert, Fuzzy verification of high resolution gridded forecasts: a review and proposed framework. Meteorol. Appl. 15, 51–64 (2008).  https://doi.org/10.1002/met.25. Available at http://www.ecmwf.int/newsevents/meetings/workshops/2007/jwgv/METspecialissueemail.pdfCrossRefGoogle Scholar
  27. E.E. Ebert, J.L. McBride, Verification of precipitation in weather systems: determination of systematic errors. J. Hydrol. 239, 179–202 (2000)CrossRefGoogle Scholar
  28. E.E. Ebert, U. Damrath, W. Wergen, M.E. Baldwin, The WGNE assessment of short-term quantitative precipitation forecasts. Bull. Am. Meteorol. Soc. 84, 481–492 (2003)CrossRefGoogle Scholar
  29. E. Ebert, L. Wilson, A. Weigel, M. Mittermaier, P. Nurmi, P. Gill, M. Göber, S. Joslyn, B. Brown, T. Fowler, A. Watkins, Progress and challenges in forecast verification. Meteorol. Appl. 20, 130–139 (2013).  https://doi.org/10.1002/met.1392CrossRefGoogle Scholar
  30. EUMETNET, ODYSSEY, the OPERA Data Centre (2014), http://www.eumetnet.eu/odyssey-opera-data-centre. Viewed 10 Apr 2014
  31. C.A.T. Ferro, D.B. Stephenson, Extremal dependence: improved verification measures for deterministic forecasts of rare binary events. Weather Forecast. 26, 699–713 (2011).  https://doi.org/10.1175/WAF-D-10-05030.1CrossRefGoogle Scholar
  32. V. Fortin, M. Abaza, F. Anctil, R. Turcotte, Why should ensemble spread match the RMSE of the ensemble mean? J. Hydrometeorol. 15, 1708–1713 (2016)CrossRefGoogle Scholar
  33. W.A. Gallus Jr., Application of object-based verification techniques to ensemble precipitation forecasts. Weather Forecast. 25, 144–158 (2009)CrossRefGoogle Scholar
  34. A. Ghelli, A. Garcia-Mendez, F. Prates, M. Dahoui, Extreme weather events in summer 2010: how did the ECMWF forecasting systems perform? ECMWF Newsl. 125, 7–11 (2010)Google Scholar
  35. R. Giacomini, H. White, Tests of conditional predictive ability. Econometrica 74, 1545–1578 (2006)CrossRefGoogle Scholar
  36. E. Gilleland, Spatial forecast verification: Baddeley’s delta metric applied to the ICP test cases. Weather Forecast. 26, 409–415 (2011).  https://doi.org/10.1175/WAF-D-10-05061.1CrossRefGoogle Scholar
  37. E. Gilleland, Testing competing precipitation forecasts accurately and efficiently: the spatial prediction comparison test. Mon. Weather Rev. 141(1), 340–355 (2013).  https://doi.org/10.1175/MWR-D-12-00155.1CrossRefGoogle Scholar
  38. E. Gilleland, T.C.M. Lee, J. Halley Gotway, R.G. Bullock, B.G. Brown, Computationally efficient spatial forecast verification using Baddeley’s Delta image metric. Mon. Weather Rev. 136(5), 1747–1757 (2008).  https://doi.org/10.1175/2007MWR2274.1CrossRefGoogle Scholar
  39. E. Gilleland, D. Ahijevych, B.G. Brown, B. Casati, E.E. Ebert, Intercomparison of spatial forecast verification methods. Weather Forecast. 24(5), 1416–1430 (2009).  https://doi.org/10.1175/2009WAF2222269.1CrossRefGoogle Scholar
  40. E. Gilleland, Coauthors, Spatial forecast verification: Image warping. NCAR Technical Note NCAR/TN-482+STR (2010). doi:10.5065/D62805JJGoogle Scholar
  41. E. Gilleland, D.A. Ahijevych, B.G. Brown, E.E. Ebert, Verifying forecasts spatially. Bull. Am. Meteorol. Soc. 91(10), 1365–1373 (2010a).  https://doi.org/10.1175/2010BAMS2819.1CrossRefGoogle Scholar
  42. E. Gilleland, L. Chen, M. DePersio, G. Do, K. Eilertson, Y. Jin, E.K. Lang, F. Lindgren, J. Lindström, R.L. Smith, C. Xia, Spatial forecast verification: image warping. NCAR Tech. Note (2010b), NCAR/TN-482 + STR doi:10.5065/D62805JJGoogle Scholar
  43. E. Gilleland, J. Lindström, F. Lindgren, Analyzing the image warp forecast verification method on precipitation fields from the ICP. Weather Forecast. 25(4), 1249–1262 (2010c).  https://doi.org/10.1175/2010WAF2222365.1CrossRefGoogle Scholar
  44. T. Gneiting, R. Ranjan, Comparing density forecasts using threshold- and quantile-weighted scoring rules. J. Bus. Econ. Stat. 29(3), 411–422 (2011)CrossRefGoogle Scholar
  45. M. Goeber, E. Zsoter, D.S. Richardson, Could a perfect model ever satisfy the forecaster? On grid box mean versus point verification. Meteorol. Appl. 15, 359–365 (2008)CrossRefGoogle Scholar
  46. T. Haiden, M. Rodwell, D. Richardson, A. Okagaki, T. Robinson, T. Hewson, Intercomparison of global model precipitation forecast skill in 2010/11 using the SEEPS score. Mon. Weather Rev. 140, 2720–2733 (2012)CrossRefGoogle Scholar
  47. T. Haiden, L. Magnusson, I. Tsonevsky, F. Wetterhall, L. Alfieri, F. Pappenberger, P. de Rosnay, J. Muñoz-Sabater, G. Balsamo, C. Albergel, R. Forbes, T. Hewson, S. Malardel, D. Richardson, ECMWF forecast performance during the June 2013 flood in Central Europe. Technical Memorandum No. 723, European Centre for Medium Range Weather Forecasts, Reading, England, vol 723 (2014)Google Scholar
  48. T.M. Hamill, J. Juras, Measuring forecast skill: is it real skill or is it the varying climatology? Q. J. Roy. Meteorol. Soc. 132, 2905–2923 (2006)CrossRefGoogle Scholar
  49. A.S. Hering, M.G. Genton, Comparing spatial predictions. Technometrics 53, 414–425 (2011)CrossRefGoogle Scholar
  50. R.N. Hoffman, Z. Liu, J.-F. Louis, C. Grassotti, Distortion representation of forecast errors. Mon. Weather Rev. 123, 1758–2770 (1995)CrossRefGoogle Scholar
  51. G.J. Huffman, R.F. Adler, D.T. Bolvin, G. Gu, E.J. Nelkin, K.P. Bowman, Y. Hong, E.F. Stocker, D.B. Woff, The TRMM multi-satellite precipitation analysis: quasi-global, multi-year, combined-sensor precipitation estimates at fine scale. J. Hydrometeorol. 8, 38–55 (2007)CrossRefGoogle Scholar
  52. G.J. Huffman, D.T. Bolvin, D. Braithwaite, K. Hsu, R. Joyce, P. Xie, S.H. Yoo, NASA global precipitation measurement (GPM) integrated multi-satellite retrievals for GPM (IMERG), in Algorithm Theoretical Basis Document, Version 4.1. NASA, Greenbelt, MD (2013)Google Scholar
  53. IPWG, International Precipitation Working Group: Data and Products (2016), http://www.isac.cnr.it/~ipwg/data.html. Viewed 18 June 2016
  54. J. Jenkner, C. Frei, C. Schwierz, Quantile-based short-range QPF evaluation over Switzerland. Meteorol. Z. 17, 827–848 (2008)CrossRefGoogle Scholar
  55. C. Keil, G.C. Craig, A displacement-based error measure applied in a regional ensemble forecasting system. Mon. Weather Rev. 135, 3248–3259 (2007).  https://doi.org/10.1175/MWR3457.1CrossRefGoogle Scholar
  56. C. Keil, G.C. Craig, A displacement and amplitude score employing an optical flow technique. Weather Forecast. 24(5), 1297–1308 (2009). doi:10.1175/2009WAF2222247.1CrossRefGoogle Scholar
  57. C. Kidd, V. Levizzani, Status of satellite precipitation retrievals. Hydrol. Earth Syst. Sci. 15, 1109–1116 (2011)CrossRefGoogle Scholar
  58. S.A. Lack, G.L. Limpert, N.I. Fox, An object-oriented multiscale verification scheme. Weather Forecast. 25(1), 79–92 (2010).  https://doi.org/10.1175/2009WAF2222245.1CrossRefGoogle Scholar
  59. F. Lalaurette, Early detection of abnormal weather conditions using a probabilistic extreme forecast index. Q. J. Roy. Meteorol. Soc. 129(594), 3037–3057 (2003)CrossRefGoogle Scholar
  60. S.M. Lazarus, C.M. Ciliberti, J.D. Horel, K.A. Brewster, Near-real-time applications of a mesoscale analysis system to complex terrain. Weather Forecast. 17, 971–1000 (2002)CrossRefGoogle Scholar
  61. S. Lerch, Verification of Probabilitsic Forecasts for Rare and Extreme Events (Ruprecht-Karls-Universitaet Heidelberg, Heidelberg, 2012)Google Scholar
  62. S. Lerch, T.L. Thorarinsdottir, Comparison of nonhomogeneous regression models for probabilistic wind speed forecasting. Tellus A 65, 21206 (2013)CrossRefGoogle Scholar
  63. C. Marsigli, A. Montani, T. Pacagnella, A spatial verification method applied to the evaluation of high-resolution ensemble forecasts. Meteorol. Appl. 15, 125–143 (2008)CrossRefGoogle Scholar
  64. C.F. Mass, D. Ovens, K. Westrick, B.A. Colle, Does increasing horizontal resolution produce more skillful forecasts? Bull. Am. Meteorol. Soc. 83, 407–430 (2002)CrossRefGoogle Scholar
  65. J.L. McBride, E.E. Ebert, Verification of quantitative precipitation forecasts from operational numerical weather prediction models over Australia. Weather Forecast. 15, 103–121 (2000)CrossRefGoogle Scholar
  66. R. Merz, G. Blöschl, A process typology of regional floods, Water Resour. Res. 39, 1340 (2003). doi:10.1029/2002WR001952, 12Google Scholar
  67. A. Micheas, N.I. Fox, S.A. Lack, C.K. Wikle, Cell identification and verification of QPF ensembles using shape analysis techniques. J. Hydrol. 344, 105–116 (2007)CrossRefGoogle Scholar
  68. J.R. Minder, P.W. Mote, J.D. Lundquist, Surface temperature lapse rates over complex terrain: lessons from the Cascade Mountains. J. Geophys. Res. Atmos. 115(D14) (2010).  https://doi.org/10.1029/2009JD013493
  69. M. Mittermaier, N. Roberts, Intercomparison of spatial forecast verification methods: identifying skillful spatial scales using the fractions skill score. Weather Forecast. 25, 343–354 (2010)CrossRefGoogle Scholar
  70. M. Mittermaier, N. Roberts, S.A. Thompson, A long‐term assessment of precipitation forecast skill using the Fractions Skill Score. Meteorol. Appl. 20, 176–186 (2013)CrossRefGoogle Scholar
  71. F. Molteni, R. Buizza, T.N. Palmer, T. Petroliagis, The ECMWF ensemble prediction system: methodology and validation. Q. J. Roy. Meteorol. Soc. 122, 73–119 (1996)CrossRefGoogle Scholar
  72. J.E. Nash, J.V. Sutcliffe, River flow forecasting through conceptual models. Part I: a discussion of principles. J. Hydrol. 10, 282–290 (1970)CrossRefGoogle Scholar
  73. R. North, M. Trueman, M. Mittermaier, M.J. Rodwell, An assessment of the SEEPS and SEDI metrics for the verification of 6 h forecast precipitation accumulations. Meteorol. Appl. 20, 164–175 (2013)CrossRefGoogle Scholar
  74. NWS, Advanced Hydrologic Prediction Service (2014), http://water.weather.gov/precip/about.php. Viewed 10 Apr 2014
  75. F. Pappenberger, M.H. Ramos, H.L. Cloke, F. Wetterhall, L. Alfieri, K. Bogner, A. Mueller, P. Salamon, How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction. J. Hydrol. 522, 697–713 (2015)CrossRefGoogle Scholar
  76. T. Petroliagis, P. Pinson, Early indication of extreme winds utilising the Extreme Forecast Index, ECMWF Newsletter No. 132 – Summer 2012, 2012Google Scholar
  77. R.-D. Reiss, M. Thomas, Statistical Analysis of Extreme Values: With Applications to Insurance, Finance, Hydrology and Other Fields, 3rd edn. (Birkhäuser, Basel, 2007), 530ppGoogle Scholar
  78. N.M. Roberts, H.W. Lean, Scale-selective verification of rainfall accumulations from high-resolution forecasts of convective events. Mon. Wea. Rev. 136, 78–96 (2008)CrossRefGoogle Scholar
  79. M.J. Rodwell, D.S. Richardson, T.D. Hewson, T. Haiden, A new equitable score suitable for verifying precipitation in numerical weather prediction. Q. J. Roy. Meteorol. Soc. 136, 1344–1363 (2010)Google Scholar
  80. Ø. Saetra, H. Hersbach, J.-R. Bidlot, D.S. Richardson, Effects of observation errors on the statistics for ensemble spread and reliability. Mon. Weather Rev. 132, 1487–1501 (2004)CrossRefGoogle Scholar
  81. B.R.J. Schwedler, M.E. Baldwin, Diagnosing the sensitivity of binary image measures to bias, location, and event frequency within a forecast verification framework. Weather Forecast. 26, 1032–1044 (2011).  https://doi.org/10.1175/WAF-D-11-00032.1CrossRefGoogle Scholar
  82. D.J. Seo, J.P. Breidenbach, Real-time correction of spatially nonuniform bias in radar rainfall data using rain gauge measurements. J. Hydrometeorol. 3, 93–111 (2002)CrossRefGoogle Scholar
  83. R. Steinacker, C. Häberli, W. Pöttschacher, A transparent method for the analysis and quality evaluation of irregularly distributed and noisy observational data. Mon. Weather Rev. 128, 2303–2316 (2000)CrossRefGoogle Scholar
  84. D.B. Stephenson, B. Casati, C.A.T. Ferro, C.A. Wilson, The extreme dependency score: a non-vanishing measure for forecasts of rare events. Meteorol. Appl. 15, 41–50 (2008).  https://doi.org/10.1002/met.53CrossRefGoogle Scholar
  85. J. Thielen-del Pozo, J. Bartholmes, M.-H. Ramos, A. de Roo, The European Flood Alert System. Part 1: concept and development. Hydrol. Earth Syst. Sci. 13, 125–140 (2009)CrossRefGoogle Scholar
  86. I. Tsonevsky,D.S. Richardson, D.S, Application of the new EFI products to a case of early snowfall in Central Europe. ECMWF Newsletter 133, Autumn 2012, p. 4, 2012Google Scholar
  87. H. Wernli, M. Paulat, M. Hagen, C. Frei, SAL – a novel quality measure for the verification of quantitative precipitation forecasts. Mon. Weather Rev. 136, 4470–4487 (2008)CrossRefGoogle Scholar
  88. T. Weusthoff, F. Ament, M. Arpagaus, M.W. Rotach, Assessing the benefits of convection-permitting models by neighborhood verification: examples from MAP D-PHASE. Mon. Weather Rev. 138, 3418–3433 (2010)CrossRefGoogle Scholar
  89. D.S. Wilks, Statistical Methods in the Atmospheric Sciences, 3rd edn. (Academic, Amsterdam, 2011), 704 ppGoogle Scholar
  90. WMO, Recommendations for the Verification and Intercomparison of QPFs and PQPFs from Operational NWP Models – Revision 2, WMO/TD-No.1485 WWRP 2009-1 (2008)Google Scholar
  91. WMO, Manual on the Global Data-Processing and Forecasting System (GDPFS). WMO-no.485, World Meteorological Organization, Geneva, vol 485 (2014)Google Scholar
  92. J. Wolff, M. Harrold, T.A. Fowler, J. Halley Gotway, L. Nance, B.G. Brown, Beyond the basics: evaluating model-based precipitation forecasts using traditional, spatial, and object-based methods. Weather Forecast. 29, 1451–1472 (2014)CrossRefGoogle Scholar
  93. P. Zacharov, D. Rezacova, Using the fractions skill score to assess the relationship between an ensemble QPF spread and skill. Atmos. Res. 94, 684–693 (2009)CrossRefGoogle Scholar
  94. M. Zimmer, H. Wernli, Verification of quantitative precipitation forecasts on short time-scales: a fuzzy approach to handle timing errors with SAL. Meteorol. Z. 20, 95–105 (2011)CrossRefGoogle Scholar
  95. E. Zsótér, Recent developments in extreme weather forecasting. ECMWF Newsl. 107(107), 8–17 (2006)Google Scholar
  96. E. Zsótér, F. Pappenberger, D. Richardson, Sensitivity of model climate to sampling configurations and the impact on the Extreme Forecast Index. Meteorol. Appl. (2014).  https://doi.org/10.1002/met.1447CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Eric Gilleland
    • 1
    Email author
  • Florian Pappenberger
    • 2
  • Barbara Brown
    • 1
  • Elizabeth Ebert
    • 3
  • David Richardson
    • 2
  1. 1.Research Applications Laboratory, Weather Systems and Assessment ProgramNational Center for Atmospheric Research NCARBoulderUSA
  2. 2.European Centre for Medium-Range Weather Forecasts, ECMWFReadingUK
  3. 3.Research and Development BranchBureau of Meteorology, BoMMelbourneAustralia

Personalised recommendations