Journal of Statistical Theory and Practice

, Volume 6, Issue 3, pp 452–467 | Cite as

Exceedance Probability Score: A Novel Measure for Comparing Probabilistic Predictions

  • I. JuutilainenEmail author
  • S. Tamminen
  • J. Röning


Accurate prediction of exceedance probabilities is important in many applications. For example, in process planning and control, engineers should anticipate the risk that a product fails to meet its specification limits. Statistical comparison between candidate probability prediction methods is commonly performed using scoring rules, like the continuous ranked probability score (CRPS) and the logarithm score (LogS). In this work, a new scoring rule, the exceedance probability score, is proposed. The experiments in simulated and real industrial data show that the new scoring rule is useful in comparing and testing differences in the predictive accuracy of competitive probabilistic predictions in regression setting. The proposed scoring rule have some similarities with CRPS and LogS, but is more directly connected to the accuracy in the prediction of exceedance probabilities.

AMS Subject Classification

62J99 62H15 62P30 


CRPS Density forecast logarithm score Probability prediction Scoring rules 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Amisano, G., and R. Giacomini. 2007. Comparing density forecasts via weighted likelihood ratio tests. J. Business Econ. Stat., 25(2), 177–190.MathSciNetCrossRefGoogle Scholar
  2. Bao, Y., T.-H. Lee, and B. Saltoglu. 2007. Comparing density forecast models. J. Forecast., 26(3), 203–225.MathSciNetCrossRefGoogle Scholar
  3. Cannon, A. J. 2011. Quantile regression neural networks: Implementation in r and application to precipitation downscaling. Comput. Geosci., 37(9), 1277–1284.CrossRefGoogle Scholar
  4. Diebold, F., and R. Mariano. 1995. Comparing predictive accuracy. J. Business Econ. Stat., 13, 253–263.Google Scholar
  5. Diks, C., V. Panchenko, and D. van Dijk. 2011. Likelihood-based scoring rules for comparing density forecasts in tails. J. Econometrics, 163(2), 215–230.MathSciNetCrossRefGoogle Scholar
  6. Gneiting, T., and A. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc., 102(477), 359–378.MathSciNetCrossRefGoogle Scholar
  7. Gneiting, T., and A. E. Raftery. 2005. Weather forecasting with ensemble methods. Science, 310(5746), 248–249.CrossRefGoogle Scholar
  8. Gneiting, T., and R. Ranjan. 2011. Comparing density forecasts using threshold- and quantile-weighted scoring rules. J. Business Econ. Stat., 29(3), 411–422.MathSciNetCrossRefGoogle Scholar
  9. Gneiting, T., L. Stanberry, E. Grimit, L. Held, and N. Johnson. 2008. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. TEST Off. J. Span. Soc. Stat. Operations Res., 17(2), 211–235.MathSciNetzbMATHGoogle Scholar
  10. Hall, P., R. Wolff, and Q. Yao. 1999. Methods for estimating a conditional distribution function. J. Am. Stat. Assoc., 94, 154–163.MathSciNetCrossRefGoogle Scholar
  11. Koenker, R., and G. Bassett. 1978. Regression quantiles. Econometrica, 46, 33–46.MathSciNetCrossRefGoogle Scholar
  12. Laio, F., and S. Tamea. 2007. Verification tools for probabilistic forecasts of continuous hydrological variables. Hydrol. Earth System Sci., 11(4), 1267–1277.CrossRefGoogle Scholar
  13. Mason, S. J., J. S. Galpin, L. Goddard, N. E. Graham, and B. Rajartnam. 2007. Conditional exceedance probabilities. Monthly Weather Rev., 135(2), 363–372.CrossRefGoogle Scholar
  14. Matheson, J. E., and R. L. Winkler. 1976. Scoring rules for continuous probability distributions. Manage. Sci., 22, 1087–1096.CrossRefGoogle Scholar
  15. Pinson, P., and G. Kariniotakis. 2010. Conditional prediction intervals of wind power generation. Power Systems IEEE Trans. 25(4), 1845–1856.CrossRefGoogle Scholar
  16. Quiñonero-Candela, J., C. Rasmussen, F. Sinz, O. Bousquet, and B. Schölkopf. 2006. Evaluating predictive uncertainty challenge. Lecture Notes Computer Sci, 3944, 1–27.CrossRefGoogle Scholar
  17. Stasinopoulos, D., and R. Rigby. 2007. Generalized additive models for location scale and shape (gamlss) in R. J. Stat. Software, 23(7), 1–46.CrossRefGoogle Scholar
  18. Tay, A. S., and K. F. Wallis. 2000. Density forecasting: A survey. J. Forecast., 19(4), 235–254.CrossRefGoogle Scholar

Copyright information

© Grace Scientific Publishing 2012

Authors and Affiliations

  1. 1.Computer Science and Engineering Laboratory, Department of Electrical and Information EngineeringUniversity of OuluOuluFinland

Personalised recommendations