Skip to main content
SpringerLink
Log in
Book cover

Macroeconomic Survey Expectations pp 65–96Cite as

Palgrave Macmillan

Assessing the Accuracy of the Probability Distributions

  • Chapter
  • First Online:

  • 577 Accesses

Part of the book series: Palgrave Texts in Econometrics ((PTEC))

Abstract

Ways of assessing the quality of the forecast densities (reported in the form of histograms) provided by the survey respondents are described and applied to the US SPF. Forecast densities can be assessed in terms of whether they could have generated the observed data: that is, whether they differ significantly from the assumed (but unknown) actual densities which gave rise to the observed data. They can also be compared to rival density forecasts, even if they are found wanting in absolute terms. In the reported assessment of the SPF aggregate and individual densities, the benchmarks are constructed to spotlight a particular aspect of the SPF densities. That is, whether the SPF respondents are able to adequately capture the time-varying uncertainty that characterized output growth and inflation. Rather than evaluating the whole densities, specific regions of interest can be considered, and this is illustrated. Finally, some scoring rules might be better suited than others when, as here, the densities are presented in the form of histograms.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   99.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    This chapter is based on Clements, M.P. (2018), Are Macroeconomic Density Forecasts Informative?, International Journal of Forecasting, 34(2), 181–198. Material from this article is used with the permission of Elsevier.

  2. 2.

    This dates back at least to Rosenblatt (1952). See Shephard (1994), Kim et al. (1998), and Diebold et al. (1998), inter alia, for recent contributions.

  3. 3.

    We do not consider this here because surveys do not report joint densities. The US SPF, for example, reports histograms for output growth, and for inflation, separately, but not for output growth and inflation jointly.

  4. 4.

    See Bao et al. (2007) and Hall and Mitchell (2009).

  5. 5.

    We could of course write (5.5) using (5.3), but this would simply introduce unknown densities, which would need to be specified.

  6. 6.

    The standard approach would be to estimate ξ 2 by \(\hat {\xi }^{2}\) where \(\hat {\xi }^{2}=\hat {\gamma }_{0}+2\sum _{j=1}^{p}\left ( \frac {p-j}{p} \right ) \hat {\gamma }_{j}\), where \(\hat {\gamma }_{0}=\frac {1}{n} \sum _{t=1}^{n}\left ( d_{t}-\overline {d}\right ) ^{2}\), \(\hat {\gamma }_{j}= \frac {1}{n}\sum _{t=j+1}^{n}\left ( d_{t}-\overline {d}\right ) \left ( d_{t-j}- \overline {d}_{j}\right ) \), and dt ≡ dt∣t−h, \(\overline {d}= \frac {1}{n}\sum _{t=1}^{n}d_{t}\), \(\overline {d_{j}}=\frac {1}{n-j} \sum _{t=j+1}^{n}d_{t-j}\).

  7. 7.

    Winkler (1967) is the classic reference on scoring rules, and more recently (Gneiting and Raftery 2007).

  8. 8.

    This is formalized by Manzan (2016) within a Bayesian framework in which an individual updates her/his prior density as new information becomes available.

  9. 9.

    The SPF provides ‘forecasts’ of the previous quarter’s value, and these are invariably set equal to the released data, that is, \( F_{80:4}^{81:1}=Y_{80:4}^{81:1}\).

  10. 10.

    We do not need to assume normality to calculate z (and hence z ∗): we could simply look at the proportion of the historical errors which are less than the realization. However, when this is 0 or 1, the calculation of z ∗ is problematic as the inverse normal CDF is not defined.

  11. 11.

    When, as here, z is calculated after fitting a Gaussian distribution to the histogram, taking the inverse standard normal CDF of z to give z ∗ results in \(z^{\ast }=(y-\hat {\mu })/\hat {\sigma }\), where y is the realization and \(\hat {\mu }\) and \(\hat {\sigma }^{2}\) are the mean and variance of the fitted distribution.

  12. 12.

    In fact we calculate the standard deviation of the aggregate histogram as the average of the individuals’ standard deviations. The results of doing so are indistinguishable from taking the square root of the average of the individuals’ variances. Abel et al. (2016) calculate the ‘average variance’ by first estimating the variance of the aggregate histogram, and then subtracting the disagreement term. See also Lahiri and Sheng (2010) and Lahiri et al. (2015).

  13. 13.

    Including for the first quarter output growth forecasts, that is, notwithstanding the failure to reject the unconditional benchmark densities on the Berkowitz tests, and the rejection of those which condition on location. The Berkowitz tests reject in part because the estimated variance of z ∗ is less than 1 (\(\hat {\sigma }^{2}=0.23\) for Q1 output growth forecasts). For the truly unconditional densities, the large forecast errors which result from not conditioning on location result in more extreme values of z ∗ and an estimated variance close to 1.

  14. 14.

    We used p = 3 in the expression given in the footnote in Sect. 5.2.

  15. 15.

    The 2% target was formally adopted by the FOMC at its meeting in January 2012 and was for the price index for personal consumption expenditures. See, for example, https://www.federalreserve.gov/faqs/money_12848.htm.

  16. 16.

    Especially as professional forecasters are not renowned for their ability to predict recessions: see, for example, Rudebusch and Williams (2009).

References

  • Abel, J., Rich, R., Song, J., & Tracy, J. (2016). The measurement and behavior of uncertainty: Evidence from the ECB survey of professional forecasters. Journal of Applied Econometrics, 31(3), 533–550.

    Article  Google Scholar 

  • Aiolfi, M., Capistrán, C., & Timmermann, A. (2011). Forecast combinations. In M. P. Clements & D. F. Hendry (Eds.), The Oxford handbook of economic forecasting (Chap. 11, pp. 355–388). Oxford: Oxford University Press.

    Google Scholar 

  • Amisano, G., & Giacomini, R. (2007). Comparing density forecasts via weighted likelihood ratio tests. Journal of Business & Economic Statistics, 25, 177–190.

    Article  Google Scholar 

  • Bao, Y., Lee, T.-H., & Saltoglu, B. (2007). Comparing density forecast models. Journal of Forecasting, 26(3), 203–225.

    Article  Google Scholar 

  • Bai, J., & Ng, S. (2005). Tests for skewness, kurtosis, and normality for time series data. Journal of Business & Economic Statistics, 23, 49–60.

    Article  Google Scholar 

  • Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 75, 1–3.

    Article  Google Scholar 

  • Berkowitz, J. (2001). Testing density forecasts, with applications to risk management. Journal of Business and Economic Statistics, 19(4), 465–474.

    Article  Google Scholar 

  • Boero, G., Smith, J., & Wallis, K. F. (2011). Scoring rules and survey density forecasts. International Journal of Forecasting, 27(2), 379–393.

    Article  Google Scholar 

  • Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583. Reprinted in Mills, T. C. (Ed.). (1999). Economic forecasting. The International Library of Critical Writings in Economics. Cheltenham: Edward Elgar.

    Google Scholar 

  • Clements, M. P. (2004). Evaluating the Bank of England density forecasts of inflation. Economic Journal, 114, 844–866.

    Article  Google Scholar 

  • Clements, M. P. (2018). Are macroeconomic density forecasts informative?. International Journal of Forecasting, 34, 181–198.

    Article  Google Scholar 

  • Clements, M. P., & Smith, J. (2000). Evaluating the forecast densities of linear and non-linear models: Applications to output growth and unemployment. Journal of Forecasting, 19, 255–276.

    Article  Google Scholar 

  • Diebold, F. X., Gunther, T. A., & Tay, A. S. (1998). Evaluating density forecasts: With applications to financial risk management. International Economic Review, 39, 863–883.

    Article  Google Scholar 

  • Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics, 13, 253–263.

    Google Scholar 

  • Diks, C., Panchenko, V., & van Dijk, D. (2011). Likelihood-based scoring rules for comparing density forecasts in tails. Journal of Econometrics, 163(2), 215–230. http://dx.doi.org/10.1016/j.jeconom.2011.04.001.

    Article  Google Scholar 

  • Engle, R. F., & Manganelli, S. (2004). CAViaR: Conditional autoregressive value at risk by regression quantiles. Journal of Business & Economic Statistics, 22, 367–381.

    Article  Google Scholar 

  • Epstein, E. S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8, 985–987.

    Article  Google Scholar 

  • Genest, C., & Zidek, J. V. (1986). Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1, 114–148.

    Article  Google Scholar 

  • Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.

    Article  Google Scholar 

  • Gneiting, T., & Ranjan, R. (2011). Comparing density forecasts using threshold and quantile weighted proper scoring rules. Journal of Business and Economic Statistics, 29, 411–422.

    Article  Google Scholar 

  • Hall, S. G., & Mitchell, J. (2009). Recent developments in density forecasting. In T. C. Mills & K. Patterson (Eds.), Palgrave handbook of econometrics, volume 2: Applied econometrics (pp. 199–239). New York: Palgrave MacMillan.

    Chapter  Google Scholar 

  • Kenny, G., Kostka, T., & Masera, F. (2014). How informative are the subjective density forecasts of macroeconomists?. Journal of Forecasting, 33(3), 163–185.

    Article  Google Scholar 

  • Kim, S., Shephard, N., & Chib, S. (1998). Stochastic volatility: Likelihood inference and comparison with ARCH models. Review of Economic Studies, 81, 361–393.

    Article  Google Scholar 

  • Knüppel, M. (2015). Evaluating the calibration of multi-step-ahead density forecasts using raw moments. Journal of Business & Economic Statistics, 33(2), 270–281.

    Article  Google Scholar 

  • Kullback, L., & Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Sciences, 22, 79–86.

    Google Scholar 

  • Lahiri, K., Peng, H., & Sheng, X. (2015). Measuring Uncertainty of a Combined Forecast and Some Tests for Forecaster Heterogeneity. Cesifo Working Paper Series 5468, CESifo Group Munich.

    Google Scholar 

  • Lahiri, K., & Sheng, X. (2010). Measuring forecast uncertainty by disagreement: The missing link. Journal of Applied Econometrics, 25, 514–538.

    Article  Google Scholar 

  • Lahiri, K., Teigland, C., & Zaporowski, M. (1988). Interest rates and the subjective probability distribution of inflation forecasts. Journal of Money, Credit and Banking, 20(2), 233–248.

    Article  Google Scholar 

  • Manski, C. F. (2011). Interpreting and combining heterogeneous survey forecasts. In M. P. Clements & D. F. Hendry (Eds.), Oxford handbook of economic forecasting (Chap. 16, pp. 457–472). Oxford: Oxford University Press.

    Google Scholar 

  • Manzan, S. (2016). Are Professional Forecasters Bayesian. Mimeo, Baruch College, CUNY, New York.

    Google Scholar 

  • Rosenblatt, M. (1952). Remarks on a multivariate transformation. Annals of Mathematical Statistics, 23, 470–472.

    Article  Google Scholar 

  • Rossi, B., & Sekhposyan, T. (2015). Macroeconomic uncertainty indices based on nowcast and forecast error distributions. American Economic Review, 105(5), 650–55.

    Article  Google Scholar 

  • Rudebusch, G. D., & Williams, J. C. (2009). Forecasting recessions: The puzzle of the enduring power of the yield curve. Journal of Business & Economic Statistics, 27(4), 492–503.

    Article  Google Scholar 

  • Shephard, N. (1994). Partial non-Gaussian state space. Biometrika, 81, 115–131.

    Article  Google Scholar 

  • Wallis, K. F. (2005). Combining density and interval forecasts: A modest proposal. Oxford Bulletin of Economics and Statistics, 67(s1), 983–994.

    Article  Google Scholar 

  • Winkler, R. L. (1967). The quantification of judgement: Some methodological suggestions. Journal of the American Statistical Association, 62, 1105–1120.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ICMA Centre, Henley Business School, University of Reading, Wheatley, UK

    Michael P. Clements

Authors
  1. Michael P. Clements
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2019 The Author(s)

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Clements, M.P. (2019). Assessing the Accuracy of the Probability Distributions. In: Macroeconomic Survey Expectations. Palgrave Texts in Econometrics. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-97223-7_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-97223-7_5

  • Published:

  • Publisher Name: Palgrave Macmillan, Cham

  • Print ISBN: 978-3-319-97222-0

  • Online ISBN: 978-3-319-97223-7

  • eBook Packages: Economics and FinanceEconomics and Finance (R0)

Publish with us

Policies and ethics

Search

Navigation