Advertisement

Assessing the Point Predictions

  • Michael P. Clements
Chapter
Part of the Palgrave Texts in Econometrics book series (PTEC)

Abstract

Point forecasts can typically be evaluated in terms of their accuracy relative to a rival set of forecasts, or in terms of whether they possess certain desirable characteristics, such as the forecast and forecast error being uncorrelated. There are differences between the evaluation of survey and model forecasts. An obvious one is that survey forecasts cannot be compared to their expected accuracy given the in-sample fit (past track record) of the model, since their provenance is unknown. Comparisons of forecasts from rival models typically seek to compare the models (in population). The evaluation of rival sets of survey forecasts rests squarely on the forecasts themselves. Tests of equal accuracy or forecast encompassing, as well as tests of whether the forecasts possess desirable characteristics, such as forecast efficiency, can be applied globally over the whole sample of forecasts, or locally on sub-samples, allowing for instabilities in performance over time. Also considered is the appropriate way of testing for forecast rationality or efficiency in a panel context, where the options include testing the aggregate, pooling the observations over individuals and time periods, or running separate regressions for each individual. A key assumption is that individuals weigh over- and under-prediction equally that the loss function is symmetric. This assumption is testable.

References

  1. Bonham, C., & Cohen, R. (2001). To aggregate, pool, or neither: Testing the rational expectations hypothesis using survey data. Journal of Business and Economic Statistics, 190, 278–291.CrossRefGoogle Scholar
  2. Capistrán, C., & Timmermann, A. (2009). Disagreement and biases in inflation expectations. Journal of Money, Credit and Banking, 41, 365–396.CrossRefGoogle Scholar
  3. Chong, Y. Y., & Hendry, D. F. (1986). Econometric evaluation of linear macro-economic models. Review of Economic Studies, 53, 671–690. Reprinted in Granger, C. W. J. (Ed.). (1990). Modelling economic series. Oxford: Clarendon Press.Google Scholar
  4. Christoffersen, P. F., & Diebold, F. X. (1997). Optimal prediction under asymmetric loss. Econometric Theory, 13, 808–817.CrossRefGoogle Scholar
  5. Clements, M. P. (1995). Rationality and the role of judgement in macroeconomic forecasting. Economic Journal, 105, 410–420.CrossRefGoogle Scholar
  6. Clements, M. P. (1997). Evaluating the rationality of fixed-event forecasts. Journal of Forecasting, 16, 225–239.CrossRefGoogle Scholar
  7. Clements, M. P. (2009). Internal consistency of survey respondents’ forecasts: Evidence based on the Survey of Professional Forecasters. In J. L. Castle & N. Shephard (Eds.), The methodology and practice of econometrics. A Festschrift in Honour of David F. Hendry (Chap. 8, pp. 206–226). Oxford: Oxford University Press.Google Scholar
  8. Clements, M. P. (2014). US inflation expectations and heterogeneous loss functions, 1968–2010. Journal of Forecasting, 33(1), 1–14.CrossRefGoogle Scholar
  9. Clements, M. P., & Harvey, D. I. (2009). Forecasting combination and encompassing. In T. C. Mills & K. Patterson (Eds.), Palgrave handbook of econometrics, volume 2: Applied econometrics (pp. 169–198). New York: Palgrave MacMillan.CrossRefGoogle Scholar
  10. Clements, M. P., & Harvey, D. I. (2010). Forecast encompassing tests and probability forecasts. Journal of Applied Econometrics, 25, 1028–1062.CrossRefGoogle Scholar
  11. Croushore, D. (2011a). Forecasting with real-time data vintages. In M. P. Clements & D. F. Hendry (Eds.), The Oxford handbook of economic forecasting (Chap. 9, pp. 247–267). Oxford: Oxford University Press.Google Scholar
  12. Croushore, D. (2011b). Frontiers of real-time data analysis. Journal of Economic Literature, 49, 72–100.CrossRefGoogle Scholar
  13. Diebold, F. X., & Mariano, R. S. (1995). Comparing predictive accuracy. Journal of Business and Economic Statistics, 13, 253–263.Google Scholar
  14. Elliott, G., Komunjer, I., & Timmermann, A. (2005). Estimation and testing of forecast rationality under flexible loss. Review of Economic Studies, 72, 1107–1125.CrossRefGoogle Scholar
  15. Elliott, G., Komunjer, I., & Timmermann, A. (2008). Biases in macroeconomic forecasts: Irrationality or asymmetric loss. Journal of the European Economic Association, 6, 122–157.CrossRefGoogle Scholar
  16. Fair, R. C., & Shiller, R. J. (1989). The informational content of ex ante forecasts. Review of Economics and Statistics, 71, 325–331.CrossRefGoogle Scholar
  17. Faust, J., & Wright, J. H. (2009). Comparing Greenbook and reduced form forecasts using a large realtime dataset. Journal of Business and Economic Statistics, 27(4), 468–479.CrossRefGoogle Scholar
  18. Figlewski, S., & Wachtel, P. (1981). The formation of inflationary expectations. Review of Economics and Statistics, 63, 1–10.CrossRefGoogle Scholar
  19. Figlewski, S., & Wachtel, P. (1983). Rational expectations, informational efficiency, and tests using survey data: A reply. The Review of Economics and Statistics, 65(3), 529–531.CrossRefGoogle Scholar
  20. Fixler, D. J., Greenaway-McGrevy, R., & Grimm, B. T. (2014, August). The revisions to GDP, GDI, and their major components. Survey of Current Business, 1–23.Google Scholar
  21. Giacomini, R. (2011). Testing conditional predictive ability. In M. P. Clements & D. F. Hendry (Eds.), The Oxford handbook of economic (pp. 441–455). Oxford: Oxford University Press.Google Scholar
  22. Giacomini, R., & White, H. (2006). Tests of conditional predictive ability. Econometrica, 74, 1545–1578.CrossRefGoogle Scholar
  23. Giordani, P., & Söderlind, P. (2003). Inflation forecast uncertainty. European Economic Review, 47(6), 1037–1059.CrossRefGoogle Scholar
  24. Granger, C. W. (1969). Prediction with a generalized cost of error function. Operations Research Quarterly, 20, 199–207.CrossRefGoogle Scholar
  25. Granger, C. W. J., & Newbold, P. (1973). Some comments on the evaluation of economic forecasts. Applied Economics, 5, 35–47. Reprinted in Mills, T. C. (Ed.). (1999). Economic forecasting. The International Library of Critical Writings in Economics. Cheltenham: Edward Elgar.Google Scholar
  26. Granger, C. W. J., & Newbold, P. (1977). Forecasting economic time series. New York: Academic Press.Google Scholar
  27. Harvey, D. I., Leybourne, S., & Newbold, P. (1998). Tests for forecast encompassing. Journal of Business and Economic Statistics, 16, 254–259. Reprinted in Mills, T. C. (Ed.). (1999). Economic forecasting. Cheltenham: Edward Elgar.Google Scholar
  28. Hendry, D. F., & Richard, J.-F. (1989). Recent developments in the theory of encompassing. In B. Cornet & H. Tulkens (Eds.), Contributions to operations research and economics. The XXth anniversary of CORE (pp. 393–440). Cambridge, MA: MIT Press. Reprinted in Campos, J., Ericsson, N. R., & Hendry, D. F. (Eds.). (2005). General to specific modelling. Cheltenham: Edward Elgar.Google Scholar
  29. Keane, M. P., & Runkle, D. E. (1990). Testing the rationality of price forecasts: New evidence from panel data. American Economic Review, 80(4), 714–735.Google Scholar
  30. Lahiri, K., & Liu, F. (2009). On the use of density forecasts to identify asymmetry in forecasters’ loss function. Business and Economic Statistics Section - JSM, 2396–2408.Google Scholar
  31. Landefeld, J. S., Seskin, E. P., & Fraumeni, B. M. (2008). Taking the pulse of the economy. Journal of Economic Perspectives, 22, 193–216.CrossRefGoogle Scholar
  32. Mincer, J., & Zarnowitz, V. (1969). The evaluation of economic forecasts. In J. Mincer (Ed.), Economic forecasts and expectations (pp. 3–46). New York: National Bureau of Economic Research.Google Scholar
  33. Mizon, G. E., & Richard, J.-F. (1986). The encompassing principle and its application to non-nested hypothesis tests. Econometrica, 54, 657–678.CrossRefGoogle Scholar
  34. Morgan, W. A. (1939–1940). A test for significance of the difference between the two variances in a sample form a normal bivariate population. Biometrika, 31, 13–19.Google Scholar
  35. Muth, J. F. (1961). Rational expectations and the theory of price movements. Econometrica, 29, 315–335.CrossRefGoogle Scholar
  36. Nelson, C. R. (1972). The prediction performance of the FRB-MIT-PENN model of the US economy. American Economic Review, 62, 902–917. Reprinted in Mills, T. C. (Ed.). (1999). Economic forecasting. The International Library of Critical Writings in Economics. Cheltenham: Edward Elgar.Google Scholar
  37. Nordhaus, W. D. (1987). Forecasting efficiency: Concepts and applications. Review of Economics and Statistics, 69, 667–674.CrossRefGoogle Scholar
  38. Patton, A. J., & Timmermann, A. (2007). Testing forecast optimality under unknown loss. Journal of the American Statistical Association, 102, 1172–1184.CrossRefGoogle Scholar
  39. Patton, A. J., & Timmermann, A. (2012). Forecast rationality tests based on multi-horizon bounds. Journal of Business & Economic Statistics, 30(1), 1–17.CrossRefGoogle Scholar
  40. Pesaran, M. H., & Weale, M. (2006). Survey expectations. In G. Elliott, C. Granger, & A. Timmermann (Eds.), Handbook of economic forecasting, volume 1. Handbook of economics 24 (pp. 715–776). Amsterdam: Elsevier, Horth-Holland.CrossRefGoogle Scholar
  41. Romer, C. D., & Romer, D. H. (2000). Federal Reserve information and the behaviour of interest rates. American Economic Review, 90(3), 429–457.CrossRefGoogle Scholar
  42. Rossi, B., & Sekhposyan, T. (2016). Forecast rationality tests in the presence of instabilities, with applications to Federal Reserve and Survey Forecasts. Journal of Applied Econometrics, 31(3), 507–532.CrossRefGoogle Scholar
  43. Varian, H. R. (1975). A Bayesian approach to real estate assessment. In S. E. Fienberg & A. Zellner (Eds.), Studies in Bayesian econometrics and statistics in honor of Leonard J. Savage (pp. 195–208). Amsterdam: North Holland.Google Scholar
  44. West, K. D. (1996). Asymptotic inference about predictive ability. Econometrica, 64, 1067–1084.CrossRefGoogle Scholar
  45. West, K. D. (2001). Tests for forecast encompassing when forecasts depend on estimated regression parameters. Journal of Business and Economic Statistics, 19, 29–33.CrossRefGoogle Scholar
  46. West, K. D., & McCracken, M. W. (1998). Regression-based tests of predictive ability. International Economic Review, 39, 817–840.CrossRefGoogle Scholar
  47. Zarnowitz, V. (1985). Rational expectations and macroeconomic forecasts. Journal of Business and Economic Statistics, 3(4), 293–311.Google Scholar
  48. Zellner, A. (1986). Biased predictors, rationality and the evaluation of forecasts. Economics Letters, 21, 45–48.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2019

Authors and Affiliations

  • Michael P. Clements
    • 1
  1. 1.ICMA Centre, Henley Business SchoolUniversity of ReadingWheatleyUK

Personalised recommendations