Bias adjustment and ensemble recalibration methods for seasonal forecasting: a comprehensive intercomparison using the C3S dataset

Abstract

This work presents a comprehensive intercomparison of different alternatives for the calibration of seasonal forecasts, ranging from simple bias adjustment (BA)—e.g. quantile mapping—to more sophisticated ensemble recalibration (RC) methods—e.g. non-homogeneous Gaussian regression, which build on the temporal correspondence between the climate model and the corresponding observations to generate reliable predictions. To be as critical as possible, we validate the raw model and the calibrated forecasts in terms of a number of metrics which take into account different aspects of forecast quality (association, accuracy, discrimination and reliability). We focus on one-month lead forecasts of precipitation and temperature from four state-of-the-art seasonal forecasting systems, three of them included in the Copernicus Climate Change Service dataset (ECMWF-SEAS5, UK Met Office-GloSea5 and Météo France-System5) for boreal winter and summer over two illustrative regions with different skill characteristics (Europe and Southeast Asia). Our results indicate that both BA and RC methods effectively correct the large raw model biases, which is of paramount importance for users, particularly when directly using the climate model outputs to run impact models, or when computing climate indices depending on absolute values/thresholds. However, except for particular regions and/or seasons (typically with high skill), there is only marginal added value—with respect to the raw model outputs—beyond this bias removal. For those cases, RC methods can outperform BA ones, mostly due to an improvement in reliability. Finally, we also show that whereas an increase in the number of members only modestly affects the results obtained from calibration, longer hindcast periods lead to improved forecast quality, particularly for RC methods.

Appendix: Description of BA and RC methods

All the methods described in this section have been applied gridbox by gridbox considering seasonal interannual series. We use the following notation: $y_{m,t}$ and $y'_{m,t}$ denote the original and calibrated values for the ensemble member m at time (season/year) t, ${\hat{y}}$ is the average of the ensemble mean (${\bar{y}}_t$) on all times t, ${\hat{o}}$ is the average of the observations on all times t, $\sigma _{f}$ is the standard deviation of the complete ensemble (pooling all member interannual time-series) and $\sigma _o$ is the standard deviation of the observed interannual time-series. Finally, $\rho$ is the interannual correlation between the ensemble mean and the observational reference.

1.1 Mean (and variance) adjustment (MVA)

This is the simplest adjustment method, with a long tradition in the context of seasonal forecasting (see, e.g., Leung et al. 1999). The ensemble mean and variance are adjusted towards the corresponding observational ones in the following form:

$$\begin{aligned} y'_{m,t} = (y_{m,t}-{\hat{y}})\frac{\sigma _{o}}{\sigma _{f}} + {\hat{o}} \end{aligned}$$

(1)

A simpler version consists of correcting just the mean (MA) and has the same formulation, but excluding the term $\sigma _{o}/\sigma _{f}$.

1.2 Empirical quantile mapping (EQM)

We have considered an empirical quantile mapping (EQM) method participating in the VALUE downscaling intercomparison initiative (Gutiérrez et al. 2018) which has been recently applied to correct seasonal precipitation forecasts (Manzanas et al. 2018; Manzanas and Gutiérrez 2018). This method calibrates the predicted empirical probability density function (PDF) by adjusting a number of quantiles based on the empirical observed PDF (Déqué 2007). In particular, here we adjust percentiles 1–99 and linearly interpolate every two consecutive percentiles inside this range. Outside this range, a constant extrapolation (using the correction obtained for the 1st or 99th percentile) is applied. This method was applied here at a ensemble-wise level; that is, the mapping was trained based on all contributing members which were pooled together (all members are supposed to be statistically indistinguishable). Then, the so-obtained unique correction factor was applied to each individual member. Note that ensemble- and member-wise approaches have been recently reported to provide very similar results (Manzanas et al. 2018).

1.3 Climate conserving recalibration (CCR)

Also known as variance inflation, this method was first introduced in Doblas-Reyes et al. (2005). It modifies the predictions to have the same interannual variance as the observational reference, while preserving their interannual correlation, and can be expressed as:

$$\begin{aligned} y'_{m,t} = \rho \frac{\sigma _{o}}{std({\bar{y}}_{t})} {\bar{y}}_{t} + \sqrt{1-\rho ^{2}} \frac{\sigma _{o}}{\sigma _{f}} (y_{m,t}-{\bar{y}}_{t}) + {\hat{o}} \end{aligned}$$

(2)

After Weigel et al. (2009), this method has been commonly referred to as climate conserving recalibration.

1.4 Ratio of predictable components (RPC)

We have also considered for this work the method introduced by Eade et al. (2014), which uses the ensemble to reduce noise and adjust the forecast variance so that the ratio of predictable components in the model and in the observations is the same (see the paper for details). In particular, they applied the following correction to adjust seasonal forecasts of the North Atlantic Oscillation (NAO), temperature and pressure in the North Atlantic region:

$$\begin{aligned} y'_{m,t}& {}= \rho \frac{\sigma _{o} }{std({\bar{y}}_{t})} ({\bar{y}}_{t}-{\hat{y}}) \nonumber \\&\quad+\, \sqrt{1-\rho ^{2}} \frac{\sigma _{o}}{\sqrt{var(y_{m,t}-{\bar{y}}_{t})}} (y_{m,t}-{\bar{y}}_{t}) + {\hat{o}} \end{aligned}$$

(3)

1.5 Linear regression recalibration (LR)

This method performs a linear regression between the ensemble mean (i.e. the time-series of ${\bar{y}}_{t}$) and the corresponding observations:

$$\begin{aligned} o_{t} = \alpha + \beta {\bar{y}}_{t} + \epsilon \end{aligned}$$

(4)

To correct the forecast variance, the standardized anomalies are rescaled by the standard deviation of the predictive distribution from the linear fit, so $y'_{m,t} = \alpha + \beta {\bar{y}}_{t} + \gamma _{t}(y_{m,t}-{\bar{y}}_{t})$, where

$$\begin{aligned} \gamma _{t} = std(\epsilon _{fit}) \sqrt{1+1/n+\frac{(y_{t}-{\bar{y}}_{t})^{2}}{(n-1)var(\epsilon _{obs})}}, \end{aligned}$$

(5)

$\epsilon _{fit}$ and $\epsilon _{obs}$ are the residuals from the regression and the observations respectively, and n the number of samples used.

1.6 Non-homogeneous Gaussian Regression (NGR)

This method (Gneiting et al. 2005) uses a constant term and the ensemble mean signal as predictors for the calibrated forecast mean and a constant term and the ensemble spread for the inflation (shrinkage) of the ensemble spread. The correction has the following form:

$$\begin{aligned} y'_{m,t} = \alpha + \beta ({\bar{y}}_{t}-{\hat{y}}) + \sqrt{\gamma ^{2}+\delta ^{2}var(y_{t})} (y_{m,t}-{\bar{y}}_{t}) \end{aligned}$$

(6)

The parameters $\alpha$, $\beta$, $\gamma$ and $\delta$ are optimized by minimizing the ensemble CRPS. NGR approaches have been applied in many previous works, but mostly in the context of short-term forecasts (see, e.g., Wilks and Hamill 2007; Thorarinsdottir and Johnson 2012; Feldmann et al. 2015; Scheuerer and Möller 2015; Markus et al. 2017). To our knowledge, only Tippett and Barnston (2008) have used it in the context of seasonal forecasting.