Ensemble reliability and the signal-to-noise paradox in ECMWF subseasonal forecasts

Abstract

Ensemble forecasts can exhibit counterintuitive statistical properties such that the correlation between ensemble means and observations ($r_{mo}$) exceeds the correlation between ensemble means and individual members ($r_{mm}$). This behaviour has been interpreted as a `signal-to-noise paradox' (SNP), which is commonly diagnosed using the ratio of predictable components ($\textnormal{RPC} = \sqrt { r_{mo}^2 / r_{mm}^2 } $). Here, we emphasise the links between ensemble-size-invariant estimates of RPC and other metrics of ensemble reliability and derive a general closed-form expression for RPC in terms of $r_{mo}$, the spread-error ratio (SER), and total variance ratio (VR). Physical constraints on the admissible solutions provide a mechanism to identify statistically paradoxical sample estimates of RPC, $r_{mo}$, SER, and VR that correspond to combinations that are not possible without sampling uncertainty. We evaluate three atmospheric circulation indices in ECMWF subseasonal reforecasts. Large-ensemble NAO forecasts evaluated over 80 start dates satisfy reliability criteria within our estimated sampling uncertainties but exhibit high RPC values at some lead times. These lead times coincide with paradoxical combinations of correlation and reliability metrics that are impossible in the large-sample limit, indicating an important role for sampling uncertainties. Nevertheless, wintertime NAO indices averaged over days 16-45 exhibit more robust evidence for unreliability characterised by RPC$approx1.5$ suggesting that SNP-like behaviour observed in daily data during the period 2001-2020 is not solely attributable to sampling artefacts. However, these results do not generalise to other configurations of the same IFS model evaluated over 3120 start dates for the period 1959-2023. In these extended reforecasts, daily NAO indices are well-calibrated and RPC$\approx1$ for all lead times.