Ensemble size effects on conditional reliability estimates: slope attenuation bias and correction methods

Abstract

The goal of ensemble forecasting is to maximise sharpness subject to reliability. Marginal reliability means that, over all cases, the ensemble is statistically consistent with reality: the ensemble mean is unbiased, the expected ensemble variance equals the expected mean-squared error of the ensemble mean, and the variance of the ensemble members matches the variance of the truth. Equivalently, forecasts that assign probability $p$ to an event verify with relative frequency $p$. However, climatological consistency is not sufficient for users acting on individual forecasts. A natural extension is to assess reliability conditional on the forecast itself, by examining whether, on average, larger ensemble means imply larger observed values, larger spreads imply larger forecast errors, or higher probabilities imply higher event frequencies. This motivates conditional reliability diagnostics such as reliability diagrams and spread-error relationships. Here we show that conditional reliability diagnostics are systematically biased for finite ensemble sizes. We present a unified framework for slope attenuation caused by finite-ensemble sampling noise, which affects conditional diagnostics for ensemble means, spreads, and probabilities. Using synthetic forecasts that are perfectly reliable by construction, we isolate finite-ensemble effects. We derive analytical expressions for the expected attenuation and propose practical estimators computable directly from ensemble data. The framework is illustrated using 2-metre temperature sub-seasonal ensemble forecasts from ECMWF, where finite-ensemble slope attenuation substantially affects the spread-error relationship and tercile-based reliability diagrams. These results demonstrate that attenuated conditional slopes should not be interpreted as evidence of forecast deficiencies unless finite-ensemble effects are explicitly taken into account.