On equitable scoring functions and optimal forecasting behaviour

Abstract

Equitable scoring functions have a long history in meteorological forecast verification and have recently gained renewed prominence through the use of the Stable Equitable Error in Probability Space (SEEPS) score for evaluating precipitation forecasts from numerical and machine-learning weather prediction systems. This paper provides a systematic analysis of equitable scoring functions through the optimal forecasts they induce. We introduce the generalized diagonal score, which defines a broad class of equitable scoring functions that includes, up to equivalence, the SEEPS, Gerrity, Peirce, Barnston and diagonal scores. Within this framework, we show that optimal single-valued and categorical forecasts are characterized by crossing points between the predictive and climatological cumulative distribution functions. Moreover, we show that the generalized diagonal score, when evaluating predictive distributions, is proper but not strictly proper, and is insensitive to substantial forecast misspecification away from crossing points. The framework also yields scoring rules that are both proper and equitable for probabilistic forecasts with categorical outcomes. For single-valued and categorical forecasts, the crossing-point characterization shows that optimal forecasts can vary across climatologies even when the underlying predictive distribution is unchanged. Consequently, identical predictive distributions may lead to substantially different optimal forecasts under equitable scores, with important implications for assessing the suitability of such scores for any given application.