"What is a realistic forecast?" Assessing data-driven weather forecasts, a journey from verification to falsification

Abstract

The artificial intelligence revolution is fuelling a paradigm shift in weather forecasting: forecasts are generated with machine learning models trained on large datasets rather than with physics-based numerical models that solve partial differential equations. This new approach proved successful in improving forecast performance as measured with standard verification metrics such as the root mean squared error. At the same time, the realism of data-driven weather forecasts is often questioned and considered an Achilles' heel of machine learning models. How forecast realism can be defined and how this forecast attribute can be assessed are the two questions simultaneously addressed here. Inspired by the seminal work of Murphy (1993) on the definition of forecast goodness, we identify 3 types of realism: a functional realism measured by scoring functions, a structural realism related to the statistical characteristics of the forecasts, and a physical realism that is apprehended through the lenses of our scientific knowledge. This conceptual setting serves as a basis for the design of a new framework for the evaluation of data-driven weather models where falsification arises as a complementary process to the well-established diagnostic and verification tasks.