arXiv subjects

Jeffrey S. Whitaker

Publications and source records attributed to Jeffrey S. Whitaker.

3 recordsLinked to original sources

arXiv2026

Long-window 4DVar for reanalysis using a differentiable weather model

Atmospheric reanalyses combine observations with model forecasts using complex data assimilation systems. We test whether a differentiable weather model permits a simpler and more accurate method based on a long-window four-dimensional variational data assimilation (4D-Var) formulation that omits the conventional background-error term. The method uses automatic differentiation to find optimal NeuralGCM initial conditions that minimize the misfit to real surface-pressure observations distributed across overlapping windows of two to seven days, assuming no model error. Cycling at 6-hour intervals for three months beginning 1 January 2015 yields a stable reanalysis with smaller error relative to ERA5 in 500-hPa geopotential height than the Twentieth Century Reanalysis version 3 (20CRv3), which uses an ensemble Kalman filter to assimilate the same observations. Every window produces smaller errors than 20CRv3, with analysis error for the four-day window approximately 55% smaller than for 20CRv3. At the end of the four-day window, which does not benefit from future observations, error remains approximately 38% smaller than 20CRv3. Analyses degrade slightly beyond four days, which we attribute to the increasing importance of model error.

physics.ao-ph↗

arXiv2024

Assimilating Observed Surface Pressure into ML Weather Prediction Models

There has been a recent surge in development of accurate machine learning (ML) weather prediction models, but evaluation of these models has mainly been focused on medium-range forecasts, not their performance in cycling data assimilation (DA) systems. Cycling DA provides a statistically optimal estimate of model initial conditions, given observations and previous model forecasts. Here, real surface pressure observations are assimilated into several popular ML models using an ensemble Kalman filter, where accurate ensemble covariance estimation is essential to constrain unobserved state variables from sparse observations. In this cycling DA system, deterministic ML models accumulate small-scale noise until they diverge. Mitigating this noise with a spectral filter can stabilize the system, but with larger errors than traditional models. Perturbation experiments illustrate that these models do not accurately represent short-term error growth, leading to poor estimation of cross-variable covariances.

physics.ao-ph↗

arXiv2024

Nonlinear ensemble filtering with diffusion models: Application to the surface quasi-geostrophic dynamics

The intersection between classical data assimilation methods and novel machine learning techniques has attracted significant interest in recent years. Here we explore another promising solution in which diffusion models are used to formulate a robust nonlinear ensemble filter for sequential data assimilation. Unlike standard machine learning methods, the proposed \textit{Ensemble Score Filter (EnSF)} is completely training-free and can efficiently generate a set of analysis ensemble members. In this study, we apply the EnSF to a surface quasi-geostrophic model and compare its performance against the popular Local Ensemble Transform Kalman Filter (LETKF), which makes Gaussian assumptions on the posterior distribution. Numerical tests demonstrate that EnSF maintains stable performance in the absence of localization and for a variety of experimental settings. We find that EnSF achieves competitive performance relative to LETKF in the case of linear observations, but leads to significant advantages when the state is nonlinearly observed and the numerical model is subject to unexpected shocks. A spectral decomposition of the analysis results shows that the largest improvements over LETKF occur at large scales (small wavenumbers) where LETKF lacks sufficient ensemble spread. Overall, this initial application of EnSF to a geophysical model of intermediate complexity is very encouraging, and motivates further developments of the algorithm for more realistic problems.

math-ph↗