arXiv subjects

Pierre Tandeo

Publications and source records attributed to Pierre Tandeo.

15 recordsLinked to original sources

arXiv2025

Neural variational Data Assimilation with Uncertainty Quantification using SPDE priors

The spatio-temporal interpolation of large geophysical datasets has historically been addressed by Optimal Interpolation (OI) and more sophisticated equation-based or data-driven Data Assimilation (DA) techniques. Recent advances in the deep learning community enables to address the interpolation problem through a neural architecture incorporating a variational data assimilation framework. The reconstruction task is seen as a joint learning problem of the prior involved in the variational inner cost, seen as a projection operator of the state, and the gradient-based minimization of the latter. Both prior models and solvers are stated as neural networks with automatic differentiation which can be trained by minimizing a loss function, typically the mean squared error between some ground truth and the reconstruction. Such a strategy turns out to be very efficient to improve the mean state estimation, but still needs complementary developments to quantify its related uncertainty. In this work, we use the theory of Stochastic Partial Differential Equations (SPDE) and Gaussian Processes (GP) to estimate both space-and time-varying covariance of the state. Our neural variational scheme is modified to embed an augmented state formulation with both state and SPDE parametrization to estimate. We demonstrate the potential of the proposed framework on a spatio-temporal GP driven by diffusion-based anisotropies and on realistic Sea Surface Height (SSH) datasets. We show how our solution reaches the OI baseline in the Gaussian case. For nonlinear dynamics, as almost always stated in DA, our solution outperforms OI, while allowing for fast and interpretable online parameter estimation.

stat.ML↗

arXiv2023

Rain regime segmentation of Sentinel-1 observation learning from NEXRAD collocations with Convolution Neural Networks

Remote sensing of rainfall events is critical for both operational and scientific needs, including for example weather forecasting, extreme flood mitigation, water cycle monitoring, etc. Ground-based weather radars, such as NOAA's Next-Generation Radar (NEXRAD), provide reflectivity and precipitation estimates of rainfall events. However, their observation range is limited to a few hundred kilometers, prompting the exploration of other remote sensing methods, particularly over the open ocean, that represents large areas not covered by land-based radars. Here we propose a deep learning approach to deliver a three-class segmentation of SAR observations in terms of rainfall regimes. SAR satellites deliver very high resolution observations with a global coverage. This seems particularly appealing to inform fine-scale rain-related patterns, such as those associated with convective cells with characteristic scales of a few kilometers. We demonstrate that a convolutional neural network trained on a collocated Sentinel-1/NEXRAD dataset clearly outperforms state-of-the-art filtering schemes such as the Koch's filters. Our results indicate high performance in segmenting precipitation regimes, delineated by thresholds at 24.7, 31.5, and 38.8 dBZ. Compared to current methods that rely on Koch's filters to draw binary rainfall maps, these multi-threshold learning-based models can provide rainfall estimation. They may be of interest in improving high-resolution SAR-derived wind fields, which are degraded by rainfall, and provide an additional tool for the study of rain cells.

cs.CV↗

arXiv2023

Reduction of rain-induced errors for wind speed estimation on SAR observations using convolutional neural networks

Synthetic Aperture Radar is known to be able to provide high-resolution estimates of surface wind speed. These estimates usually rely on a Geophysical Model Function (GMF) that has difficulties accounting for non-wind processes such as rain events. Convolutional neural network, on the other hand, have the capacity to use contextual information and have demonstrated their ability to delimit rainfall areas. By carefully building a large dataset of SAR observations from the Copernicus Sentinel-1 mission, collocated with both GMF and atmospheric model wind speeds as well as rainfall estimates, we were able to train a wind speed estimator with reduced errors under rain. Collocations with in-situ wind speed measurements from buoys show a root mean square error that is reduced by 27% (resp. 45%) under rainfall estimated at more than 1 mm/h (resp. 3 mm/h). These results demonstrate the capacity of deep learning models to correct rain-related errors in SAR products.

cs.CV↗

arXiv2023

Online machine-learning forecast uncertainty estimation for sequential data assimilation

Quantifying forecast uncertainty is a key aspect of state-of-the-art numerical weather prediction and data assimilation systems. Ensemble-based data assimilation systems incorporate state-dependent uncertainty quantification based on multiple model integrations. However, this approach is demanding in terms of computations and development. In this work a machine learning method is presented based on convolutional neural networks that estimates the state-dependent forecast uncertainty represented by the forecast error covariance matrix using a single dynamical model integration. This is achieved by the use of a loss function that takes into account the fact that the forecast errors are heterodastic. The performance of this approach is examined within a hybrid data assimilation method that combines a Kalman-like analysis update and the machine learning based estimation of a state-dependent forecast error covariance matrix. Observing system simulation experiments are conducted using the Lorenz'96 model as a proof-of-concept. The promising results show that the machine learning method is able to predict precise values of the forecast covariance matrix in relatively high-dimensional states. Moreover, the hybrid data assimilation method shows similar performance to the ensemble Kalman filter outperforming it when the ensembles are relatively small.

physics.ao-ph↗

arXiv2023

Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review

Data Assimilation (DA) and Uncertainty quantification (UQ) are extensively used in analysing and reducing error propagation in high-dimensional spatial-temporal dynamics. Typical applications span from computational fluid dynamics (CFD) to geoscience and climate systems. Recently, much effort has been given in combining DA, UQ and machine learning (ML) techniques. These research efforts seek to address some critical challenges in high-dimensional dynamical systems, including but not limited to dynamical system identification, reduced order surrogate modelling, error covariance specification and model error correction. A large number of developed techniques and methodologies exhibit a broad applicability across numerous domains, resulting in the necessity for a comprehensive guide. This paper provides the first overview of the state-of-the-art researches in this interdisciplinary field, covering a wide range of applications. This review aims at ML scientists who attempt to apply DA and UQ techniques to improve the accuracy and the interpretability of their models, but also at DA and UQ experts who intend to integrate cutting-edge ML approaches to their systems. Therefore, this article has a special focus on how ML methods can overcome the existing limits of DA and UQ, and vice versa. Some exciting perspectives of this rapidly developing research field are also discussed.

cs.LG↗

arXiv2022

Evaluation of Machine Learning Techniques for Forecast Uncertainty Quantification

Ensemble forecasting is, so far, the most successful approach to produce relevant forecasts with an estimation of their uncertainty. The main limitations of ensemble forecasting are the high computational cost and the difficulty to capture and quantify different sources of uncertainty, particularly those associated with model errors. In this work we perform toy-model and state-of-the-art model experiments to analyze to what extent artificial neural networks (ANNs) are able to model the different sources of uncertainty present in a forecast. In particular those associated with the accuracy of the initial conditions and those introduced by the model error. We also compare different training strategies: one based on a direct training using the mean and spread of an ensemble forecast as target, the other ones rely on an indirect training strategy using an analyzed state as target in which the uncertainty is implicitly learned from the data. Experiments using the Lorenz'96 model show that the ANNs are able to emulate some of the properties of ensemble forecasts like the filtering of the most unpredictable modes and a state-dependent quantification of the forecast uncertainty. Moreover, ANNs provide a reliable estimation of the forecast uncertainty in the presence of model error. Preliminary experiments conducted with a state-of-the-art forecasting system also confirm the ability of ANNs to produce a reliable quantification of the forecast uncertainty.

cs.LG↗

arXiv2022

Two-dimensional structure functions to characterize convective rolls in the marine atmospheric boundary layer from Sentinel-1 SAR images

We study the shape of convective rolls in the Marine Atmospheric Boundary Layer from Synthetic Aperture Radar images of the ocean. We propose a multiscale analysis with structure functions which allow an easy generalization to analyse high-order statistics and so to finely describe the shape of the rolls. The two main results are : 1) second order structure function characterizes the size and direction of rolls just like correlation or power spectrum do, 2) high order statistics can be studied with skewness and Flatness which characterize the asymmetry and intermittency of rolls respectively. From the best of our knowledge, this is the first time that the asymmetry and intermittency of rolls is shown from radar images of the ocean surface.

cs.IT↗

arXiv2021

Probability distributions for analog-to-target distances

Some properties of chaotic dynamical systems can be probed through features of recurrences, also called analogs. In practice, analogs are nearest neighbours of the state of a system, taken from a large database called the catalog. Analogs have been used in many atmospheric applications including forecasts, downscaling, predictability estimation, and attribution of extreme events. The distances of the analogs to the target state condition the performances of analog applications. These distances can be viewed as random variables, and their probability distributions can be related to the catalog size and properties of the system at stake. A few studies have focused on the first moments of return time statistics for the best analog, fixing an objective of maximum distance from this analog to the target state. However, for practical use and to reduce estimation variance, applications usually require not just one, but many analogs. In this paper, we evaluate from a theoretical standpoint and with numerical experiments the probability distributions of the $K$-best analog-to-target distances. We show that dimensionality plays a role on the size of the catalog needed to find good analogs, and also on the relative means and variances of the $K$-best analogs. Our results are based on recently developed tools from dynamical systems theory. These findings are illustrated with numerical simulations of a well-known chaotic dynamical system and on 10m-wind reanalysis data in north-west France. A practical application of our derivations for the purpose of objective-based dimension reduction is shown using the same reanalysis data.

math.DS↗

arXiv2020

A Review of Innovation-Based Methods to Jointly Estimate Model and Observation Error Covariance Matrices in Ensemble Data Assimilation

Data assimilation combines forecasts from a numerical model with observations. Most of the current data assimilation algorithms consider the model and observation error terms as additive Gaussian noise, specified by their covariance matrices Q and R, respectively. These error covariances, and specifically their respective amplitudes, determine the weights given to the background (i.e., the model forecasts) and to the observations in the solution of data assimilation algorithms (i.e., the analysis). Consequently, Q and R matrices significantly impact the accuracy of the analysis. This review aims to present and to discuss, with a unified framework, different methods to jointly estimate the Q and R matrices using ensemble-based data assimilation techniques. Most of the methodologies developed to date use the innovations, defined as differences between the observations and the projection of the forecasts onto the observation space. These methodologies are based on two main statistical criteria: (i) the method of moments, in which the theoretical and empirical moments of the innovations are assumed to be equal, and (ii) methods that use the likelihood of the observations, themselves contained in the innovations. The reviewed methods assume that innovations are Gaussian random variables, although extension to other distributions is possible for likelihood-based methods. The methods also show some differences in terms of levels of complexity and applicability to high-dimensional systems. The conclusion of the review discusses the key challenges to further develop estimation methods for Q and R. These challenges include taking into account time-varying error covariances, using limited observational coverage, estimating additional deterministic error terms, or accounting for correlated noises.

stat.ME↗

arXiv2020

Model error covariance estimation in particle and ensemble Kalman filters using an online expectation-maximization algorithm

The performance of ensemble-based data assimilation techniques that estimate the state of a dynamical system from partial observations depends crucially on the prescribed uncertainty of the model dynamics and of the observations. These are not usually known and have to be inferred. Many approaches have been proposed to tackle this problem, including fully Bayesian, likelihood maximization and innovation-based techniques. This work focuses on maximization of the likelihood function via the expectation-maximization (EM) algorithm to infer the model error covariance combined with ensemble Kalman filters and particle filters to estimate the state. The classical application of the EM algorithm in a data assimilation context involves filtering and smoothing a fixed batch of observations in order to complete a single iteration. This is an inconvenience when using sequential filtering in high-dimensional applications. Motivated by this, an adaptation of the algorithm that can process observations and update the parameters on the fly, with some underlying simplifications, is presented. The proposed technique was evaluated and achieved good performance in experiments with the Lorenz-63 and the 40-variable Lorenz-96 dynamical systems designed to represent some common scenarios in data assimilation such as non-linearity, chaoticity and model misspecification.

stat.CO↗

arXiv2019

Coupling Oceanic Observation Systems to Study Mesoscale Ocean Dynamics

Understanding local currents in the North Atlantic region of the ocean is a key part of modelling heat transfer and global climate patterns. Satellites provide a surface signature of the temperature of the ocean with a high horizontal resolution while in situ autonomous probes supply high vertical resolution, but horizontally sparse, knowledge of the ocean interior thermal structure. The objective of this paper is to develop a methodology to combine these complementary ocean observing systems measurements to obtain a three-dimensional time series of ocean temperatures with high horizontal and vertical resolution. Within an observation-driven framework, we investigate the extent to which mesoscale ocean dynamics in the North Atlantic region may be decomposed into a mixture of dynamical modes, characterized by different local regressions between Sea Surface Temperature (SST), Sea Level Anomalies (SLA) and Vertical Temperature fields. Ultimately we propose a Latent-class regression method to improve prediction of vertical ocean temperature.

physics.ao-ph↗

arXiv2018

An efficient particle-based method for maximum likelihood estimation in nonlinear state-space models

Data assimilation methods aim at estimating the state of a system by combining observations with a physical model. When sequential data assimilation is considered, the joint distribution of the latent state and the observations is described mathematically using a state-space model, and filtering or smoothing algorithms are used to approximate the conditional distribution of the state given the observations. The most popular algorithms in the data assimilation community are based on the Ensemble Kalman Filter and Smoother (EnKF/EnKS) and its extensions. In this paper we investigate an alternative approach where a Conditional Particle Filter (CPF) is combined with Backward Simulation (BS). This allows to explore efficiently the latent space and simulate quickly relevant trajectories of the state conditionally to the observations. We also tackle the difficult problem of parameter estimation. Indeed, the models generally involve statistical parameters in the physical models and/or in the stochastic models for the errors. These parameters strongly impact the results of the data assimilation algorithm and there is a need for an efficient method to estimate them. Expectation-Maximization (EM) is the most classical algorithm in the statistical literature to estimate the parameters in models with latent variables. It consists in updating sequentially the parameters by maximizing a likelihood function where the state is approximated using a smoothing algorithm. In this paper, we propose an original Stochastic Expectation-Maximization (SEM) algorithm combined to the CPF-BS smoother to estimate the statistical parameters. We show on several toy models that this algorithm provides, with reasonable computational cost, accurate estimations of the statistical parameters and the state in highly nonlinear state-space models, where the application of EM algorithms using EnKS is limited. We also provide a Python source code of the algorithm.

stat.ME↗

arXiv2017

EddyNet: A Deep Neural Network For Pixel-Wise Classification of Oceanic Eddies

This work presents EddyNet, a deep learning based architecture for automated eddy detection and classification from Sea Surface Height (SSH) maps provided by the Copernicus Marine and Environment Monitoring Service (CMEMS). EddyNet is a U-Net like network that consists of a convolutional encoder-decoder followed by a pixel-wise classification layer. The output is a map with the same size of the input where pixels have the following labels \{'0': Non eddy, '1': anticyclonic eddy, '2': cyclonic eddy\}. We investigate the use of SELU activation function instead of the classical ReLU+BN and we use an overlap based loss function instead of the cross entropy loss. Keras Python code, the training datasets and EddyNet weights files are open-source and freely available on https://github.com/redouanelg/EddyNet.

cs.CV↗

arXiv2017

Stochastic parameterization identification using ensemble Kalman filtering combined with expectation-maximization and Newton-Raphson maximum likelihood methods

For modelling geophysical systems, large-scale processes are described through a set of coarse-grained dynamical equations while small-scale processes are represented via parameterizations. This work proposes a method for identifying the best possible stochastic parameterization from noisy data. State-the-art sequential estimation methods such as Kalman and particle filters do not achieve this goal succesfully because both suffer from the collapse of the parameter posterior distribution. To overcome this intrinsic limitation, we propose two statistical learning methods. They are based on the combination of two methodologies: the maximization of the likelihood via Expectation-Maximization (EM) and Newton-Raphson (NR) algorithms which are mainly applied in the statistic and machine learning communities, and the ensemble Kalman filter (EnKF). The methods are derived using a Bayesian approach for a hidden Markov model. They are applied to infer deterministic and stochastic physical parameters from noisy observations in coarse-grained dynamical models. Numerical experiments are conducted using the Lorenz-96 dynamical system with one and two scales as a proof-of-concept. The imperfect coarse-grained model is modelled through a one-scale Lorenz-96 system in which a stochastic parameterization is incorpored to represent the small-scale dynamics. The algorithms are able to identify an optimal stochastic parameterization with a good accuracy under moderate observational noise. The proposed EnKF-EM and EnKF-NR are promising statistical learning methods for developing stochastic parameterizations in high-dimensional geophysical models.

physics.ao-ph↗

arXiv2015

DADA: Data Assimilation for the Detection and Attribution of Weather- and Climate-related Events

We describe a new approach allowing for systematic causal attribution of weather and climate-related events, in near-real time. The method is purposely designed to facilitate its implementation at meteorological centers by relying on data treatments that are routinely performed when numerically forecasting the weather. Namely, we show that causal attribution can be obtained as a by-product of so-called data assimilation procedures that are run on a daily basis to update the meteorological model with new atmospheric observations; hence, the proposed methodology can take advantage of the powerful computational and observational capacity of weather forecasting centers. We explain the theoretical rationale of this approach and sketch the most prominent features of a "data assimilation-based detection and attribution" (DADA) procedure. The proposal is illustrated in the context of the classical three-variable Lorenz model with additional forcing. Several theoretical and practical research questions that need to be addressed to make the proposal readily operational within weather forecasting centers are finally laid out.

stat.AP↗