arXiv subjects

Ibrahim Hoteit

Publications and source records attributed to Ibrahim Hoteit.

At least 19 recordsLinked to original sources

arXiv2026

A Structurally Localized Ensemble Kalman Filtering Approach

State-of-the-art ensemble Kalman filtering (EnKF) algorithms require incorporating localization techniques to cope with the rank deficiency and the inherited spurious correlations in their error covariance matrices. Localization techniques are mostly ad-hoc, based on some distances between the state and observation variables, requiring demanding manual tuning. This work introduces a new ensemble filtering approach, which is inherently localized, avoiding the need for any auxiliary localization technique. Instead of explicitly applying localization on ensembles, the idea is to first localize the continuous analysis probability density function (pdf) before ensemble sampling. The localization of the analysis pdf is performed through an approximation by a product of independent marginal pdfs corresponding to small partitions of the state vector, using the variational Bayesian optimization. These marginals are then sampled following stochastic EnKF and deterministic ensemble transform Kalman filtering (ETKF) procedures, using ensembles larger than the partitions' size. The resulting filters involve the same forecast steps as their standard EnKF and ETKF counterparts but different analysis steps, iteratively adjusting the EnKF and ETKF updates of each partition based on the ensemble means of the other partitions. Numerical experiments are conducted with the Lorenz-96 model under different scenarios to demonstrate the potential of the proposed filters. The new filters' performances are comparable to those of the EnKF and ETKF with already tuned localization, both in terms of computational burden and estimation accuracy.

physics.ao-ph↗

arXiv2025

GraphComp: Extreme Error-bounded Compression of Scientific Data via Temporal Graph Autoencoders

The generation of voluminous scientific data poses significant challenges for efficient storage, transfer, and analysis. Recently, error-bounded lossy compression methods emerged due to their ability to achieve high compression ratios while controlling data distortion. However, they often overlook the inherent spatial and temporal correlations within scientific data, thus missing opportunities for higher compression. In this paper we propose GRAPHCOMP, a novel graph-based method for error-bounded lossy compression of scientific data. We perform irregular segmentation of the original grid data and generate a graph representation that preserves the spatial and temporal correlations. Inspired by Graph Neural Networks (GNNs), we then propose a temporal graph autoencoder to learn latent representations that significantly reduce the size of the graph, effectively compressing the original data. Decompression reverses the process and utilizes the learnt graph model together with the latent representation to reconstruct an approximation of the original data. The decompressed data are guaranteed to satisfy a user-defined point-wise error bound. We compare our method against the state-of-the-art error-bounded lossy methods (i.e., HPEZ, SZ3.1, SPERR, and ZFP) on large-scale real and synthetic data. GRAPHCOMP consistently achieves the highest compression ratio across most datasets, outperforming the second-best method by margins ranging from 22% to 50%.

cs.LG↗

arXiv2024

Data Assimilation in Chaotic Systems Using Deep Reinforcement Learning

Data assimilation (DA) plays a pivotal role in diverse applications, ranging from climate predictions and weather forecasts to trajectory planning for autonomous vehicles. A prime example is the widely used ensemble Kalman filter (EnKF), which relies on linear updates to minimize variance among the ensemble of forecast states. Recent advancements have seen the emergence of deep learning approaches in this domain, primarily within a supervised learning framework. However, the adaptability of such models to untrained scenarios remains a challenge. In this study, we introduce a novel DA strategy that utilizes reinforcement learning (RL) to apply state corrections using full or partial observations of the state variables. Our investigation focuses on demonstrating this approach to the chaotic Lorenz '63 system, where the agent's objective is to minimize the root-mean-squared error between the observations and corresponding forecast states. Consequently, the agent develops a correction strategy, enhancing model forecasts based on available system state observations. Our strategy employs a stochastic action policy, enabling a Monte Carlo-based DA framework that relies on randomly sampling the policy to generate an ensemble of assimilated realizations. Results demonstrate that the developed RL algorithm performs favorably when compared to the EnKF. Additionally, we illustrate the agent's capability to assimilate non-Gaussian data, addressing a significant limitation of the EnKF.

math.DS↗

arXiv2023

Downscaling Using CDAnet Under Observational and Model Noises: The Rayleigh-Benard Convection Paradigm

Efficient downscaling of large ensembles of coarse-scale information is crucial in several applications, such as oceanic and atmospheric modeling. The determining form map is a theoretical lifting function from the low-resolution solution trajectories of a dissipative dynamical system to their corresponding fine-scale counterparts. Recently, a physics-informed deep neural network ("CDAnet") was introduced, providing a surrogate of the determining form map for efficient downscaling. CDAnet was demonstrated to efficiently downscale noise-free coarse-scale data in a deterministic setting. Herein, the performance of well-trained CDAnet models is analyzed in a stochastic setting involving (i) observational noise, (ii) model noise, and (iii) a combination of observational and model noises. The analysis is performed employing the Rayleigh-Benard convection paradigm, under three training conditions, namely, training with perfect, noisy, or downscaled data. Furthermore, the effects of noises, Rayleigh number, and spatial and temporal resolutions of the input coarse-scale information on the downscaled fields are examined. The results suggest that the expected l2-error of CDAnet behaves quadratically in terms of the standard deviations of the observational and model noises. The results also suggest that CDAnet responds to uncertainties similar to the theorized and numerically-validated CDA behavior with an additional error overhead due to CDAnet being a surrogate model of the determining form map.

math.DS↗

arXiv2023

Oil Spill Risk Analysis For The NEOM Shoreline

A risk analysis is conducted considering several release sources located around the NEOM shoreline. The sources are selected close to the coast and in neighboring regions of high marine traffic. The evolution of oil spills released by these sources is simulated using the MOHID model, driven by validated, high-resolution met-ocean fields of the Red Sea. For each source, simulations are conducted over a 4-week period, starting from first, tenth and twentieth days of each month, covering five consecutive years. A total of 48 simulations are thus conducted for each source location, adequately reflecting the variability of met-ocean conditions in the region. The risk associated with each source is described in terms of amount of oil beached, and by the elapsed time required for the spilled oil to reach the NEOM coast, extending from the Gulf of Aqaba in the North to Duba in the South. A finer analysis is performed by segmenting the NEOM shoreline, based on important coastal development and installation sites. For each subregion, source and release event considered, a histogram of the amount of volume beached is generated, also classifying individual events in terms of the corresponding arrival times. In addition, for each subregion considered, an inverse analysis is conducted to identify regions of dependence of the cumulative risk, estimated using the collection of all sources and events considered. The transport of oil around the NEOM shorelines is promoted by chaotic circulations and northwest winds in summer, and a dominant cyclonic eddy in winter. Hence, spills originating from release sources located close to the NEOM shorelines are characterized by large monthly variations in arrival times, ranging from less than a week to more than two weeks. Large variations in the volume fraction of beached oil, ranging from less then 50\% to more than 80% are reported.

physics.ao-ph↗

arXiv2022

Continuous and Discrete Data Assimilation with Noisy Observations for the Rayleigh-Benard Convection: A Computational Study

Obtaining accurate high-resolution representations of model outputs is essential to describe the system dynamics. In general, however, only spatially- and temporally-coarse observations of the system states are available. These observations can also be corrupted by noise. Downscaling is a process/scheme in which one uses coarse scale observations to reconstruct the high-resolution solution of the system states. Continuous Data Assimilation (CDA) is a recently introduced downscaling algorithm that constructs an increasingly accurate representation of the system states by continuously nudging the large scales using the coarse observations. We introduce a Discrete Data Assimilation (DDA) algorithm as a downscaling algorithm based on CDA with discrete-in-time nudging. We then investigate the performance of the CDA and DDA algorithms for downscaling noisy observations of the Rayleigh-Bénard convection system in the chaotic regime. In this computational study, a set of noisy observations was generated by perturbing a reference solution with Gaussian noise before downscaling them. The downscaled fields are then assessed using various error- and ensemble-based skill scores. The CDA solution was shown to converge towards the reference solution faster than that of DDA but at the cost of a higher asymptotic error. The numerical results also suggest a quadratic relationship between the $\ell_2$ error and the noise level for both CDA and DDA. Cubic and quadratic dependences of the DDA and CDA expected errors on the spatial resolution of the observations were obtained, respectively.

math.DS↗

arXiv2022

Enhanced Simulation of the Indian Summer Monsoon Rainfall Using Regional Climate Modeling and Continuous Data Assimilation

This study assesses a Continuous Data Assimilation (CDA) dynamical-downscaling algorithm for enhancing the simulation of the Indian summer monsoon (ISM) system. CDA is a mathematically rigorous technique that has been recently introduced to constrain the large-scale features of high-resolution atmospheric models with coarse spatial scale data. It is similar to spectral nudging but does not require any spectral decomposition for scales separation. This is expected to be particularly relevant for ISM, which involves various interactions between large-scale circulations and regional physical processes. Along with a control simulation, several downscaling simulations were conducted with the Weather Research and Forecasting (WRF) model using CDA, spectral (retaining different wavenumbers) and grid nudging for three ISM seasons: normal (2016), excess (2013), and drought (2009). The simulations are nested within the NCEP Final Analysis and the model outputs are evaluated against the observations. Compared to grid and spectral nudging, the simulations using CDA produce enhanced ISM features over the Indian subcontinent including the low-level jet, tropical easterly jet, easterly wind shear, and rainfall distributions for all investigated ISM seasons. The major ISM processes, in particular the monsoon inversion over the Arabian Sea, tropospheric temperature gradients and moist static energy over central India, and zonal wind shear over the monsoon region, are all better simulated with CDA. Spectral nudging outputs are found to be sensitive to the choice of the wavenumber, requiring careful tuning to provide robust simulations of the ISM system. In contrast, control and grid nudging generally fail to well reproduce some of the main ISM features.

physics.ao-ph↗

arXiv2020

A Visual Analytics Based Decision Making Environment for COVID-19 Modeling and Visualization

Public health officials dealing with pandemics like COVID-19 have to evaluate and prepare response plans. This planning phase requires not only looking into the spatiotemporal dynamics and impact of the pandemic using simulation models, but they also need to plan and ensure the availability of resources under different spread scenarios. To this end, we have developed a visual analytics environment that enables public health officials to model, simulate, and explore the spread of COVID-19 by supplying county-level information such as population, demographics, and hospital beds. This environment facilitates users to explore spatiotemporal model simulation data relevant to COVID-19 through a geospatial map with linked statistical views, apply different decision measures at different points in time, and understand their potential impact. Users can drill-down to county-level details such as the number of sicknesses, deaths, needs for hospitalization, and variations in these statistics over time. We demonstrate the usefulness of this environment through a use case study and also provide feedback from domain experts. We also provide details about future extensions and potential applications of this work.

cs.HC↗

arXiv2020

Impact of Atmospheric and Model Physics Perturbations On a High-Resolution Ensemble Data Assimilation System of the Red Sea

The Ensemble Adjustment Kalman Filter (EAKF) of the Data Assimilation Research Testbed (DART) is implemented to assimilate observations of satellite sea surface temperature, altimeter sea surface height and in situ ocean temperature and salinity profiles into an eddy-resolving 4 km Massachusetts Institute of Technology general circulation model (MITgcm) of the Red Sea. We investigate the impact of three different ensemble generation strategies (1) Iexp, uses ensemble of ocean states to initialize the model on 1st January, 2011 and inflates filter error covariance by 10 percent, (2) IAexp, adds ensemble of atmospheric forcing to Iexp, and (3) IAPexp, adds perturbed model physics to IAexp. The assimilation experiments are run for one year, starting from the same initial ensemble and assimilating data every three days. Results demonstrate that the Iexp mainly improved the model outputs with respect to assimilation free MITgcm run in the first few months, before showing signs of dynamical imbalances in the ocean estimates, particularly in the data-sparse subsurface layers. The IAexp yielded substantial improvements throughout the assimilation period with almost no signs of imbalances, including the subsurface layers. It further well preserved the model mesoscale features resulting in an improved forecasts for eddies, both in terms of intensity and location. Perturbing model physics in IAPexp slightly improved the forecast statistics and also the placement of basin scale eddies. Increasing hydrographic coverage further improved the results of IAPexp compared to IAexp in the subsurface layers. Switching off multiplicative inflation in IAexp and IAPexp leads to further improvements, especially in the subsurface layers.

physics.ao-ph↗

arXiv2019

Efficient Dynamical Downscaling of General Circulation Models Using Continuous Data Assimilation

Continuous data assimilation (CDA) is successfully implemented for the first time for efficient dynamical downscaling of a global atmospheric reanalysis. A comparison of the performance of CDA with the standard grid and spectral nudging techniques for representing long- and short-scale features in the downscaled fields using the Weather Research and Forecast (WRF) model is further presented and analyzed. The WRF model is configured at 25km horizontal resolution and is driven by 250km initial and boundary conditions from NCEP/NCAR reanalysis fields. Downscaling experiments are performed over a one-month period in January, 2016. The similarity metric is used to evaluate the performance of the downscaling methods for large and small scales. Similarity results are compared for the outputs of the WRF model with different downscaling techniques, NCEP reanalysis, and Final Analysis. Both spectral nudging and CDA describe better the small-scale features compared to grid nudging. The choice of the wave number is critical in spectral nudging; increasing the number of retained frequencies generally produced better small-scale features, but only up to a certain threshold after which its solution gradually became closer to grid nudging. CDA maintains the balance of the large- and small-scale features similar to that of the best simulation achieved by the best spectral nudging configuration, without the need of a spectral decomposition. The different downscaled atmospheric variables, including rainfall distribution, with CDA is most consistent with the observations. The Brier skill score values further indicate that the added value of CDA is distributed over the entire model domain. The overall results clearly suggest that CDA provides an efficient new approach for dynamical downscaling by maintaining better balance between the global model and the downscaled fields.

physics.ao-ph↗

arXiv2017

Optimal projection of observations in a Bayesian setting

Optimal dimensionality reduction methods are proposed for the Bayesian inference of a Gaussian linear model with additive noise in presence of overabundant data. Three different optimal projections of the observations are proposed based on information theory: the projection that minimizes the Kullback-Leibler divergence between the posterior distributions of the original and the projected models, the one that minimizes the expected Kullback-Leibler divergence between the same distributions, and the one that maximizes the mutual information between the parameter of interest and the projected observations. The first two optimization problems are formulated as the determination of an optimal subspace and therefore the solution is computed using Riemannian optimization algorithms on the Grassmann manifold. Regarding the maximization of the mutual information, it is shown that there exists an optimal subspace that minimizes the entropy of the posterior distribution of the reduced model; a basis of the subspace can be computed as the solution to a generalized eigenvalue problem; an a priori error estimate on the mutual information is available for this particular solution; and that the dimensionality of the subspace to exactly conserve the mutual information between the input and the output of the models is less than the number of parameters to be inferred. Numerical applications to linear and nonlinear models are used to assess the efficiency of the proposed approaches, and to highlight their advantages compared to standard approaches based on the principal component analysis of the observations.

math.ST↗

arXiv2016

Quantifying Uncertainties in Fault Slip Distribution during the Tōhoku Tsunami using Polynomial Chaos

An efficient method for inferring Manning's $n$ coefficients using water surface elevation data was presented in Sraj et al. (2014) focusing on a test case based on data collected during the $Tōhoku$ earthquake and tsunami. Polynomial chaos expansions were used to build an inexpensive surrogate for the numerical model Geoclaw, which were then used to perform a sensitivity analysis in addition to the inversion. In this paper, a new analysis is performed with the goal of inferring the fault slip distribution of the $Tōhoku$ earthquake using a similar problem setup. The same approach to constructing the PC surrogate did not lead to a converging expansion, however an alternative approach based on Basis-Pursuit DeNoising was found to be suitable. Our result shows that the fault slip distribution can be inferred using water surface elevation data whereas the inferred values minimizes the error between observations and the numerical model. The numerical approach and the resulting inversion are presented in this work.

stat.ME↗

arXiv2015

A Bayesian Consistent Dual Ensemble Kalman Filter for State-Parameter Estimation in Subsurface Hydrology

Ensemble Kalman filtering (EnKF) is an efficient approach to addressing uncertainties in subsurface groundwater models. The EnKF sequentially integrates field data into simulation models to obtain a better characterization of the model's state and parameters. These are generally estimated following joint and dual filtering strategies, in which, at each assimilation cycle, a forecast step by the model is followed by an update step with incoming observations. The Joint-EnKF directly updates the augmented state-parameter vector while the Dual-EnKF employs two separate filters, first estimating the parameters and then estimating the state based on the updated parameters. In this paper, we reverse the order of the forecast-update steps following the one-step-ahead (OSA) smoothing formulation of the Bayesian filtering problem, based on which we propose a new dual EnKF scheme, the Dual-EnKF$_{\rm OSA}$. Compared to the Dual-EnKF, this introduces a new update step to the state in a fully consistent Bayesian framework, which is shown to enhance the performance of the dual filtering approach without any significant increase in the computational cost. Numerical experiments are conducted with a two-dimensional synthetic groundwater aquifer model to assess the performance and robustness of the proposed Dual-EnKF$_{\rm OSA}$, and to evaluate its results against those of the Joint- and Dual-EnKFs. The proposed scheme is able to successfully recover both the hydraulic head and the aquifer conductivity, further providing reliable estimates of their uncertainties. Compared with the standard Joint- and Dual-EnKFs, the proposed scheme is found more robust to different assimilation settings, such as the spatial and temporal distribution of the observations, and the level of noise in the data. Based on our experimental setups, it yields up to 25% more accurate state and parameters estimates.

physics.data-an↗

arXiv2015

Polynomial Chaos-based Bayesian Inference of K-Profile Parametrization in a General Circulation Model of the Tropical Pacific

The authors present a Polynomial Chaos (PC)-based Bayesian inference method for quantifying the uncertainties of the K-Profile Parametrization (KPP) within the MIT General Circulation Model (MITgcm) of the tropical pacific. The inference of the uncertain parameters is based on a Markov Chain Monte Carlo (MCMC) scheme that utilizes a newly formulated test statistic taking into account the different components representing the structures of turbulent mixing on both daily and seasonal timescales in addition to the data quality, and filters for the effects of parameter perturbations over those due to changes in the wind. To avoid the prohibitive computational cost of integrating the MITgcm model at each MCMC iteration, we build a surrogate model for the test statistic using the PC method. To filter out the noise in the model predictions and avoid related convergence issues, we resort to a Basis-Pursuit-DeNoising (BPDN) compressed sensing approach to determine the PC coefficients of a representative surrogate model. The PC surrogate is then used to evaluate the test statistic in the MCMC step for sampling the posterior of the uncertain parameters. Results of the posteriors indicate good agreement with the default values for two parameters of the KPP model namely the critical bulk and gradient Richardson numbers; while the posteriors of the remaining parameters were barely informative.

stat.ME↗

arXiv2015

Coordinate Transformation and Polynomial Chaos for the Bayesian Inference of a Gaussian Process with Parametrized Prior Covariance Function

This paper addresses model dimensionality reduction for Bayesian inference based on prior Gaussian fields with uncertainty in the covariance function hyper-parameters. The dimensionality reduction is traditionally achieved using the Karhunen-\Loeve expansion of a prior Gaussian process assuming covariance function with fixed hyper-parameters, despite the fact that these are uncertain in nature. The posterior distribution of the Karhunen-Lo\`{e}ve coordinates is then inferred using available observations. The resulting inferred field is therefore dependent on the assumed hyper-parameters. Here, we seek to efficiently estimate both the field and covariance hyper-parameters using Bayesian inference. To this end, a generalized Karhunen-Lo\`{e}ve expansion is derived using a coordinate transformation to account for the dependence with respect to the covariance hyper-parameters. Polynomial Chaos expansions are employed for the acceleration of the Bayesian inference using similar coordinate transformations, enabling us to avoid expanding explicitly the solution dependence on the uncertain hyper-parameters. We demonstrate the feasibility of the proposed method on a transient diffusion equation by inferring spatially-varying log-diffusivity fields from noisy data. The inferred profiles were found closer to the true profiles when including the hyper-parameters' uncertainty in the inference formulation.

math.NA↗

arXiv2014

Ensemble Kalman filtering with a divided state-space strategy for coupled data assimilation problems

This study considers the data assimilation problem in coupled systems, which consists of two components (sub-systems) interacting with each other through certain coupling terms. A straightforward way to tackle the assimilation problem in such systems is to concatenate the states of the sub-systems into one augmented state vector, so that a standard ensemble Kalman filter (EnKF) can be directly applied. In this work we present a divided state-space estimation strategy, in which data assimilation is carried out with respect to each individual sub-system, involving quantities from the sub-system itself and correlated quantities from other coupled sub-systems. On top of the divided state-space estimation strategy, we also consider the possibility to run the sub-systems separately. Combining these two ideas, a few variants of the EnKF are derived. The introduction of these variants is mainly inspired by the current status and challenges in coupled data assimilation problems, and thus might be of interest from a practical point of view. Numerical experiments with a multi-scale Lorentz 96 model are conducted to evaluate the performance of these variants against that of the conventional EnKF. In addition, specific for coupled data assimilation problems, two prototypes of extensions of the presented methods are also developed in order to achieve a trade-off between efficiency and accuracy.

physics.ao-ph↗

arXiv2014

Ensemble Kalman filtering with residual nudging: an extension to state estimation problems with nonlinear observation operators

The ensemble Kalman filter (EnKF) is an efficient algorithm for many data assimilation problems. In certain circumstances, however, divergence of the EnKF might be spotted. In previous studies, the authors proposed an observation-space-based strategy, called residual nudging, to improve the stability of the EnKF when dealing with linear observation operators. The main idea behind residual nudging is to monitor and, if necessary, adjust the distances (misfits) between the real observations and the simulated ones of the state estimates, in the hope that by doing so one may be able to obtain better estimation accuracy. In the present study, residual nudging is extended and modified in order to handle nonlinear observation operators. Such extension and modification result in an iterative filtering framework that, under suitable conditions, is able to achieve the objective of residual nudging for data assimilation problems with nonlinear observation operators. The 40 dimensional Lorenz 96 model is used to illustrate the performance of the iterative filter. Numerical results show that, while a normal EnKF may diverge with nonlinear observation operators, the proposed iterative filter remains stable and leads to reasonable estimation accuracy under various experimental settings.

physics.ao-ph↗

arXiv2013

Continuity in time of solutions of a phase-field model

A phase field model proposed by G. Caginalp for the description of phase changes in materials is under consideration. It is assumed that the medium is located in a container with heat conductive walls that are not subjected to phase changes. Therefore, the temperature variable is defined both in the medium and wall regions, whereas the phase variable is only considered in the medium part. The case of Lipschitz domains in two and three dimensions is studied. We show that the temperature and phase variables are continuous in time functions with values in $L^2$ and $H^1$, respectively, provided that the initial values of them are from $L^2$ and $H^1$, respectively. Moreover, continuous dependence of solutions on the initial data and boundary conditions is proved.

math.FA↗