arXiv subjects

Neil K. Chada

Publications and source records attributed to Neil K. Chada.

At least 19 recordsLinked to original sources

arXiv2026

Kinetic Langevin Splitting Schemes for Constrained Sampling

Constrained sampling is an important and challenging task in computational statistics, concerned with generating samples from a distribution under certain constraints. There are numerous types of algorithm aimed at this task, ranging from general Markov chain Monte Carlo, to unadjusted Langevin methods. In this article we propose a series of new sampling algorithms based on the latter of these, specifically the kinetic Langevin dynamics. Our series of algorithms are motivated on advanced numerical methods which are splitting order schemes, which include the BU and BAO families of splitting schemes.Their advantage lies in the fact that they have favorable strong order (bias) rates and computationally efficiency. In particular we provide a number of theoretical insights which include a Wasserstein contraction and convergence results. We are able to demonstrate favorable results, such as improved complexity bounds over existing non-splitting methodologies. Our results are verified through numerical experiments on a range of models with constraints, which include a toy example and Bayesian linear regression.

stat.ME↗

arXiv2025

Multilevel Localized Ensemble Kalman Bucy Filters

In this article we propose and develop a new methodology which is inspired from Kalman filtering and multilevel Monte Carlo (MLMC), entitle the multilevel localized ensemble Kalman--Bucy Filter (MLLEnKBF). Based on the work of Chada et al. \cite{CJY20}, we provide an important extension on this which is to include the technique of covariance localization. Localization is important as it can induce stability and remove long spurious correlations, particularly with a small ensemble size. Our resulting algorithm is used for both state and parameter estimation, for the later we exploit our method for normalizing constant estimation. As of yet, MLMC has only been applied to localized data assimilation methods in a discrete-time setting, therefore this work acts as a first in the continuous-time setting. Numerical results indicate its performance, and benefit through a range of model problems, which include a linear Ornstein--Uhlenbeck process, of moderately high dimension, and the Lorenz 96 model, for parameter estimation. Our results demonstrate improved stability, and that with MLMC, one can reduce the computational complexity to attain an order is MSE $\mathcal{O}(\epsilon^2)$, for $\epsilon>0$.

stat.CO↗

arXiv2025

Unbiased Parameter Estimation for Bayesian Inverse Problems

In this paper we consider the estimation of unknown parameters in Bayesian inverse problems. In most cases of practical interest, there are several barriers to performing such estimation, This includes a numerical approximation of a solution of a differential equation and, even if exact solutions are available, an analytical intractability of the marginal likelihood and its associated gradient, which is used for parameter estimation. The focus of this article is to deliver unbiased estimates of the unknown parameters, that is, stochastic estimators that, in expectation, are equal to the maximize of the marginal likelihood, and possess no numerical approximation error. Based upon the ideas of [4] we develop a new approach for unbiased parameter estimation for Bayesian inverse problems. We prove unbiasedness and establish numerically that the associated estimation procedure is faster than the current state-of-the-art methodology for this problem. We demonstrate the performance of our methodology on a range of problems which include a PDE and ODE.

stat.ME↗

arXiv2024

Unbiased Approximations for Stationary Distributions of McKean-Vlasov SDEs

We consider the development of unbiased estimators, to approximate the stationary distribution of Mckean-Vlasov stochastic differential equations (MVSDEs). These are an important class of processes, which frequently appear in applications such as mathematical finance, biology and opinion dynamics. Typically the stationary distribution is unknown and indeed one cannot simulate such processes exactly. As a result one commonly requires a time-discretization scheme which results in a discretization bias and a bias from not being able to simulate the associated stationary distribution. To overcome this bias, we present a new unbiased estimator taking motivation from the literature on unbiased Monte Carlo. We prove the unbiasedness of our estimator, under assumptions. In order to prove this we require developing ergodicity results of various discrete time processes, through an appropriate discretization scheme, towards the invariant measure. Numerous numerical experiments are provided, on a range of MVSDEs, to demonstrate the effectiveness of our unbiased estimator. Such examples include the Currie-Weiss model, a 3D neuroscience model and a parameter estimation problem.

stat.ME↗

arXiv2024

Learning dynamical systems from data: Gradient-based dictionary optimization

The Koopman operator plays a crucial role in analyzing the global behavior of dynamical systems. Existing data-driven methods for approximating the Koopman operator or discovering the governing equations of the underlying system typically require a fixed set of basis functions, also called dictionary. The optimal choice of basis functions is highly problem-dependent and often requires domain knowledge. We present a novel gradient descent-based optimization framework for learning suitable and interpretable basis functions from data and show how it can be used in combination with EDMD, SINDy, and PDE-FIND. We illustrate the efficacy of the proposed approach with the aid of various benchmark problems such as the Ornstein-Uhlenbeck process, Chua's circuit, a nonlinear heat equation, as well as protein-folding data.

math.DS↗

arXiv2024

Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics

We propose a scalable kinetic Langevin dynamics algorithm for sampling parameter spaces of big data and AI applications. Our scheme combines a symmetric forward/backward sweep over minibatches with a symmetric discretization of Langevin dynamics. For a particular Langevin splitting method (UBU), we show that the resulting Symmetric Minibatch Splitting-UBU (SMS-UBU) integrator has bias $O(h^2 d^{1/2})$ in dimension $d>0$ with stepsize $h>0$, despite only using one minibatch per iteration, thus providing excellent control of the sampling bias as a function of the stepsize. We apply the algorithm to explore local modes of the posterior distribution of Bayesian neural networks (BNNs) and evaluate the calibration performance of the posterior predictive probabilities for neural networks with convolutional neural network architectures for classification problems on three different datasets (Fashion-MNIST, Celeb-A and chest X-ray). Our results indicate that BNNs sampled with SMS-UBU can offer significantly better calibration performance compared to standard methods of training and stochastic weight averaging.

stat.ML↗

arXiv2024

A Stochastic Iteratively Regularized Gauss-Newton Method

This work focuses on developing and motivating a stochastic version of a wellknown inverse problem methodology. Specifically, we consider the iteratively regularized Gauss-Newton method, originally proposed by Bakushinskii for infinite-dimensional problems. Recent work have extended this method to handle sequential observations, rather than a single instance of the data, demonstrating notable improvements in reconstruction accuracy. In this paper, we further extend these methods to a stochastic framework through mini-batching, introducing a new algorithm, the stochastic iteratively regularized Gauss-Newton method (SIRGNM). Our algorithm is designed through the use randomized sketching. We provide an analysis for the SIRGNM, which includes a preliminary error decomposition and a convergence analysis, related to the residuals. We provide numerical experiments on a 2D elliptic PDE example. This illustrates the effectiveness of the SIRGNM, through maintaining a similar level of accuracy while reducing on the computational time.

math.NA↗

arXiv2024

The Ensemble Kalman Filter for Dynamic Inverse Problems

In inverse problems, the goal is to estimate unknown model parameters from noisy observational data. Traditionally, inverse problems are solved under the assumption of a fixed forward operator describing the observation model. In this article, we consider the extension of this approach to situations where we have a dynamic forward model, motivated by applications in scientific computation and engineering. We specifically consider this extension for a derivative-free optimizer, the ensemble Kalman inversion (EKI). We introduce and justify a new methodology called dynamic-EKI, which is a particle-based method with a changing forward operator. We analyze our new method, presenting results related to the control of our particle system through its covariance structure. This analysis includes moment bounds and an ensemble collapse, which are essential for demonstrating a convergence result. We establish convergence in expectation and validate our theoretical findings through experiments with dynamic-EKI applied to a 2D Darcy flow partial differential equation.

math.NA↗

arXiv2023

Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients

We present an unbiased method for Bayesian posterior means based on kinetic Langevin dynamics that combines advanced splitting methods with enhanced gradient approximations. Our approach avoids Metropolis correction by coupling Markov chains at different discretization levels in a multilevel Monte Carlo approach. Theoretical analysis demonstrates that our proposed estimator is unbiased, attains finite variance, and satisfies a central limit theorem. It can achieve accuracy $\epsilon>0$ for estimating expectations of Lipschitz functions in $d$ dimensions with $\mathcal{O}(d^{1/4}\epsilon^{-2})$ expected gradient evaluations, without assuming warm start. We exhibit similar bounds using both approximate and stochastic gradients, and our method's computational cost is shown to scale independently of the size of the dataset. The proposed method is tested using a multinomial regression problem on the MNIST dataset and a Poisson regression model for soccer scores. Experiments indicate that the number of gradient evaluations per effective sample is independent of dimension, even when using inexact gradients. For product distributions, we give dimension-independent variance bounds. Our results demonstrate that in large-scale applications, the unbiased algorithm we present can be 2-3 orders of magnitude more efficient than the ``gold-standard" randomized Hamiltonian Monte Carlo.

stat.CO↗

arXiv2023

Unbiased Estimation using Underdamped Langevin Dynamics

In this work we consider the unbiased estimation of expectations w.r.t.~probability measures that have non-negative Lebesgue density, and which are known point-wise up-to a normalizing constant. We focus upon developing an unbiased method via the underdamped Langevin dynamics, which has proven to be popular of late due to applications in statistics and machine learning. Specifically in continuous-time, the dynamics can be constructed {so that as the time goes to infinity they} admit the probability of interest as a stationary measure. {In many cases, time-discretized versions of the underdamped Langevin dynamics are used in practice which are run only with a fixed number of iterations.} We develop a novel scheme based upon doubly randomized estimation as in \cite{ub_grad,disc_model}, which requires access only to time-discretized versions of the dynamics. {The proposed scheme aims to remove the dicretization bias and the bias resulting from running the dynamics for a finite number of iterations}. We prove, under standard assumptions, that our estimator is of finite variance and either has finite expected cost, or has finite cost with a high probability. To illustrate our theoretical findings we provide numerical experiments which verify our theory, which include challenging examples from Bayesian statistics and statistical physics.

stat.CO↗

arXiv2023

A Statistical Framework and Analysis for Perfect Radar Pulse Compression

Perfect radar pulse compression coding is a potential emerging field which aims at providing rigorous analysis and fundamental limit radar experiments. It is based on finding non-trivial pulse codes, which we can make statistically equivalent, to the radar experiments carried out with elementary pulses of some shape. A common engineering-based radar experiment design, regarding pulse-compression, often omits the rigorous theory and mathematical limitations. In this work our aim is to develop a mathematical theory which coincides with understanding the radar experiment in terms of the theory of comparison of statistical experiments. We review and generalize some properties of the It\^{o} measure. We estimate the unknown i.e. the structure function in the context of Bayesian statistical inverse problems. We study the posterior for generalized $d$-dimensional inverse problems, where we consider both real-valued and complex-valued inputs for posteriori analysis. Finally this is then extended to the infinite dimensional setting, where our analysis suggests the underlying posterior is non-Gaussian.

math.ST↗

arXiv2023

The Stochastic Steepest Descent Method for Robust Optimization in Banach Spaces

Stochastic gradient methods have been a popular and powerful choice of optimization methods, aimed at minimizing functions. Their advantage lies in the fact that that one approximates the gradient as opposed to using the full Jacobian matrix. One research direction, related to this, has been on the application to infinite-dimensional problems, where one may naturally have a Hilbert space framework. However, there has been limited work done on considering this in a more general setup, such as where the natural framework is that of a Banach space. This article aims to address this by the introduction of a novel stochastic method, the stochastic steepest descent method (SSD). The SSD will follow the spirit of stochastic gradient descent, which utilizes Riesz representation to identify gradients and derivatives. Our choice for using such a method is that it naturally allows one to adopt a Banach space setting, for which recent applications have exploited the benefit of this, such as in PDE-constrained shape optimization. We provide a convergence theory related to this under mild assumptions. Furthermore, we demonstrate the performance of this method on a couple of numerical applications, namely a $p$-Laplacian and an optimal control problem. Our assumptions are verified in these applications.

math.NA↗

arXiv2023

Bayesian inversion with α-stable priors

We propose to use Lévy α-stable distributions for constructing priors for Bayesian inverse problems. The construction is based on Markov fields with stable-distributed increments. Special cases include the Cauchy and Gaussian distributions, with stability indices α = 1, and α = 2, respectively. Our target is to show that these priors provide a rich class of priors for modelling rough features. The main technical issue is that the α-stable probability density functions do not have closed-form expressions in general, and this limits their applicability. For practical purposes, we need to approximate probability density functions through numerical integration or series expansions. Current available approximation methods are either too time-consuming or do not function within the range of stability and radius arguments needed in Bayesian inversion. To address the issue, we propose a new hybrid approximation method for symmetric univariate and bivariate α-stable distributions, which is both fast to evaluate, and accurate enough from a practical viewpoint. Then we use approximation method in the numerical implementation of α-stable random field priors. We demonstrate the applicability of the constructed priors on selected Bayesian inverse problems which include the deconvolution problem, and the inversion of a function governed by an elliptic partial differential equation. We also demonstrate hierarchical α-stable priors in the one-dimensional deconvolution problem. We employ maximum-a-posterior-based estimation at all the numerical examples. To that end, we exploit the limited-memory BFGS and its bounded variant for the estimator.

stat.CO↗

arXiv2022

A Data-Adaptive Prior for Bayesian Learning of Kernels in Operators

Kernels are efficient in representing nonlocal dependence and they are widely used to design operators between function spaces. Thus, learning kernels in operators from data is an inverse problem of general interest. Due to the nonlocal dependence, the inverse problem can be severely ill-posed with a data-dependent singular inversion operator. The Bayesian approach overcomes the ill-posedness through a non-degenerate prior. However, a fixed non-degenerate prior leads to a divergent posterior mean when the observation noise becomes small, if the data induces a perturbation in the eigenspace of zero eigenvalues of the inversion operator. We introduce a data-adaptive prior to achieve a stable posterior whose mean always has a small noise limit. The data-adaptive prior's covariance is the inversion operator with a hyper-parameter selected adaptive to data by the L-curve method. Furthermore, we provide a detailed analysis on the computational practice of the data-adaptive prior, and demonstrate it on Toeplitz matrices and integral operators. Numerical tests show that a fixed prior can lead to a divergent posterior mean in the presence of any of the four types of errors: discretization error, model error, partial observation and wrong noise assumption. In contrast, the data-adaptive prior always attains posterior means with small noise limits.

stat.ML↗

arXiv2022

Improved Efficiency of Multilevel Monte Carlo for Stochastic PDE through Strong Pairwise Coupling

Multilevel Monte Carlo (MLMC) has become an important methodology in applied mathematics for reducing the computational cost of weak approximations. For many problems, it is well-known that strong pairwise coupling of numerical solutions in the multilevel hierarchy is needed to obtain efficiency gains. In this work, we show that strong pairwise coupling indeed is also important when (MLMC) is applied to stochastic partial differential equations (SPDE) of reaction-diffusion type, as it can improve the rate of convergence and thus improve tractability. For the (MLMC) method with strong pairwise coupling that was developed and studied numerically on filtering problems in [{\it Chernov et al., Numer. Math., 147 (2021), 71-125}], we prove that the rate of computational efficiency is higher than for existing methods. We also provide numerical comparisons with alternative coupling ideas on linear and nonlinear SPDE to illustrate the importance of this feature.

math.NA↗

arXiv2022

Multilevel Estimation of Normalization Constants Using the Ensemble Kalman-Bucy Filter

In this article we consider the application of multilevel Monte Carlo, for the estimation of normalizing constants. In particular we will make use of the filtering algorithm, the ensemble Kalman-Bucy filter (EnKBF), which is an N-particle representation of the Kalma-Bucy filter (KBF). The EnKBF is of interest as it coincides with the optimal filter in the continuous-linear setting, i.e. the KBF. This motivates our particular setup in the linear setting. The resulting methodology we will use is the multilevel ensemble Kalman-Bucy filter (MLEnKBF). We provide an analysis based on deriving Lq-bounds for the normalizing constants using both the single-level, and the multilevel algorithms. Our results will be highlighted through numerical results, where we firstly demonstrate the error-to-cost rates of the MLEnKBF comparing it to the EnKBF on a linear Gaussian model. Our analysis will be specific to one variant of the MLEnKBF, whereas the numerics will be tested on different variants. We also exploit this methodology for parameter estimation, where we test this on the models arising in atmospheric sciences, such as the stochastic Lorenz 63 and 96 model.

math.NA↗

arXiv2022

Unbiased Estimation of the Vanilla and Deterministic Ensemble Kalman-Bucy Filters

In this article we consider the development of an unbiased estimator for the ensemble Kalman--Bucy filter (EnKBF). The EnKBF is a continuous-time filtering methodology which can be viewed as a continuous-time analogue of the famous discrete-time ensemble Kalman filter. Our unbiased estimators will be motivated from recent work [Rhee \& Glynn 2010, [31]] which introduces randomization as a means to produce unbiased and finite variance estimators. The randomization enters through both the level of discretization, and through the number of samples at each level. Our estimator will be specific to linear and Gaussian settings, where we know that the EnKBF is consistent, in the particle limit $N \rightarrow \infty$, with the KBF. We highlight this for two particular variants of the EnKBF, i.e. the deterministic and vanilla variants, and demonstrate this on a linear Ornstein--Uhlenbeck process. We compare this with the EnKBF and the multilevel (MLEnKBF), for experiments with varying dimension size. We also provide a proof of the multilevel deterministic EnKBF, which provides a guideline for some of the unbiased methods.

stat.ME↗

arXiv2022

On a Dynamic Variant of the Iteratively Regularized Gauss-Newton Method with Sequential Data

For numerous parameter and state estimation problems, assimilating new data as they become available can help produce accurate and fast inference of unknown quantities. While most existing algorithms for solving those kind of ill-posed inverse problems can only be used with a single instance of the observed data, in this work we propose a new framework that enables existing algorithms to invert multiple instances of data in a sequential fashion. Specifically we will work with the well-known iteratively regularized Gauss-Newton method (IRGNM), a variational methodology for solving nonlinear inverse problems. We develop a theory of convergence analysis for a proposed dynamic IRGNM algorithm in the presence of Gaussian white noise. We combine this algorithm with the classical IRGNM to deliver a practical (hybrid) algorithm that can invert data sequentially while producing fast estimates. Our work includes the proof of well-definedness of the proposed iterative scheme, as well as various error bounds that rely on standard assumptions for nonlinear inverse problems. We use several numerical experiments to verify our theoretical findings, and to highlight the benefits of incorporating sequential data. The context of the numerical experiments comprises various parameter identification problems including a Darcy flow elliptic PDE example, and that of electrical impedance tomography.

math.NA↗