arXiv subjects

Peter A. Whalley

Publications and source records attributed to Peter A. Whalley.

14 recordsLinked to original sources

arXiv2026

Theoretical guarantees for stochastic gradient Langevin dynamics

We prove asymptotic bias bounds for stochastic gradient Langevin dynamics in Wasserstein distance of order two. We assume that the negative log-density is strongly convex with a Lipschitz gradient, and that the stochastic gradient estimator is unbiased with an error satisfying a mean-square Lipschitz condition. The bounds are of order $h$ under a fourth moment assumption on the stochastic gradient error and of order $h^{1/2}$ under only a second moment assumption, where $h$ is the stepsize. A spiked-noise example shows that a second moment assumption alone is insufficient for a bound of order $h$ that is uniform over noise distributions with a fixed variance.

math.PR↗

arXiv2026

Wasserstein mixing time of the unadjusted Langevin algorithm

We provide new estimates in Wasserstein distance for the asymptotic bias of the unadjusted Langevin algorithm, in the classical setting of log-smooth strongly log-concave measures. Our bound implies a Wasserstein mixing time of order $κ\sqrt{d}/\varepsilon$, where $κ$ is the condition number, $d$ is the dimension, and $\varepsilon$ is the target precision: this improves by a factor of $\sqrt{d}/\varepsilon$ over the previous state-of-the-art results.

stat.CO↗

arXiv2026

Theoretical guarantees for stochastic gradient sampling methods via Gaussian convolution inequalities

We derive first-order (in the stepsize) bounds on the bias in Wasserstein distances of the invariant measure of stochastic gradient kinetic Langevin dynamics with minimal assumptions on the stochastic gradient noise. These bounds sharpen existing non-asymptotic guarantees for stochastic-gradient MCMC methods and provide a quantitative resolution of a previously open problem on invariant measure accuracy. The main technical ingredients are new Gaussian convolution inequalities controlling the Wasserstein-$p$ distance between a Gaussian convolved with a mean-zero perturbation and the Gaussian itself. We anticipate that these inequalities will be of independent interest beyond the present application. To complement the theory, we illustrate the benefits of stochastic gradient sampling methods on a one-dimensional model, a Bayesian logistic regression example and a Bayesian random-effect model including comparison with pseudo-marginal MCMC methods.

stat.CO↗

arXiv2026

Cluster formation for weakly interacting kinetic Langevin dynamics

In this paper, we study the formation of clusters for stochastic interacting particle systems (IPS) that interact through short-range attractive potentials in a periodic domain. We consider kinetic (underdamped) Langevin dynamics and focus on the low-friction regime. Employing a linear stability analysis for the kinetic McKean-Vlasov equation, we show that, at sufficiently low temperatures, and for sufficiently short-ranged interactions, the particles form clusters that correspond to metastable states of the mean-field dynamics. We derive the friction and particle-count dependent cluster-formation time and numerically measure the friction-dependent times to reach a stationary state (given by a state in which all particles are bound in a single cluster). By providing both theory and numerical methods in the inertial stochastic setting, this work acts as a bridge between cluster formation studies in overdamped Langevin dynamics and the Hamiltonian (microcanonical) limit.

math.NA↗

arXiv2026

Accelerated sampling using SamAdams variable timesteps and position-adaptive Langevin dynamics

We introduce an accelerated Langevin-based sampling method that is based on two complementary devices: \emph{SamAdams} adaptive timestepping, which automatically shrinks the effective integration step in stiff regions of phase space using a relaxed stiffness monitor, and \emph{position-adaptive Langevin} (PAL) dynamics, which concentrates friction along the local force direction while preserving the canonical distribution as the exact invariant measure. The resulting combined scheme (SA-PAL) is implemented in a palindromic integrator which requires only one force evaluation per iteration through suitable organisation of the integration steps and by exploiting the rank-one-plus-scalar structure of the PAL friction tensor. We test the method on various model problems: the Rosenbrock function, a thin entropic channel, the Mueller-Brown potential, and a Bayesian parameterisation problem with a sparsity-inducing shrinkage prior. On the Rosenbrock and Mueller-Brown potentials mixing rates are improved by 1.5-3 times compared to fixed stepsize integration. Efficiency gains of more than an order of magnitude are documented in the other examples.

math.NA↗

arXiv2025

Random Reshuffling for Stochastic Gradient Langevin Dynamics

We examine the use of different randomisation policies for stochastic gradient algorithms used in sampling, based on first-order (or overdamped) Langevin dynamics, the most popular of which is known as Stochastic Gradient Langevin Dynamics. Conventionally, this algorithm is combined with a specific stochastic gradient strategy, called Robbins-Monro. In this work, we study an alternative strategy, Random Reshuffling, and show convincingly that it leads to improved performance via: a) a proof of reduced bias in the Wasserstein metric for strongly convex, gradient Lipschitz potentials; b) an analytical demonstration of reduced bias for a Gaussian model problem; and c) an empirical demonstration of reduced bias in numerical experiments for some logistic regression problems. This is especially important since Random Reshuffling is typically more efficient due to memory access and cache reasons. Such acceleration for the Random Reshuffling policy is familiar from the optimisation literature on stochastic gradient descent.

math.NA↗

arXiv2025

Unbiased Kinetic Langevin Monte Carlo with Inexact Gradients

We present an unbiased method for Bayesian posterior means based on kinetic Langevin dynamics that combines advanced splitting methods with enhanced gradient approximations. Our approach avoids Metropolis correction by coupling Markov chains at different discretization levels in a multilevel Monte Carlo approach. Theoretical analysis demonstrates that our proposed estimator is unbiased, attains finite variance, and satisfies a central limit theorem. It can achieve accuracy $ε>0$ for estimating expectations of Lipschitz functions in $d$ dimensions with $\mathcal{O}(d^{1/4}ε^{-2})$ expected gradient evaluations, without assuming warm start. We exhibit similar bounds using both approximate and stochastic gradients, and our method's computational cost is shown to scale independently of the size of the dataset. The proposed method is tested using a multinomial regression problem on the MNIST dataset and a Poisson regression model for soccer scores. Experiments indicate that the number of gradient evaluations per effective sample is independent of dimension, even when using inexact gradients. For product distributions, we give dimension-independent variance bounds. Our results demonstrate that in large-scale applications, the unbiased algorithm we present can be 2-3 orders of magnitude more efficient than the ``gold-standard" randomized Hamiltonian Monte Carlo.

stat.CO↗

arXiv2025

Convergence of kinetic Langevin samplers for non-convex potentials

We study three kinetic Langevin samplers including the Euler discretization, the BU and the UBU splitting scheme. We provide contraction results in $L^1$-Wasserstein distance for non-convex potentials. These results are based on a carefully tailored distance function and an appropriate coupling construction. Additionally, the error in the $L^1$-Wasserstein distance between the true target measure and the invariant measure of the discretization scheme is bounded. To get an $\varepsilon$-accuracy in $L^1$-Wasserstein distance, we show complexity guarantees of order $\mathcal{O}(\sqrt{d}/\varepsilon)$ for the Euler scheme and $\mathcal{O}(d^{1/4}/\sqrt{\varepsilon})$ for the UBU scheme under appropriate assumptions on the target measure. The results are applicable to interacting particle systems and provide bounds for sampling probability measures of mean-field type.

math.PR↗

arXiv2025

Randomised Splitting Methods and Stochastic Gradient Descent

We explore an explicit link between stochastic gradient descent using common batching strategies and splitting methods for ordinary differential equations. From this perspective, we introduce a new minibatching strategy (called Symmetric Minibatching Strategy) for stochastic gradient optimisation which shows greatly reduced stochastic gradient bias (from $\mathcal{O}(h^2)$ to $\mathcal{O}(h^4)$ in the optimiser stepsize $h$), when combined with momentum-based optimisers. We justify why momentum is needed to obtain the improved performance using the theory of backward analysis for splitting integrators and provide a detailed analytic computation of the stochastic gradient bias on a simple example. Further, we provide improved convergence guarantees for this new minibatching strategy using Lyapunov techniques that show reduced stochastic gradient bias for a fixed stepsize (or learning rate) over the class of strongly-convex and smooth objective functions. Via the same techniques we also improve the known results for the Random Reshuffling strategy for stochastic gradient descent methods with momentum. We argue that this also leads to a faster convergence rate when considering a decreasing stepsize schedule. Both the reduced bias and efficacy of decreasing stepsizes are demonstrated numerically on several motivating examples.

math.OC↗

arXiv2025

Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics

We propose a scalable kinetic Langevin dynamics algorithm for sampling parameter spaces of big data and AI applications. Our scheme combines a symmetric forward/backward sweep over minibatches with a symmetric discretization of Langevin dynamics. For a particular Langevin splitting method (UBU), we show that the resulting Symmetric Minibatch Splitting-UBU (SMS-UBU) integrator has bias $O(h^2 d^{1/2})$ in dimension $d>0$ with stepsize $h>0$, despite only using one minibatch per iteration, thus providing excellent control of the sampling bias as a function of the stepsize. We apply the algorithm to explore local modes of the posterior distribution of Bayesian neural networks (BNNs) and evaluate the calibration performance of the posterior predictive probabilities for neural networks with convolutional neural network architectures for classification problems on three different datasets (Fashion-MNIST, Celeb-A and chest X-ray). Our results indicate that BNNs sampled with SMS-UBU can offer significantly better calibration performance compared to standard methods of training and stochastic weight averaging.

stat.ML↗

arXiv2024

Correction to "Wasserstein distance estimates for the distributions of numerical approximations to ergodic stochastic differential equations"

A method for analyzing non-asymptotic guarantees of numerical discretizations of ergodic SDEs in Wasserstein-2 distance is presented by Sanz-Serna and Zygalakis in ``Wasserstein distance estimates for the distributions of numerical approximations to ergodic stochastic differential equations". They analyze the UBU integrator which is strong order two and only requires one gradient evaluation per step, resulting in desirable non-asymptotic guarantees, in particular $\mathcal{O}(d^{1/4}ε^{-1/2})$ steps to reach a distance of $ε> 0$ in Wasserstein-2 distance away from the target distribution. However, there is a mistake in the local error estimates in Sanz-Serna and Zygalakis (2021), in particular, a stronger assumption is needed to achieve these complexity estimates. This note reconciles the theory with the dimension dependence observed in practice in many applications of interest.

stat.ML↗

arXiv2024

Contraction and Convergence Rates for Discretized Kinetic Langevin Dynamics

We provide a framework to analyze the convergence of discretized kinetic Langevin dynamics for $M$-$\nabla$Lipschitz, $m$-convex potentials. Our approach gives convergence rates of $\mathcal{O}(m/M)$, with explicit stepsize restrictions, which are of the same order as the stability threshold for Gaussian targets and are valid for a large interval of the friction parameter. We apply this methodology to various integration schemes which are popular in the molecular dynamics and machine learning communities. Further, we introduce the property ``$γ$-limit convergent" (GLC) to characterize underdamped Langevin schemes that converge to overdamped dynamics in the high-friction limit and which have stepsize restrictions that are independent of the friction parameter; we show that this property is not generic by exhibiting methods from both the class and its complement. Finally, we provide asymptotic bias estimates for the BAOAB scheme, which remain accurate in the high-friction limit by comparison to a modified stochastic dynamics which preserves the invariant measure.

math.NA↗

arXiv2023

Contraction Rate Estimates of Stochastic Gradient Kinetic Langevin Integrators

In previous work, we introduced a method for determining convergence rates for integration methods for the kinetic Langevin equation for $M$-$\nabla$Lipschitz $m$-log-concave densities [arXiv:2302.10684, 2023]. In this article, we exploit this method to treat several additional schemes including the method of Brunger, Brooks and Karplus (BBK) and stochastic position/velocity Verlet. We introduce a randomized midpoint scheme for kinetic Langevin dynamics, inspired by the recent scheme of Bou-Rabee and Marsden [arXiv:2211.11003, 2022]. We also extend our approach to stochastic gradient variants of these schemes under minimal extra assumptions. We provide convergence rates of $\mathcal{O}(m/M)$, with explicit stepsize restriction, which are of the same order as the stability thresholds for Gaussian targets and are valid for a large interval of the friction parameter. We compare the contraction rate estimates of many kinetic Langevin integrators from molecular dynamics and machine learning. Finally we present numerical experiments for a Bayesian logistic regression example.

math.NA↗

arXiv2022

Randomized Time Riemannian Manifold Hamiltonian Monte Carlo

Hamiltonian Monte Carlo (HMC) algorithms which combine numerical approximation of Hamiltonian dynamics on finite intervals with stochastic refreshment and Metropolis correction are popular sampling schemes, but it is known that they may suffer from slow convergence in the continuous time limit. A recent paper of Bou-Rabee and Sanz-Serna (Ann. Appl. Prob., 27:2159-2194, 2017) demonstrated that this issue can be addressed by simply randomizing the duration parameter of the Hamiltonian paths. In this article, we use the same idea to enhance the sampling efficiency of a constrained version of HMC, with potential benefits in a variety of application settings. We demonstrate both the conservation of the stationary distribution and the ergodicity of the method. We also compare the performance of various schemes in numerical studies of model problems, including an application to high-dimensional covariance estimation.

stat.CO↗