Searcharxiv⌕ Search

arXiv subjects

Gareth O. Roberts

Publications and source records attributed to Gareth O. Roberts.

At least 19 recordsLinked to original sources

Limitations of nonreversible lifts for subgeometric diffusions

We study the effect of lifting on continuous-time Markov processes with potentially subgeometric convergence rates. Using a series of weak Poincaré-type inequalities, we derive a bound on the maximum acceleration that can be obtained through lifting. We present several examples to explore the sharpness of our derived bound, where the resulting bound has the same square-root order as in the spectral gap setting. Our results provide a framework for understanding the potential and limitations of non-reversible lifting beyond geometric convergence.

stat.CO↗

Large sample scaling analysis of the Zig-Zag algorithm for Bayesian inference

Piecewise deterministic Markov processes provide scalable methods for sampling from the posterior distributions in big data settings by admitting principled sub-sampling strategies that do not bias the output. An important example is the Zig-Zag process of [Ann. Stats. 47 (2019) 1288 - 1320] where clever sub-sampling has been shown to produce an essentially independent sample at a cost that does not scale with the size of the data. However, sub-sampling also leads to slower convergence and poor mixing of the process, a behaviour which questions the promised scalability of the algorithm. We provide a large sample scaling analysis of the Zig-Zag process and its sub-sampling versions in settings of parametric Bayesian inference. In the transient phase of the algorithm, we show that the Zig-Zag trajectories are well approximated by the solution to a system of ODEs. These ODEs possess a drift in the direction of decreasing KL-divergence between the assumed model and the true distribution and are explicitly characterized in the paper. In the stationary phase, we give weak convergence results for different versions of the Zig-Zag process. Based on our results, we estimate that for large data sets of size n, using suitable control variates with sub-sampling in Zig-Zag, the algorithm costs O(1) to obtain an essentially independent sample; a computational speed-up of O(n) over the canonical version of Zig-Zag and other traditional MCMC methods

stat.CO↗

Approximate Bayesian Fusion

Distributed inference plays an important role in modern computational statistics and machine learning. In the frequentist setting, methods can often aggregate results by simply averaging estimators computed on data subsets. However, distributed Bayesian inference is more challenging, as it requires reconstructing the full posterior by combining subposterior distributions computed on different machines. Bayesian Fusion performs distributed Bayesian inference exactly, but at a sizeable computational cost. We propose a method that uses numerical approximations to reduce the cost of the most computationally expensive steps of Bayesian Fusion, and expands its domain of applicability. We bound the numerical error as a function of the step size of the approximation. Under regularity conditions, our approach can approximate the target posterior distribution to an arbitrary degree of accuracy, and is a practical and efficient alternative for large-scale distributed Bayesian inference. In numerical experiments on a logistic regression with more than $300{,}000$ observations, our method reaches the accuracy reported for Divide-and-Conquer Bayesian Fusion about two orders of magnitude faster. On a proportional hazards model with non-Gaussian posteriors, which lacks the lower bounds that Bayesian Fusion requires, our method reproduces funnel features of the posterior that simpler methods like Consensus Monte Carlo miss.

stat.CO↗

Weak Poincaré inequalities for two-component Deterministic-scan Metropolis-within-Gibbs samplers

Using the framework of weak Poincaré inequalities, we analyze the convergence properties of deterministic-scan Metropolis-within-Gibbs samplers, an important class of Markov chain Monte Carlo algorithms. Our analysis applies to nonreversible Markov chains and yields explicit (subgeometric) convergence bounds through novel comparison techniques based on Dirichlet forms. We show that the joint chain inherits the convergence behavior of the marginal chain and conversely. In addition, we establish several fundamental results for weak Poincaré inequalities for discrete-time Markov chains, such as a tensorization property for independent chains. We apply our theoretical results through applications to algorithms for Bayesian inference for a hierarchical regression model and a diffusion model under discretely-observed data.

stat.CO↗

Optimal sampling strategies in event-chain Monte Carlo

Event-chain Monte Carlo (ECMC) has revolutionised computational sampling over recent years, providing a powerful alternative to the molecular-dynamics (MD) and Hamiltonian Monte Carlo (HMC) algorithms. Each method outperforms the ubiquitous random-walk Metropolis algorithm by advancing particles along deterministic trajectories, but ECMC achieves this without being constrained by Newtonian dynamics. Recent advances exploited this dynamical freedom to induce a collective particle dynamics that relax local density variations on fast timescales. In a foundational model of $N$ pairwise-interacting particles on the 1D torus, this led to an $O(N^{3/4})$ improvement on the industry-leading computational efficiency of MD and HMC - but with an impractically small prefactor at low/high mean particle density for general attractive/repulsive interactions. Here we present a universal framework that generalises this high-efficiency sampling strategy to all translationally symmetric pairwise models on the torus - creating the potential to surpass MD and HMC as the state of the art. We also numerically elucidate the collective particle dynamics and discuss broad impact across the physical sciences and computational statistics.

cond-mat.stat-mech↗

Annealed Leap-Point Sampler for Multimodal Target Distributions

In Bayesian statistics, exploring high-dimensional multimodal posterior distributions poses major challenges for existing MCMC approaches. This paper introduces the Annealed Leap-Point Sampler (ALPS), which augments the target distribution state space with modified annealed (cooled) distributions, in contrast to traditional tempering approaches. The coldest state is chosen such that its annealed density is well-approximated locally by a Laplace approximation. This allows for automated setup of a scalable mode-leaping independence sampler. ALPS requires an exploration component to search for the mode locations, which can either be run adaptively in parallel to improve these mode-jumping proposals, or else as a pre-computation step. A theoretical analysis shows that for a d-dimensional problem the coolest temperature level required only needs to be linear in dimension, $\mathcal{O}\left(d\right)$, implying that the number of iterations needed for ALPS to converge is $\mathcal{O}\left(d\right)$ (typically leading to overall complexity $\mathcal{O}\left(d^3\right)$ when computational cost per iteration is taken into account). ALPS is illustrated on several complex, multimodal distributions that arise from real-world applications. This includes a seemingly-unrelated regression (SUR) model of longitudinal data from U.S. manufacturing firms, as well as a spectral density model that is used in analytical chemistry for identification of molecular biomarkers.

stat.ME↗

Adaptive Stereographic MCMC

In order to tackle the problem of sampling from heavy tailed, high dimensional distributions via Markov Chain Monte Carlo (MCMC) methods, Yang, Latuszyński, and Roberts (2022) (arXiv:2205.12112) introduces the stereographic projection as a tool to compactify $\mathbb{R}^d$ and transform the problem into sampling from a density on the unit sphere $\mathbb{S}^d$. However, the improvement in algorithmic efficiency, as well as the computational cost of the implementation, are still significantly impacted by the parameters used in this transformation. To address this, we introduce adaptive versions three stereographic MCMC algorithms - the Stereographic Random Walk (SRW), the Stereographic Slice Sampler (SSS), and the Stereographic Bouncy Particle Sampler (SBPS) - which automatically update the parameters of the algorithms as the run progresses. The adaptive setup allows to better exploit the power of the stereographic projection, even when the target distribution is neither centered nor homogeneous. Unlike Hamiltonian Monte Carlo (HMC) and other off-the-shelf MCMC samplers, the resulting algorithms are robust to starting far from the mean in heavy-tailed, high-dimensional settings. To prove convergence properties, we develop a novel framework for the analysis of adaptive MCMC algorithms over collections of simultaneously uniformly ergodic Markov operators, which is applicable to continuous-time processes, such as SBPS. This framework allows us to obtain $\mathcal{L}^2$ and almost sure convergence results, and a CLT for our adaptive stereographic algorithms.

stat.CO↗

Bridging Theory and Practice in Efficient Gaussian Process-Based Statistical Modeling for Large Datasets

Geostatistics is a branch of statistics concerned with stochastic processes over continuous domains, with Gaussian processes (GPs) providing a flexible and principled modelling framework. However, the high computational cost of simulating or computing likelihoods with GPs limits their scalability to large datasets. This paper introduces the piecewise continuous Gaussian process (PCGP), a new process that retains the rich probabilistic structure of traditional GPs while offering substantial computational efficiency. As will be shown and discussed, existing scalable approaches that define stochastic processes on continuous domains -- such as the nearest-neighbour GP (NNGP) and the radial-neighbour GP (RNGP) -- rely on conditional independence structures that effectively constrain the measurable space on which the processes are defined, which may induce undesirable probabilistic behaviour and compromise their practical applicability, particularly in complex latent GP models. The PCGP mitigates these limitations and provides a theoretically grounded and computationally efficient alternative, as demonstrated through numerical illustrations.

stat.CO↗

Privacy Guarantees in Posterior Sampling under Contamination

In recent years, differential privacy has been adopted by tech-companies and governmental agencies as the standard for measuring privacy in algorithms. In this article, we study differential privacy in Bayesian posterior sampling settings. We begin by considering differential privacy in the most common privatisation setting in which Laplace or Gaussian noise is injected into the output. In an effort to achieve better differential privacy, we consider adopting {\em Huber's contamination model} for use within privacy settings, and replace at random data points with samples from a heavy-tailed distribution ({\em instead} of injecting noise into the output). We derive bounds for the differential privacy level $(ε,δ)$ of our approach, without requiring bounded observation and parameter spaces, a restriction commonly imposed in the literature. We further consider for our approach the effect of sample size on the privacy level and the rate at which $(ε,δ)$ converges to zero. Asymptotically, our contamination approach is fully private with no information loss. We also provide examples of inference models for which our approach applies, with theoretical convergence rate analysis and simulation studies.

math.ST↗

On micromodes in Bayesian posterior distributions and their implications for MCMC

We investigate the existence and severity of local modes in posterior distributions from Bayesian analyses. These are known to occur in posterior tails resulting from heavy-tailed error models such as those used in robust regression. To understand this phenomenon clearly, we consider in detail location models with Student-$t$ errors in dimension $d$ with sample size $n$. For sufficiently heavy-tailed data-generating distributions, extreme observations become increasingly isolated as $n \to \infty$. We show that each such observation induces a unique local posterior mode with probability tending to $1$. We refer to such a local mode as a micromode. These micromodes are typically small in height but their domains of attraction are large and grow polynomially with $n$. We then connect this posterior geometry to computation. We establish an Arrhenius law for the time taken by one-dimensional piecewise deterministic Monte Carlo algorithms to exit these micromodes. Our analysis identifies a phase transition where a misspecified and overly underdispersed model causes exit times to increase sharply, leading to a pronounced deterioration in sampling performance.

math.ST↗

Sub-Cauchy Sampling: Escaping the Dark Side of the Moon

We introduce a Markov chain Monte Carlo algorithm based on Sub-Cauchy Projection, a geometric transformation that generalizes stereographic projection by mapping Euclidean space into a spherical cap of a hyper-sphere, referred to as the complement of the dark side of the moon. We prove that our proposed method is uniformly ergodic for sub-Cauchy targets, namely targets whose tails are at most as heavy as a multidimensional Cauchy distribution, and show empirically its performance for challenging high-dimensional problems. The simplicity and broad applicability of our approach open new opportunities for Bayesian modeling and computation with heavy-tailed distributions in settings where most existing methods are unreliable.

stat.CO↗

Scalability of Metropolis-within-Gibbs schemes for high-dimensional Bayesian models

We study general coordinate-wise MCMC schemes (such as Metropolis-within-Gibbs samplers), which are commonly used to fit Bayesian non-conjugate hierarchical models. We relate their convergence properties to the ones of the corresponding (potentially not implementable) Gibbs sampler through the notion of conditional conductance. This allows us to study the performances of popular Metropolis-within-Gibbs schemes for non-conjugate hierarchical models, in high-dimensional regimes where both number of datapoints and parameters increase. Given random data-generating assumptions, we establish dimension-free convergence results, which are in close accordance with numerical evidences. Applications to Bayesian models for binary regression with unknown hyperparameters and discretely observed diffusions are also discussed. Motivated by such statistical applications, auxiliary results of independent interest on approximate conductances and perturbation of Markov operators are provided.

stat.CO↗

A discomfort-informed adaptive Gibbs sampler for finite mixture models

Finite mixture models are frequently used to uncover latent structures in high-dimensional datasets (e.g.\ identifying clusters of patients in electronic health records). The inference of such structures can be performed in a Bayesian framework, and involves the use of sampling algorithms such as Gibbs samplers aimed at deriving posterior distribution of the probabilities of observations to belong to specific clusters. Unfortunately, traditional implementations of Gibbs samplers in this context often face critical challenges, such as inefficient use of computational resources and unnecessary updates for observations that are highly likely to remain in their current cluster. This paper introduces a new adaptive Gibbs sampler that improves the convergence efficiency over existing methods. In particular, our sampler is guided by a function that, at each iteration, uses the past of the chain to focus the updating on observations potentially misclassified in the current clustering, i.e.\ those with a low probability of belonging to their current component. Through simulation studies and two real data analyses, we empirically demonstrate that, in terms of convergence time, our method tends to perform more efficiently compared to state-of-the-art approaches.

stat.ME↗

Differential privacy guarantees of Markov chain Monte Carlo algorithms

This paper aims to provide differential privacy (DP) guarantees for Markov chain Monte Carlo (MCMC) algorithms. In a first part, we establish DP guarantees on samples output by MCMC algorithms as well as Monte Carlo estimators associated with these methods under assumptions on the convergence properties of the underlying Markov chain. In particular, our results highlight the critical condition of ensuring the target distribution is differentially private itself. In a second part, we specialise our analysis to the unadjusted Langevin algorithm and stochastic gradient Langevin dynamics and establish guarantees on their (Rényi) DP. To this end, we develop a novel methodology based on Girsanov's theorem combined with a perturbation trick to obtain bounds for an unbounded domain and in a non-convex setting. We establish: (i) uniform in $n$ privacy guarantees when the state of the chain after $n$ iterations is released, (ii) bounds on the privacy of the entire chain trajectory. These findings provide concrete guidelines for privacy-preserving MCMC.

stat.ML↗

Almost sure convergence rates of adaptive increasingly rare Markov chain Monte Carlo

We consider adaptive increasingly rare Markov chain Monte Carlo (MCMC) algorithms, which are adaptive MCMC methods, where the adaptation concerning the "past'' happens less and less frequently over time. Under a contraction assumption with respect to a Wasserstein-like function we deduce upper bounds of the convergence rate of Monte Carlo sums taking a renormalisation factor into account that is "almost'' the one that appears in a law of the iterated logarithm. We demonstrate the applicability of our results by considering different settings, among which are those of simultaneous geometric and uniform ergodicity. All proofs are carried out on an augmented state space, including the classical non-augmented setting as a special case. In contrast to other adaptive MCMC limit theory, some technical assumptions, like diminishing adaptation, are not needed.

math.NA↗

Transient regime of piecewise deterministic Monte Carlo algorithms

Piecewise Deterministic Markov Processes (PDMPs) such as the Bouncy Particle Sampler and the Zig-Zag Sampler, have gained attention as continuous-time counterparts of classical Markov chain Monte Carlo. We study their transient regime under convex potentials, namely how trajectories that start in low-probability regions move toward higher-probability sets. Using fluid-limit arguments with a decomposition of the generator into fast and slow parts, we obtain deterministic ordinary differential equation descriptions of early-stage behaviour. The fast dynamics alone are non-ergodic because once the event rate reaches zero it does not restart. The slow component reactivates the dynamics, so averaging remains valid when taken over short micro-cycles rather than with respect to an invariant law. Using the expected number of jump events as a cost proxy for gradient evaluations, we find that for Gaussian targets the transient cost of PDMP methods is comparable to that of random-walk Metropolis. For convex heavy-tailed families with subquadratic growth, PDMP methods can be more efficient when event simulation is implemented well. Forward Event-Chain and Coordinate Samplers can, under the same assumptions, reach the typical set with an order-one expected number of jumps. For the Zig-Zag Sampler we show that, under a diagonal-dominance condition, the transient choice of direction coincides with the solution of a box-constrained quadratic program; outside that regime we give a formal derivation and a piecewise-smooth update rule that clarifies the roles of the gradient and the Hessian. These results provide theoretical insight and practical guidance for the use of PDMP samplers in large-scale inference.

stat.CO↗

Statistical Disaggregation -- a Monte Carlo Approach for Imputation under Constraints

Equality-constrained models naturally arise in problems in which measurements are taken at different levels of resolution. The challenge in this setting is that the models usually induce a joint distribution which is intractable. Resorting to instead sampling from the joint distribution by means of a Monte Carlo approach is also challenging. For example, a naive rejection sampling does not work when the probability mass of the constraint is zero. A typical example of such constrained problems is to learn energy consumption for a higher resolution level based on data at a lower resolution, e.g., to decompose a daily reading into readings at a finer level. We introduce a novel Monte Carlo sampling algorithm based on Langevin diffusions and rejection sampling to solve the problem of sampling from equality-constrained models. Our method has the advantage of being exact for linear constraints and naturally deals with multimodal distributions on arbitrary constraints. We test our method on statistical disaggregation problems for electricity consumption datasets, and our approach provides better uncertainty estimation and accuracy in data imputation compared with other naive/unconstrained methods.

stat.CO↗

Quantifying the Speed-Up from Non-Reversibility in MCMC Tempering Algorithms

We investigate the increase in efficiency of simulated and parallel tempering MCMC algorithms when using non-reversible updates to give them "momentum". By making a connection to a certain simple discrete Markov chain, we show that, under appropriate assumptions, the non-reversible algorithms still exhibit diffusive behaviour, just on a different time scale. We use this to argue that the optimally scaled versions of the non-reversible algorithms are indeed more efficient than the optimally scaled versions of their traditional reversible counterparts, but only by a modest speed-up factor of about 42%.

math.ST↗