Searcharxiv⌕ Search

arXiv subjects

Jack Jewson

Publications and source records attributed to Jack Jewson.

17 recordsLinked to original sources
arXiv2026

Variational predictive resampling

Bayesian inference provides principled uncertainty quantification, but accurate posterior sampling with MCMC can be computationally prohibitive for modern applications. Variational inference (VI) offers a scalable alternative and often yields accurate predictive distributions, but cheap variational families such as mean-field (MF) can produce over-concentrated approximations that miss posterior dependence. We propose variational predictive resampling (VPR), a scalable posterior sampling method that exploits VI's predictive strength within a predictive-resampling framework to better approximate the Bayesian posterior. Given a prior-likelihood pair, VPR repeatedly imputes future observations from the current variational predictive, updates the variational approximation after each imputation, and records the parameter value implied by the completed sample. We establish conditions under which the law of the parameter returned by VPR is well defined and show that its finite-horizon approximation converges to this limit. In a tractable Gaussian location model, we show that VPR with MF variational predictives converges to the exact Bayesian posterior, whereas the optimal MF-VI approximation retains a non-vanishing asymptotic gap. Experiments on linear regression, logistic regression, and hierarchical linear mixed-effects models demonstrate that VPR substantially improves posterior uncertainty quantification and recovers posterior dependence missed by MF-VI, while remaining computationally competitive with, and often more efficient than, MCMC.

stat.ME↗
arXiv2026

Bayesian Variable Selection with the Quasi-Posterior

The Bayesian approach provides powerful methods for variable selection. The ability to incorporate sparsity through prior beliefs and account for parameter uncertainty allows Bayesian variable selection to consistently identify which of the variables are active and exhibit strong finite-sample performance. However, Bayesian methods require the correct specification of full likelihoods for the data, and there is increasing awareness of the problems that model misspecification causes for variable selection. Current approaches to mitigate misspecification either require complex models, detracting from the interpretability of the variable selection task, or move outside rigorous Bayesian uncertainty quantification and provide no recognised method for variable selection. This paper establishes the model quasi-posterior as a principled tool for variable selection. We prove that the model quasi-posterior shares desirable properties of Bayesian variable selection without requiring full likelihood specification. Instead, the quasi-posterior combines a prior with a quasi-likelihood and requires only specification of mean and variance functions, and is therefore robust to other aspects of the data. Marginalising the quasi-likelihood is analytically possible for linear regression, and Laplace approximations are used beyond this to ensure computational tractability. Extensive simulation studies illustrate improved variable selection accuracy across diverse data-generating scenarios when compared with likelihood-based Bayesian variable selection and lasso-penalized methods. We further demonstrate practical relevance through applications to real datasets from social science and genomics.

stat.ME↗
arXiv2025

Bayesian computation for high-dimensional Gaussian Graphical Models with spike-and-slab priors

Gaussian graphical models are widely used to infer dependence structures. Bayesian methods are appealing to quantify uncertainty associated with structural learning, i.e., the plausibility of conditional independence statements given the data, and parameter estimates. However, computational demands have limited their application when the number of variables is large, which prompted the use of pseudo-Bayesian approaches. We propose fully Bayesian algorithms that provably scale to high dimensions when the data-generating precision matrix is sparse, at a similar cost to the best pseudo-Bayesian methods. First, a Metropolis-Hastings-within-Block-Gibbs algorithm that allows row-wise updates of the precision matrix, using local moves. Second, a global proposal that enables adding or removing multiple edges within a row, which can help explore multi-modal posteriors. We obtain mixing bounds for both samplers relative to an ideal Gibbs sampler that are dimension-free under suitable settings, and prove that this ideal Gibbs sampler is geometrically ergodic. We also provide worst-case polynomial bounds on per-iteration costs, though in practice the cost is lower by using sparse linear algebra. Our examples show that the methods extend the applicability of exact Bayesian inference from roughly 100 to roughly 1000 variables (equivalently, from 5,000 edges to 500,000 edges).

stat.ME↗
arXiv2025

Probabilistic Programming with Sufficient Statistics for faster Bayesian Computation

Probabilistic programming methods have revolutionised Bayesian inference, making it easier than ever for practitioners to perform Markov-chain-Monte-Carlo sampling from non-conjugate posterior distributions. Here we focus on Stan, arguably the most used probabilistic programming tool for Bayesian inference (Carpenter et al., 2017), and its interface with R via the brms (Burkner, 2017) and rstanarm (Goodrich et al., 2024) packages. Although easy to implement, these tools can become computationally prohibitive when applied to datasets with many observations or models with numerous parameters. While the use of sufficient statistics is well-established in theory, it has been surprisingly overlooked in state-of-the-art Stan software. We show that when the likelihood can be written in terms of sufficient statistics, considerable computational improvements can be made to current implementations. We demonstrate how this approach provides accurate inference at a fraction of the time than state-of-the-art implementations for Gaussian linear regression models with non-conjugate priors, hierarchical random effects models, and factor analysis models. Our results also show that moderate computational gains can be achieved even in models where the likelihood can only be partially written in terms of sufficient statistics.

stat.CO↗
arXiv2024

Exact Sampling of Gibbs Measures with Estimated Losses

A popular strategy for ameliorating some of the shortcomings of Bayesian posterior inference is to instead target a Gibbs measure based on losses that connect a parameter of interest to observed data, and which are known to be robust to misspecification. Existing theory for these procedures treats these losses as being analytically available, but in many situations these losses must be stochastically estimated using pseudo-observations. In these settings, and even under strong assumptions, we show that posteriors based on standard Markov chain Monte Carlo (MCMC) algorithms exhibit strong dependence on the number of these pseudo-observations, and require utilizing a diverging number of pseudo-observations to ensure posterior concentration. To remedy this issue, we introduce a modified piecewise deterministic Markov process (PDMP) sampler, and formally show that its posterior draws have no dependence on the number of pseudo-observations used to estimate the loss within a Gibbs measure. We verify the practical utility of this approach on five examples spanning intractable likelihoods and intractable losses.

math.ST↗
arXiv2023

Differentially Private Statistical Inference through $\beta$-Divergence One Posterior Sampling

Differential privacy guarantees allow the results of a statistical analysis involving sensitive data to be released without compromising the privacy of any individual taking part. Achieving such guarantees generally requires the injection of noise, either directly into parameter estimates or into the estimation process. Instead of artificially introducing perturbations, sampling from Bayesian posterior distributions has been shown to be a special case of the exponential mechanism, producing consistent, and efficient private estimates without altering the data generative process. The application of current approaches has, however, been limited by their strong bounding assumptions which do not hold for basic models, such as simple linear regressors. To ameliorate this, we propose $\beta$D-Bayes, a posterior sampling scheme from a generalised posterior targeting the minimisation of the $\beta$-divergence between the model and the data generating process. This provides private estimation that is generally applicable without requiring changes to the underlying model and consistently learns the data generating parameter. We show that $\beta$D-Bayes produces more precise inference estimation for the same privacy guarantees, and further facilitates differentially private estimation via posterior sampling for complex classifiers and continuous regression models such as neural networks for the first time.

stat.ML↗
arXiv2023

Bayesian Sparse Vector Autoregressive Switching Models with Application to Human Gesture Phase Segmentation

We propose a sparse vector autoregressive (VAR) hidden semi-Markov model (HSMM) for modeling temporal and contemporaneous (e.g. spatial) dependencies in multivariate nonstationary time series. The HSMM's generic state distribution is embedded in a special transition matrix structure, facilitating efficient likelihood evaluations and arbitrary approximation accuracy. To promote sparsity of the VAR coefficients, we deploy an $l_1$-ball projection prior, which combines differentiability with a positive probability of obtaining exact zeros, achieving variable selection within each switching state. This also facilitates posterior estimation via Hamiltonian Monte Carlo (HMC). We further place non-local priors on the parameters of the HSMM dwell distribution improving the ability of Bayesian model selection to distinguish whether the data is better supported by the simpler hidden Markov model (HMM), or the more flexible HSMM. Our proposed methodology is illustrated via an application to human gesture phase segmentation based on sensor data, where we successfully identify and characterize the periods of rest and active gesturing, as well as the dynamical patterns involved in the gesture movements associated with each of these states.

stat.AP↗
arXiv2023

On the Stability of General Bayesian Inference

We study the stability of posterior predictive inferences to the specification of the likelihood model and perturbations of the data generating process. In modern big data analyses, useful broad structural judgements may be elicited from the decision-maker but a level of interpolation is required to arrive at a likelihood model. As a result, an often computationally convenient canonical form is used in place of the decision-maker's true beliefs. Equally, in practice, observational datasets often contain unforeseen heterogeneities and recording errors and therefore do not necessarily correspond to how the process was idealised by the decision-maker. Acknowledging such imprecisions, a faithful Bayesian analysis should ideally be stable across reasonable equivalence classes of such inputs. We are able to guarantee that traditional Bayesian updating provides stability across only a very strict class of likelihood models and data generating processes, requiring the decision-maker to elicit their beliefs and understand how the data was generated with an unreasonable degree of accuracy. On the other hand, a generalised Bayesian alternative using the $\beta$-divergence loss function is shown to be stable across practical and interpretable neighbourhoods, providing assurances that posterior inferences are not overly dependent on accidentally introduced spurious specifications or data collection errors. We illustrate this in linear regression, binary classification, and mixture modelling examples, showing that stable updating does not compromise the ability to learn about the data generating process. These stability results provide a compelling justification for using generalised Bayes to facilitate inference under simplified canonical models.

stat.ME↗
arXiv2022

Graphical model inference with external network data

We consider two applications where we study how dependence structure between many variables is linked to external network data. We first study the interplay between social media connectedness and the co-evolution of the COVID-19 pandemic across USA counties. We next study study how the dependence between stock market returns across firms relates to similarities in economic and policy indicators from text regulatory filings. Both applications are modelled via Gaussian graphical models where one has external network data. We develop spike-and-slab and graphical LASSO frameworks to integrate the network data, both facilitating the interpretation of the graphical model and improving inference. The goal is to detect when the network data relates to the graphical model and, if so, explain how. We found that counties strongly connected on Facebook are more likely to have similar COVID-19 evolution (positive partial correlations), accounting for various factors driving the mean. We also found that the association in stock market returns depends in a stronger fashion on economic than on policy indicators. The examples show that data integration can improve interpretation, statistical accuracy, and out-of-sample prediction, in some instances using significantly sparser graphical models.

stat.AP↗
arXiv2021

Mitigating Statistical Bias within Differentially Private Synthetic Data

Increasing interest in privacy-preserving machine learning has led to new and evolved approaches for generating private synthetic data from undisclosed real data. However, mechanisms of privacy preservation can significantly reduce the utility of synthetic data, which in turn impacts downstream tasks such as learning predictive models or inference. We propose several re-weighting strategies using privatised likelihood ratios that not only mitigate statistical bias of downstream estimators but also have general applicability to differentially private generative models. Through large-scale empirical evaluation, we show that private importance weighting provides simple and effective privacy-compliant augmentation for general applications of synthetic data.

stat.ML↗
arXiv2021

General Bayesian Loss Function Selection and the use of Improper Models

Statisticians often face the choice between using probability models or a paradigm defined by minimising a loss function. Both approaches are useful and, if the loss can be re-cast into a proper probability model, there are many tools to decide which model or loss is more appropriate for the observed data, in the sense of explaining the data's nature. However, when the loss leads to an improper model, there are no principled ways to guide this choice. We address this task by combining the Hyv\"arinen score, which naturally targets infinitesimal relative probabilities, and general Bayesian updating, which provides a unifying framework for inference on losses and models. Specifically we propose the H-score, a general Bayesian selection criterion and prove that it consistently selects the (possibly improper) model closest to the data-generating truth in Fisher's divergence. We also prove that an associated H-posterior consistently learns optimal hyper-parameters featuring in loss functions, including a challenging tempering parameter in generalised Bayesian inference. As salient examples, we consider robust regression and non-parametric density estimation where popular loss functions define improper models for the data and hence cannot be dealt with using standard model selection tools. These examples illustrate advantages in robustness-efficiency trade-offs and provide a Bayesian implementation for kernel density estimation, opening a new avenue for Bayesian non-parametrics.

stat.ME↗
arXiv2020

Foundations of Bayesian Learning from Synthetic Data

There is significant growth and interest in the use of synthetic data as an enabler for machine learning in environments where the release of real data is restricted due to privacy or availability constraints. Despite a large number of methods for synthetic data generation, there are comparatively few results on the statistical properties of models learnt on synthetic data, and fewer still for situations where a researcher wishes to augment real data with another party's synthesised data. We use a Bayesian paradigm to characterise the updating of model parameters when learning in these settings, demonstrating that caution should be taken when applying conventional learning algorithms without appropriate consideration of the synthetic data generating process and learning task. Recent results from general Bayesian updating support a novel and robust approach to Bayesian synthetic-learning founded on decision theory that outperforms standard approaches across repeated experiments on supervised learning and inference problems.

cs.LG↗
arXiv2020

Bayesian Approximations to Hidden Semi-Markov Models for Telemetric Monitoring of Physical Activity

We propose a Bayesian hidden Markov model for analyzing time series and sequential data where a special structure of the transition probability matrix is embedded to model explicit-duration semi-Markovian dynamics. Our formulation allows for the development of highly flexible and interpretable models that can integrate available prior information on state durations while keeping a moderate computational cost to perform efficient posterior inference. We show the benefits of choosing a Bayesian approach for HSMM estimation over its frequentist counterpart, in terms of model selection and out-of-sample forecasting, also highlighting the computational feasibility of our inference procedure whilst incurring negligible statistical error. The use of our methodology is illustrated in an application relevant to e-Health, where we investigate rest-activity rhythms using telemetric activity data collected via a wearable sensing device. This analysis considers for the first time Bayesian model selection for the form of the explicit state dwell distribution. We further investigate the inclusion of a circadian covariate into the emission density and estimate this in a data-driven manner.

stat.ME↗
arXiv2019

Generalized Variational Inference: Three arguments for deriving new Posteriors

We advocate an optimization-centric view on and introduce a novel generalization of Bayesian inference. Our inspiration is the representation of Bayes' rule as infinite-dimensional optimization problem (Csiszar, 1975; Donsker and Varadhan; 1975, Zellner; 1988). First, we use it to prove an optimality result of standard Variational Inference (VI): Under the proposed view, the standard Evidence Lower Bound (ELBO) maximizing VI posterior is preferable to alternative approximations of the Bayesian posterior. Next, we argue for generalizing standard Bayesian inference. The need for this arises in situations of severe misalignment between reality and three assumptions underlying standard Bayesian inference: (1) Well-specified priors, (2) well-specified likelihoods, (3) the availability of infinite computing power. Our generalization addresses these shortcomings with three arguments and is called the Rule of Three (RoT). We derive it axiomatically and recover existing posteriors as special cases, including the Bayesian posterior and its approximation by standard VI. In contrast, approximations based on alternative ELBO-like objectives violate the axioms. Finally, we study a special case of the RoT that we call Generalized Variational Inference (GVI). GVI posteriors are a large and tractable family of belief distributions specified by three arguments: A loss, a divergence and a variational family. GVI posteriors have appealing properties, including consistency and an interpretation as approximate ELBO. The last part of the paper explores some attractive applications of GVI in popular machine learning models, including robustness and more appropriate marginals. After deriving black box inference schemes for GVI posteriors, their predictive performance is investigated on Bayesian Neural Networks and Deep Gaussian Processes, where GVI can comprehensively improve upon existing methods.

stat.ML↗
arXiv2018

Doubly Robust Bayesian Inference for Non-Stationary Streaming Data with $β$-Divergences

We present the very first robust Bayesian Online Changepoint Detection algorithm through General Bayesian Inference (GBI) with $β$-divergences. The resulting inference procedure is doubly robust for both the parameter and the changepoint (CP) posterior, with linear time and constant space complexity. We provide a construction for exponential models and demonstrate it on the Bayesian Linear Regression model. In so doing, we make two additional contributions: Firstly, we make GBI scalable using Structural Variational approximations that are exact as $β\to 0$. Secondly, we give a principled way of choosing the divergence parameter $β$ by minimizing expected predictive loss on-line. Reducing False Discovery Rates of CPs from more than 90% to 0% on real world data, this offers the state of the art.

stat.ML↗
arXiv2018

Principles of Bayesian Inference using General Divergence Criteria

When it is acknowledged that all candidate parameterised statistical models are misspecified relative to the data generating process, the decision maker (DM) must currently concern themselves with inference for the parameter value minimising the KL-divergence between the model and the process (Walker, 2013). However, it has long been known that minimising the KL-divergence places a large weight on correctly capturing the tails of the sample distribution. As a result the DM is required to worry about the robustness of their model to tail misspecifications if they want to conduct principled inference. In this paper we alleviate these concerns for the DM. We advance recent methodological developments in general Bayesian updating (Bissiri, Holmes and Walker, 2016) to propose a statistically well principled Bayesian updating of beliefs targeting the minimisation of more general divergence criteria. We improve both the motivation and the statistical foundations of existing Bayesian minimum divergence estimation (Hooker and Vidyashankar, 2014; Ghosh and Basu, 2016), allowing the well principled Bayesian to target predictions from the model that are close to the genuine model in terms of some alternative divergence measure to the KL-divergence. Our principled formulation allows us to consider a broader range of divergences than have previously been considered. In fact we argue defining the divergence measure forms an important, subjective part of any statistical analysis, and aim to provide some decision theoretic rational for this selection. We illustrate how targeting alternative divergence measures can impact the conclusions of simple inference tasks, and discuss then how our methods might apply to more complicated, high dimensional models.

math.ST↗