arXiv subjects

David Nott

Publications and source records attributed to David Nott.

14 recordsLinked to original sources

arXiv2025

Fast Variational Boosting for Latent Variable Models

We consider the problem of estimating complex statistical latent variable models using variational Bayes methods. These methods are used when exact posterior inference is either infeasible or computationally expensive, and they approximate the posterior density with a family of tractable distributions. The parameters of the approximating distribution are estimated using optimisation methods. This article develops a flexible Gaussian mixture variational approximation, where we impose sparsity in the precision matrix of each Gaussian component to reflect the appropriate conditional independence structure in the model. By introducing sparsity in the precision matrix and parameterising it using the Cholesky factor, each Gaussian mixture component becomes parsimonious (with a reduced number of non-zero parameters), while still capturing the dependence in the posterior distribution. Fast estimation methods based on global and local variational boosting moves combined with natural gradients and variance reduction methods are developed. The local boosting moves adjust an existing mixture component, and optimisation is only carried out on a subset of the variational parameters of a new component. The subset is chosen to target improvement of the current approximation in aspects where it is poor. The local boosting moves are fast because only a small number of variational parameters need to be optimised. The efficacy of the approach is illustrated by using simulated and real datasets to estimate generalised linear mixed models and state space models.

stat.ME↗

arXiv2024

Multi-objective Bayesian optimization for Likelihood-Free inference in sequential sampling models of decision making

Statistical models are often defined by a generative process for simulating synthetic data, but this can lead to intractable likelihoods. Likelihood free inference (LFI) methods enable Bayesian inference to be performed in this case. Extending a popular approach to simulation-efficient LFI for single-source data, we propose Multi-objective Bayesian Optimization for Likelihood Free Inference (MOBOLFI) to perform LFI using multi-source data. MOBOLFI models a multi-dimensional discrepancy between observed and simulated data, using a separate discrepancy for each data source. The use of a multivariate discrepancy allows for approximations to individual data source likelihoods in addition to the joint likelihood, enabling detection of conflicting information and deeper understanding of the importance of different data sources in estimating individual parameters. The adaptive choice of simulation parameters using multi-objective Bayesian optimization ensures simulation efficient approximation of likelihood components for all data sources. We illustrate our approach in sequential sampling models (SSMs), which are widely used in psychology and consumer-behavior modeling. SSMs are often fitted using multi-source data, such as choice and response time. The advantages of our approach are illustrated in comparison with a single discrepancy for an SSM fitted to data assessing preferences of ride-hailing drivers in Singapore to rent electric vehicles.

stat.ME↗

arXiv2024

Regression Copulas for Multivariate Responses

We propose a novel distributional regression model for a multivariate response vector based on a copula process over the covariate space. It uses the implicit copula of a Gaussian multivariate regression, which we call a ``regression copula''. To allow for large covariate vectors their coefficients are regularized using a novel multivariate extension of the horseshoe prior. Bayesian inference and distributional predictions are evaluated using efficient variational inference methods, allowing application to large datasets. An advantage of the approach is that the marginal distributions of the response vector can be estimated separately and accurately, resulting in predictive distributions that are marginally-calibrated. Two substantive applications of the methodology highlight its efficacy in multivariate modeling. The first is the econometric modeling and prediction of half-hourly regional Australian electricity prices. Here, our approach produces more accurate distributional forecasts than leading benchmark methods. The second is the evaluation of multivariate posteriors in likelihood-free inference (LFI) of a model for tree species abundance data, extending a previous univariate regression copula LFI method. In both applications, we demonstrate that our new approach exhibits a desirable marginal calibration property.

stat.ME↗

arXiv2023

Wasserstein Gaussianization and Efficient Variational Bayes for Robust Bayesian Synthetic Likelihood

The Bayesian Synthetic Likelihood (BSL) method is a widely-used tool for likelihood-free Bayesian inference. This method assumes that some summary statistics are normally distributed, which can be incorrect in many applications. We propose a transformation, called the Wasserstein Gaussianization transformation, that uses a Wasserstein gradient flow to approximately transform the distribution of the summary statistics into a Gaussian distribution. BSL also implicitly requires compatibility between simulated summary statistics under the working model and the observed summary statistics. A robust BSL variant which achieves this has been developed in the recent literature. We combine the Wasserstein Gaussianization transformation with robust BSL, and an efficient Variational Bayes procedure for posterior approximation, to develop a highly efficient and reliable approximate Bayesian inference method for likelihood-free problems.

stat.CO↗

arXiv2021

Flexible Variational Bayes based on a Copula of a Mixture

Variational Bayes methods approximate the posterior density by a family of tractable distributions whose parameters are estimated by optimisation. Variational approximation is useful when exact inference is intractable or very costly. Our article develops a flexible variational approximation based on a copula of a mixture, which is implemented by combining boosting, natural gradient, and a variance reduction method. The efficacy of the approach is illustrated by using simulated and real datasets to approximate multimodal, skewed and heavy-tailed posterior distributions, including an application to Bayesian deep feedforward neural network regression models.

stat.CO↗

arXiv2020

Variational Approximation of Factor Stochastic Volatility Models

Estimation and prediction in high dimensional multivariate factor stochastic volatility models is an important and active research area because such models allow a parsimonious representation of multivariate stochastic volatility. Bayesian inference for factor stochastic volatility models is usually done by Markov chain Monte Carlo methods, often by particle Markov chain Monte Carlo, which are usually slow for high dimensional or long time series because of the large number of parameters and latent states involved. Our article makes two contributions. The first is to propose fast and accurate variational Bayes methods to approximate the posterior distribution of the states and parameters in factor stochastic volatility models. The second contribution is to extend this batch methodology to develop fast sequential variational updates for prediction as new observations arrive. The methods are applied to simulated and real datasets and shown to produce good approximate inference and prediction compared to the latest particle Markov chain Monte Carlo approaches, but are much faster.

stat.CO↗

arXiv2018

Bayesian Deep Net GLM and GLMM

Deep feedforward neural networks (DFNNs) are a powerful tool for functional approximation. We describe flexible versions of generalized linear and generalized linear mixed models incorporating basis functions formed by a DFNN. The consideration of neural networks with random effects is not widely used in the literature, perhaps because of the computational challenges of incorporating subject specific parameters into already complex models. Efficient computational methods for high-dimensional Bayesian inference are developed using Gaussian variational approximation, with a parsimonious but flexible factor parametrization of the covariance matrix. We implement natural gradient methods for the optimization, exploiting the factor structure of the variational covariance matrix in computation of the natural gradient. Our flexible DFNN models and Bayesian inference approach lead to a regression and classification method that has a high prediction accuracy, and is able to quantify the prediction uncertainty in a principled and convenient way. We also describe how to perform variable selection in our deep learning method. The proposed methods are illustrated in a wide range of simulated and real-data examples, and the results compare favourably to a state of the art flexible regression and classification method in the statistical literature, the Bayesian additive regression trees (BART) method. User-friendly software packages in Matlab, R and Python implementing the proposed methods are available at https://github.com/VBayesLab

stat.CO↗

arXiv2018

Checking the Model and the Prior for the Constrained Multinomial

The multinomial model is one of the simplest statistical models. When constraints are placed on the possible values for the probabilities, however, it becomes much more difficult to deal with. Model checking and checking for prior-data conflict is considered here for such models. A theorem is proved that establishes the consistency of the check on the prior. Applications are presented to models that arise in quantum state estimation as well as the Bayesian analysis of models for ordered probabilities.

math.ST↗

arXiv2017

Multilevel Monte Carlo in Approximate Bayesian Computation

In the following article we consider approximate Bayesian computation (ABC) inference. We introduce a method for numerically approximating ABC posteriors using the multilevel Monte Carlo (MLMC). A sequential Monte Carlo version of the approach is developed and it is shown under some assumptions that for a given level of mean square error, this method for ABC has a lower cost than i.i.d. sampling from the most accurate ABC approximation. Several numerical examples are given.

stat.ME↗

arXiv2013

A sequential algorithm for fast fitting of Dirichlet process mixture models

In this article we propose an improvement on the sequential updating and greedy search (SUGS) algorithm Wang and Dunson for fast fitting of Dirichlet process mixture models. The SUGS algorithm provides a means for very fast approximate Bayesian inference for mixture data which is particularly of use when data sets are so large that many standard Markov chain Monte Carlo (MCMC) algorithms cannot be applied efficiently, or take a prohibitively long time to converge. In particular, these ideas are used to initially interrogate the data, and to refine models such that one can potentially apply exact data analysis later on. SUGS relies upon sequentially allocating data to clusters and proceeding with an update of the posterior on the subsequent allocations and parameters which assumes this allocation is correct. Our modification softens this approach, by providing a probability distribution over allocations, with a similar computational cost; this approach has an interpretation as a variational Bayes procedure and hence we term it variational SUGS (VSUGS). It is shown in simulated examples that VSUGS can out-perform, in terms of density estimation and classification, the original SUGS algorithm in many scenarios. In addition, we present a data analysis for flow cytometry data, and SNP data via a three-class dirichlet process mixture model illustrating the apparent improvement over SUGS.

stat.CO↗

arXiv2010

Bayesian Adaptive Lasso

We propose the Bayesian adaptive Lasso (BaLasso) for variable selection and coefficient estimation in linear regression. The BaLasso is adaptive to the signal level by adopting different shrinkage for different coefficients. Furthermore, we provide a model selection machinery for the BaLasso by assessing the posterior conditional mode estimates, motivated by the hierarchical Bayesian interpretation of the Lasso. Our formulation also permits prediction using a model averaging strategy. We discuss other variants of this new approach and provide a unified framework for variable selection using flexible penalties. Empirical evidence of the attractiveness of the method is demonstrated via extensive simulation studies and data analysis.

stat.ME↗

arXiv2010

The Predictive Lasso

We propose a shrinkage procedure for simultaneous variable selection and estimation in generalized linear models (GLMs) with an explicit predictive motivation. The procedure estimates the coefficients by minimizing the Kullback-Leibler divergence of a set of predictive distributions to the corresponding predictive distributions for the full model, subject to an $l_1$ constraint on the coefficient vector. This results in selection of a parsimonious model with similar predictive performance to the full model. Thanks to its similar form to the original lasso problem for GLMs, our procedure can benefit from available $l_1$-regularization path algorithms. Simulation studies and real-data examples confirm the efficiency of our method in terms of predictive performance on future observations.

stat.ME↗

arXiv2009

Bayesian projection approaches to variable selection and exploring model uncertainty

A Bayesian approach to variable selection which is based on the expected Kullback-Leibler divergence between the full model and its projection onto a submodel has recently been suggested in the literature. Here we extend this idea by considering projections onto subspaces defined via some form of $L_1$ constraint on the parameter in the full model. This leads to Bayesian model selection approaches related to the lasso. In the posterior distribution of the projection there is positive probability that some components are exactly zero and the posterior distribution on the model space induced by the projection allows exploration of model uncertainty. We also consider use of the approach in structured variable selection problems such as ANOVA models where it is desired to incorporate main effects in the presence of interactions. Here we make use of projections related to the non-negative garotte which are able to respect the hierarchical constraints. We also prove a consistency result concerning the posterior distribution on the model induced by the projection, and show that for some projections related to the adaptive lasso and non-negative garotte the posterior distribution concentrates on the true model asymptotically.

stat.ME↗

arXiv2007

Variable Selection and Model Averaging in Semiparametric Overdispersed Generalized Linear Models

We express the mean and variance terms in a double exponential regression model as additive functions of the predictors and use Bayesian variable selection to determine which predictors enter the model, and whether they enter linearly or flexibly. When the variance term is null we obtain a generalized additive model, which becomes a generalized linear model if the predictors enter the mean linearly. The model is estimated using Markov chain Monte Carlo simulation and the methodology is illustrated using real and simulated data sets.

stat.ME↗