arXiv subjects

Philippe Gagnon

Publications and source records attributed to Philippe Gagnon.

16 recordsLinked to original sources

arXiv2025

Exponential Convergence of CAVI for Bayesian PCA

Probabilistic principal component analysis (PCA) and its Bayesian variant (BPCA) are widely used for dimension reduction in machine learning and statistics. The main advantage of probabilistic PCA over the traditional formulation is allowing uncertainty quantification. The parameters of BPCA are typically learned using mean-field variational inference, and in particular, the coordinate ascent variational inference (CAVI) algorithm. So far, the convergence speed of CAVI for BPCA has not been characterized. In our paper, we fill this gap in the literature. Firstly, we prove a precise exponential convergence result in the case where the model uses a single principal component (PC). Interestingly, this result is established through a connection with the classical $\textit{power iteration algorithm}$ and it indicates that traditional PCA is retrieved as points estimates of the BPCA parameters. Secondly, we leverage recent tools to prove exponential convergence of CAVI for the model with any number of PCs, thus leading to a more general result, but one that is of a slightly different flavor. To prove the latter result, we additionally needed to introduce a novel lower bound for the symmetric Kullback--Leibler divergence between two multivariate normal distributions, which, we believe, is of independent interest in information theory.

stat.ML↗

arXiv2025

Simple proof of robustness for Bayesian heavy-tailed linear regression models

In the Bayesian literature, a line of research called resolution of conflict is about the characterization of robustness against outliers of statistical models. The robustness characterization of a model is achieved by establishing the limiting behaviour of the posterior distribution under an asymptotic framework in which the outliers move away from the bulk of the data. The proofs of the robustness characterization results, especially the recent ones for regression models, are technical and not intuitive, limiting the accessibility and preventing the development of theory in that line of research. In this paper, we highlight that the proof complexity is due to the generality of the assumptions on the prior distribution. To address the issue of accessibility, we present a significantly simpler proof for a linear regression model with a specific class of prior distributions, among which we find typically used prior distributions. The class of prior distributions is such that each regression coefficient has a sub-exponential distribution, which allows to exploit a tail bound, contrarily to previous approaches. The proof is intuitive and uses classical results of probability theory. The generality of the assumption on the error distribution is also appealing; essentially, it can be any distribution with regularly varying or log-regularly varying tails. So far, there does not exist a result in such generality for models with regularly varying distributions. We also investigate the necessity of the assumptions. To promote the development of theory in resolution of conflict, we highlight how the key steps of the proof can be adapted for other models and present an application of the proof technique in the context of generalized linear models.

math.ST↗

arXiv2024

Reconciling Bayesian and frequentist approaches to robustness against outliers

Heavy-tailed models are used as a way to gain robustness against outliers in Bayesian analyses. In frequentist analyses, M-estimators are often employed. In this paper, the two approaches are tentatively reconciled by considering M-estimators as maximum likelihood estimators of heavy-tailed models. From this perspective, it is realized that a fundamental difference exists as frequentists, contrarily to Bayesians, do not require these heavy-tailed models to be proper. For instance, a popular robust estimator in linear regression, Tukey's biweight M-estimator, does not correspond to a proper heavy-tailed model. Thus, a Bayesian practitioner does not have access to the same range of tools as a frequentist practitioner. It is shown through two real-data linear regression analyses that the former may in consequence obtain significantly different estimation results than the latter, where the difference is due to a more pronounced influence by the outliers in the former case. It is highlighted that a way to give these practitioners access to the same range of tools is for the Bayesian to adopt the generalized Bayesian framework of Bissiri et al. (2016) which allows the use of improper models, in combination with proper prior distributions yielding proper generalized posterior distributions. A complete reconciliation of the Bayesian and frequentist approaches to robustness is then achieved. An extensive theoretical study of the generalized Bayesian counterpart of Tukey's biweight M-estimator is provided, which includes a robustness characterization result, a strong consistency result and a Bernstein--von Mises result, the latter allowing to calibrate the generalized posterior distribution for meaningful uncertainty quantification. After adopting the generalized Bayesian framework, the Bayesian practitioner obtains similar results as the frequentist practitioner in the aforementioned examples.

stat.ME↗

arXiv2024

Theoretical guarantees for lifted samplers

Lifted samplers form a class of Markov chain Monte Carlo methods which has drawn a lot attention in recent years due to superior performance in challenging Bayesian applications. A canonical example of lifted samplers is the one that is derived from a random walk Metropolis algorithm for a totally-ordered state space such as the integers or the real numbers. The lifted sampler is derived by splitting into two the proposal distribution: one part in the increasing direction, and the other part in the decreasing direction. It keeps following a direction, until a rejection occurs, upon which it flips the direction. In terms of asymptotic variances, it outperforms the random walk Metropolis algorithm, regardless of the target distribution, at no additional computational cost. Other studies show, however, that beyond this simple case, lifted samplers do not always outperform their Metropolis counterparts. In this paper, we leverage the celebrated work of Tierney (1998) to provide an analysis in a general framework encompassing a broad class of lifted samplers. Our finding is that, essentially, the asymptotic variances cannot increase by a factor of more than 2, regardless of the target distribution, the way the directions are induced, and the type of algorithm from which the lifted sampler is derived (be it a Metropolis--Hastings algorithm, a reversible jump algorithm, etc.). This result indicates that, while there is potentially a lot to gain from lifting a sampler, there is not much to lose.

stat.CO↗

arXiv2024

An asymptotic Peskun ordering and its application to lifted samplers

A Peskun ordering between two samplers, implying a dominance of one over the other, is known among the Markov chain Monte Carlo community for being a remarkably strong result. It is however also known for being a result that is notably difficult to establish. Indeed, one has to prove that the probability to reach a state $\mathbf{y}$ from a state $\mathbf{x}$, using a sampler, is greater than or equal to the probability using the other sampler, and this must hold for all pairs $(\mathbf{x}, \mathbf{y})$ such that $\mathbf{x} \neq \mathbf{y}$. We provide in this paper a weaker version that does not require an inequality between the probabilities for all these states: essentially, the dominance holds asymptotically, as a varying parameter grows without bound, as long as the states for which the probabilities are greater than or equal to belong to a mass-concentrating set. The weak ordering turns out to be useful to compare lifted samplers for partially-ordered discrete state-spaces with their Metropolis--Hastings counterparts. An analysis in great generality yields a qualitative conclusion: they asymptotically perform better in certain situations (and we are able to identify them), but not necessarily in others (and the reasons why are made clear). A quantitative study in a specific context of graphical-model simulation is also conducted.

stat.CO↗

arXiv2024

Robust heavy-tailed versions of generalized linear models with applications in actuarial science

Generalized linear models (GLMs) form one of the most popular classes of models in statistics. The gamma variant is used, for instance, in actuarial science for the modelling of claim amounts in insurance. A flaw of GLMs is that they are not robust against outliers (i.e., against erroneous or extreme data points). A difference in trends in the bulk of the data and the outliers thus yields skewed inference and predictions. To address this problem, robust methods have been introduced. The most commonly applied robust method is frequentist and consists in an estimator which is derived from a modification of the derivative of the log-likelihood. We propose an alternative approach which is modelling-based and thus fundamentally different. It allows for an understanding and interpretation of the modelling, and it can be applied for both frequentist and Bayesian statistical analyses. The approach possesses appealing theoretical and empirical properties.

stat.ME↗

arXiv2023

Improving multiple-try Metropolis with local balancing

Multiple-try Metropolis (MTM) is a popular Markov chain Monte Carlo method with the appealing feature of being amenable to parallel computing. At each iteration, it samples several candidates for the next state of the Markov chain and randomly selects one of them based on a weight function. The canonical weight function is proportional to the target density. We show both theoretically and empirically that this weight function induces pathological behaviours in high dimensions, especially during the convergence phase. We propose to instead use weight functions akin to the locally-balanced proposal distributions of Zanella (2020), thus yielding MTM algorithms that do not exhibit those pathological behaviours. To theoretically analyse these algorithms, we study the high-dimensional performance of ideal schemes that can be thought of as MTM algorithms which sample an infinite number of candidates at each iteration, as well as the discrepancy between such schemes and the MTM algorithms which sample a finite number of candidates. Our analysis unveils a strong distinction between the convergence and stationary phases: in the former, local balancing is crucial and effective to achieve fast convergence, while in the latter, the canonical and novel weight functions yield similar performance. Numerical experiments include an application in precision medicine involving a computationally-expensive forward model, which makes the use of parallel computing within MTM iterations beneficial.

stat.CO↗

arXiv2023

Robustness against conflicting prior information in regression

Including prior information about model parameters is a fundamental step of any Bayesian statistical analysis. It is viewed positively by some as it allows, among others, to quantitatively incorporate expert opinion about model parameters. It is viewed negatively by others because it sets the stage for subjectivity in statistical analysis. Certainly, it creates problems when the inference is skewed due to a conflict with the data collected. According to the theory of conflict resolution (O'Hagan and Pericchi, 2012), a solution to such problems is to diminish the impact of conflicting prior information, yielding inference consistent with the data. This is typically achieved by using heavy-tailed priors. We study both theoretically and numerically the efficacy of such a solution in a regression framework where the prior information about the coefficients takes the form of a product of density functions with known location and scale parameters. We study functions with regularly varying tails (Student distributions), log-regularly-varying tails (as introduced in Desgagné (2015)), and propose functions with slower tail decays that allow to resolve any conflict that can happen under that regression framework, contrarily to the two previous types of functions. The code to reproduce all numerical experiments is available online.

stat.ME↗

arXiv2023

Theoretical properties of Bayesian Student-$t$ linear regression

Bayesian Student-$t$ linear regression is a common robust alternative to the normal model, but its theoretical properties are not well understood. We aim to fill some gaps by providing analyses in two different asymptotic scenarios. The results allow to precisely characterize the trade-off between robustness and efficiency controlled through the degrees of freedom (at least asymptotically).

math.ST↗

arXiv2022

Optimal scaling of random-walk Metropolis algorithms using Bayesian large-sample asymptotics

High-dimensional limit theorems have been shown useful to derive tuning rules for finding the optimal scaling in random-walk Metropolis algorithms. The assumptions under which weak convergence results are proved are however restrictive: the target density is typically assumed to be of a product form. Users may thus doubt the validity of such tuning rules in practical applications. In this paper, we shed some light on optimal-scaling problems from a different perspective, namely a large-sample one. This allows to prove weak convergence results under realistic assumptions and to propose novel parameter-dimension-dependent tuning guidelines. The proposed guidelines are consistent with previous ones when the target density is close to having a product form, and the results highlight that the correlation structure has to be accounted for to avoid performance deterioration if that is not the case, while justifying the use of a natural (asymptotically exact) approximation to the correlation matrix that can be employed for the very first algorithm run.

stat.ME↗

arXiv2021

Informed reversible jump algorithms

Incorporating information about the target distribution in proposal mechanisms generally produces efficient Markov chain Monte Carlo algorithms (or at least, algorithms that are more efficient than uninformed counterparts). For instance, it has proved successful to incorporate gradient information in fixed-dimensional algorithms, as seen with algorithms such as Hamiltonian Monte Carlo. In trans-dimensional algorithms, Green (2003) recommended to sample the parameter proposals during model switches from normal distributions with informative means and covariance matrices. These proposal distributions can be viewed as asymptotic approximations to the parameter distributions, where the limit is with regard to the sample size. Models are typically proposed using uninformed uniform distributions. In this paper, we build on the approach of Zanella (2020) for discrete spaces to incorporate information about neighbouring models. We rely on approximations to posterior model probabilities that are asymptotically exact. We prove that, in some scenarios, the samplers combining this approach with that of Green (2003) behave like ideal ones that use the exact model probabilities and sample from the correct parameter distributions, in the large-sample regime. We show that the implementation of the proposed samplers is straightforward in some cases. The methodology is applied to a real-data example. The code is available online.

stat.CO↗

arXiv2020

Non-reversible jump algorithms for Bayesian nested model selection

Non-reversible Markov chain Monte Carlo methods often outperform their reversible counterparts in terms of asymptotic variance of ergodic averages and mixing properties. Lifting the state-space (Chen et al., 1999; Diaconis et al., 2000) is a generic technique for constructing such samplers. The idea is to think of the random variables we want to generate as position variables and to associate to them direction variables so as to design Markov chains which do not have the diffusive behaviour often exhibited by reversible schemes. In this paper, we explore the benefits of using such ideas in the context of Bayesian model choice for nested models, a class of models for which the model indicator variable is an ordinal random variable. By lifting this model indicator variable, we obtain non-reversible jump algorithms, a non-reversible version of the popular reversible jump algorithms introduced by Green (1995). This simple algorithmic modification provides samplers which can empirically outperform their reversible counterparts at no extra computational cost. The code to reproduce all experiments is available online.

stat.CO↗

arXiv2020

An automatic robust Bayesian approach to principal component regression

Principal component regression uses principal components as regressors. It is particularly useful in prediction settings with high-dimensional covariates. The existing literature treating of Bayesian approaches is relatively sparse. We introduce a Bayesian approach that is robust to outliers in both the dependent variable and the covariates. Outliers can be thought of as observations that are not in line with the general trend. The proposed approach automatically penalises these observations so that their impact on the posterior gradually vanishes as they move further and further away from the general trend, corresponding to a concept in Bayesian statistics called whole robustness. The predictions produced are thus consistent with the bulk of the data. The approach also exploits the geometry of principal components to efficiently identify those that are significant. Individual predictions obtained from the resulting models are consolidated according to model-averaging mechanisms to account for model uncertainty. The approach is evaluated on real data and compared to its nonrobust Bayesian counterpart, the traditional frequentist approach, and a commonly employed robust frequentist method. Detailed guidelines to automate the entire statistical procedure are provided. All required code is made available, see ArXiv:1711.06341.

stat.ME↗

arXiv2019

A New Bayesian Approach to Robustness Against Outliers in Linear Regression

Linear regression is ubiquitous in statistical analysis. It is well understood that conflicting sources of information may contaminate the inference when the classical normality of errors is assumed. The contamination caused by the light normal tails follows from an undesirable effect: the posterior concentrates in an area in between the different sources with a large enough scaling to incorporate them all. The theory of conflict resolution in Bayesian statistics (O'Hagan and Pericchi (2012)) recommends to address this problem by limiting the impact of outliers to obtain conclusions consistent with the bulk of the data. In this paper, we propose a model with super heavy-tailed errors to achieve this. We prove that it is wholly robust, meaning that the impact of outliers gradually vanishes as they move further and further away form the general trend. The super heavy-tailed density is similar to the normal outside of the tails, which gives rise to an efficient estimation procedure. In addition, estimates are easily computed. This is highlighted via a detailed user guide, where all steps are explained through a simulated case study. The performance is shown using simulation. All required code is given.

stat.ME↗

arXiv2019

Weak Convergence and Optimal Tuning of the Reversible Jump Algorithm

The reversible jump algorithm is a useful Markov chain Monte Carlo method introduced by Green (1995) that allows switches between subspaces of differing dimensionality, and therefore, model selection. Although this method is now increasingly used in key areas (e.g. biology and finance), it remains a challenge to implement it. In this paper, we focus on a simple sampling context in order to obtain theoretical results that lead to an optimal tuning procedure for the considered reversible jump algorithm, and consequently, to easy implementation. The key result is the weak convergence of the sequence of stochastic processes engendered by the algorithm. It represents the main contribution of this paper as it is, to our knowledge, the first weak convergence result for the reversible jump algorithm. The sampler updating the parameters according to a random walk, this result allows to retrieve the well-known 0.234 rule for finding the optimal scaling. It also leads to an answer to the question: "with what probability should a parameter update be proposed comparatively to a model switch at each iteration?"

stat.ME↗

arXiv2018

Bayesian Robustness to Outliers in Linear Regression and Ratio Estimation

Whole robustness is a nice property to have for statistical models. It implies that the impact of outliers gradually vanishes as they approach plus or minus infinity. So far, the Bayesian literature provides results that ensure whole robustness for the location-scale model. In this paper, we make two contributions. First, we generalise the results to attain whole robustness in simple linear regression through the origin, which is a necessary step towards results for general linear regression models. We allow the variance of the error term to depend on the explanatory variable. This flexibility leads to the second contribution: we provide a simple Bayesian approach to robustly estimate finite population means and ratios. The strategy to attain whole robustness is simple since it lies in replacing the traditional normal assumption on the error term by a super heavy-tailed distribution assumption. As a result, users can estimate the parameters as usual, using the posterior distribution.

stat.ME↗