On the asymptotic behaviour of $α$-divergence generalized posteriors

Abstract

We study the asymptotic behaviour of the solutions to the unconstrained generalized variational inference problem regularized by an $α$-divergence. Under mild conditions, the minimizer is explicit up to a scalar normalizing constant, and we show that it admits a rescaling under which it converges in $L^1$ to a fixed non-Gaussian law, at a concentration scale around the empirical loss minimizer that departs from the typical $n^{-1/2}$ in the classical Bernstein--von Mises theorem. For $α\in(0,1)$ and $d<2/(1-α)$ the limit is a Student-$t$ distribution and the posterior width shrinks faster than the sampling error of the empirical minimizer. Conversely, for $α>1$ the limiting distribution is a compactly supported Pearson type-II distribution, and the posterior width shrinks more slowly than the sampling error. The KL divergence ($α=1$) is the unique member of the $α$-divergence family whose posterior width matches the sampling scale and whose limit is Gaussian. We discuss the implications of these qualitatively different regimes for the frequentist coverage of credible regions, and a toy example on intercept-only logistic regression illustrates the theoretical claims.