Bayesian Effective Dimension from Information Growth

Abstract

Nominal dimension can greatly overstate how much a Bayesian model actually learns from data. We quantify this gap through a prior-dependent effective dimension based on parameter--data mutual information, normalized so that regular parametric information growth provides the benchmark scale. The main theoretical result shows that, in infinite-dimensional Gaussian experiments, information growth is determined by the spectral counting function: regularly varying spectra yield a sharp asymptotic law with an explicit constant, while exponential spectral decay produces a distinct polylogarithmic regime. In Gaussian regression the same information scale has a spectral representation and is linked exactly to ridge degrees of freedom. For scalar Gaussian scale-mixture priors, we separate information about latent scales from conditional Gaussian information and show that the full marginal information remains finite under polynomially heavy-tailed mixing.