Beliefs Beyond Posteriors: Local-Consistency Optimisation for Bayesian Neural Networks

Abstract

The standard training objectives of Bayesian deep learning are posterior-seeking: their optimum over the belief is the posterior of a fitted model, or its KL projection. We show that the shared target is a removable constraint on the belief-not an ideal that training can only approximate: posterior-seeking objectives define a binding map from model parameters to beliefs. From the local-normaliser structure of the Bethe/EP functional we derive a shared-cavity objective that carries this binding as an optional constraint, its per-observation data term a strictly proper predictive score for any likelihood. Our proposal is to drop the constraint. Free routing trains the belief as an optimisation variable of this objective; what trains on the predictive score is still a belief over weights, its prior and noise hyperparameters learned in the same gradient pass. We instantiate this at the Gaussian last layer, where exact inference is available: the freed belief departs from the posterior by a closed-form gap-the residual heteroscedasticity its variance family expresses. The instance, SCROLL, is single-pass, with no last-layer regularisation weight to cross-validate; it steps off the exact corner by a change of estimand (the shared cavity). SCROLL improves on the exact evidence corner at that corner's own learned features on seven of eight UCI datasets, matches or beats validation-tuned references on predictive likelihood and calibration-granted the ensembles' own five-member budget, it leads that tier on likelihood as well-and carries from UCI through frozen embeddings to end-to-end deep learning.