Approximate Bayesian Fusion

Abstract

Distributed inference plays an important role in modern computational statistics and machine learning. In the frequentist setting, methods can often aggregate results by simply averaging estimators computed on data subsets. However, distributed Bayesian inference is more challenging, as it requires reconstructing the full posterior by combining subposterior distributions computed on different machines. Bayesian Fusion performs distributed Bayesian inference exactly, but at a sizeable computational cost. We propose a method that uses numerical approximations to reduce the cost of the most computationally expensive steps of Bayesian Fusion, and expands its domain of applicability. We bound the numerical error as a function of the step size of the approximation. Under regularity conditions, our approach can approximate the target posterior distribution to an arbitrary degree of accuracy, and is a practical and efficient alternative for large-scale distributed Bayesian inference. In numerical experiments on a logistic regression with more than $300{,}000$ observations, our method reaches the accuracy reported for Divide-and-Conquer Bayesian Fusion about two orders of magnitude faster. On a proportional hazards model with non-Gaussian posteriors, which lacks the lower bounds that Bayesian Fusion requires, our method reproduces funnel features of the posterior that simpler methods like Consensus Monte Carlo miss.