Contrastive Bayesian Inference for Unnormalized Models

Abstract

Unnormalized (or energy-based) models provide a flexible framework for capturing the characteristics of data with complex dependency structures. However, the application of standard Bayesian inference methods has been severely limited because the parameter-dependent normalizing constant is either analytically intractable or computationally prohibitive to evaluate. A promising approach is score-based generalized Bayesian inference, which avoids evaluating the normalizing constant by replacing the likelihood with a scoring rule. However, this approach still requires careful tuning of the likelihood information, and it may fail to yield valid inference without appropriate control. To overcome this difficulty, we propose a fully Bayesian framework for inference on unnormalized models that does not require such tuning. We build on noise-contrastive estimation, which recasts inference as a binary classification problem between observed and noise samples, and treat the normalizing constant as an additional unknown parameter within the resulting likelihood. For exponential families, the classification likelihood becomes conditionally Gaussian via P\'olya-Gamma data augmentation, leading to a simple Gibbs sampler. We further establish posterior concentration and a Bernstein-von Mises theorem for our proposed method, providing theoretical justification for its uncertainty quantification. We demonstrate the proposed approach through two models: time-varying density models of temporal point processes and sparse torus graph models of multivariate circular data. Through simulation studies and real-data analyses, our proposed method provides accurate point estimates and principled uncertainty quantification.