Gaussian mixtures and non-parametric likelihoods through the lens of statistical mechanics

Abstract

In this work, we investigate Gaussian Mixture Models ({\it abbrv} GMM) and the related problem of non parametric maximum likelihood estimation ({\it abbrv} NPMLE) from the perspective of statistical mechanics. In particular, we establish stability guarantees for the NPMLE procedure that extend well beyond the state of the art. Crucially, we obtain guarantees on the Kullback-Leibler divergence between NPMLE estimators and the ground truth, a type of result which has been known to be challenging in the literature on this problem. In particular, we provide high probability upper bounds on the KL divergence between the NPMLE and the true density that are of the order of $\min\big\{\frac{(\log n)^{d+2}}{n} , \frac{\log n}{\sqrt n}\big\}$, which cover a wide range of scenarios for the comparative sizes of $n$ and $d$. We obtain similar guarantees for approximate solutions to the NPMLE problem, addressing realistic situations wherein optimization algorithms need to be stopped in finite time, allowing access only to approximations to the true NPMLE. A cornerstone of our approach is an analysis of the function class complexity of logarithms of gaussian mixture densities, which is able to handle their unboundedness, and could be of wider interest. Our methods lead to novel confidence-interval guarantees for entropy estimation in Gaussian mixtures, demonstrating their wider impact. We also establish correspondences between stability phenomena in the NPMLE problem and concepts such as chaos and multiple valleys in random energy landscapes of statistical mechanics models. While these correspondences are largely of a conceptual nature at this point, we believe that these connections, especially those with concentration phenomena and Langevin dynamics, may be developed into a toolbox for studying a wide variety of random optimization problems in statistics and machine learning.