From Gradient Descent to Harmonic Interpolation: A Geometric Theory of Binary Classification

Abstract

We propose a dictionary between binary classification in machine learning and differential geometry. Classifiers are parallel sections of vector bundles over the data space; training labels become Dirichlet boundary conditions; the kernel of an RKHS interpolant is the Green's function of the Laplace-Beltrami operator; and backpropagation is the degenerate flat-geometry limit of an exact geometric problem. The central contribution is the identification that harmonic interpolation - find the minimum-Dirichlet-energy classifier satisfying the Laplace equation away from the data with prescribed values at training points - is precisely what RKHS interpolation already solves. The kernel is the Green's function, the coefficients are electrostatic capacitances, and the decision boundary is the zero equipotential of the resulting potential field. This reframes results of Kimeldorf-Wahba (1971) and Lindgren-Rue-Lindstrom (2011) as classical potential theory on a Riemannian manifold. For finite data on any smooth manifold, flat O(2) solutions always exist. The density of O(2) harmonic interpolants in the space of continuous classifiers is universal kernel theory (Steinwart; Micchelli-Xu-Zhang) in geometric language. The two arbitrary choices of classical ML - activation function and kernel - are identified as two independent geometric choices: the structure group G (fiber geometry, governing expressivity) and the Riemannian metric g (base geometry, determining the kernel). Code is available on GitHub.