Weight space structure and analysis using a finite replica number in the Ising perceptron

Abstract

The weight space of the Ising perceptron in which a set of random patterns is stored is examined using the generating function of the partition function $ϕ(n)=(1/N)\log [Z^n]$ as the dimension of the weight vector $N$ tends to infinity, where $Z$ is the partition function and $[ ... ]$ represents the configurational average. We utilize $ϕ(n)$ for two purposes, depending on the value of the ratio $α=M/N$, where $M$ is the number of random patterns. For $α< α_{\rm s}=0.833 ...$, we employ $ϕ(n)$, in conjunction with Parisi's one-step replica symmetry breaking scheme in the limit of $n \to 0$, to evaluate the complexity that characterizes the number of disjoint clusters of weights that are compatible with a given set of random patterns, which indicates that, in typical cases, the weight space is equally dominated by a single large cluster of exponentially many weights and exponentially many small clusters of a single weight. For $α> α_{\rm s}$, on the other hand, $ϕ(n)$ is used to assess the rate function of a small probability that a given set of random patterns is atypically separable by the Ising perceptrons. We show that the analyticity of the rate function changes at $α= α_{\rm GD}=1.245 ... $, which implies that the dominant configuration of the atypically separable patterns exhibits a phase transition at this critical ratio. Extensive numerical experiments are conducted to support the theoretical predictions.