Unified Description of Learning Dynamics in the Soft Committee Machine from Finite to Ultra-Wide Regimes

Abstract

We study the learning dynamics of the soft committee machine (SCM) with Rectified Linear Unit (ReLU) activation using a statistical-mechanics approach within the annealed approximation. The SCM consists of a student network with $N$ input units and $K$ hidden units trained to reproduce the output of a teacher network with $M$ hidden units. We introduce a reduced set of macroscopic order parameters that yields a unified description valid from the conventional regime $K \ll N$ to the ultra-wide limit $K \ge N$. The control parameter $\alpha$, proportional to the ratio of training samples to adjustable weights, serves as an effective measure of dataset size. For small $\gamma = M/N$, we recover a continuous phase transition at $\alpha_{c} \approx 2\pi$ from an unspecialized, permutation-symmetric state to a specialized state in which student units align with the teacher. For finite $\gamma$, the transition disappears and the generalization error decreases smoothly with dataset size, reaching a low plateau when $\gamma=1$. In the asymptotic limit $\alpha \to \infty$, the error scales as $\varepsilon_{g} \propto 1/\alpha$, independent of $\gamma$ and $K$. The results highlight the central role of network dimensions in SCM learning and provide a framework extendable to other activations and quenched analyses.