Glassy dynamics near the interpolation transition in deep recurrent networks

Abstract

We examine learning dynamics in deep recurrent networks, focusing on the behavior near the boundary in the depth-width plane separating under- from over-parametrized networks, known as the interpolation transition. The training data are Bach chorales in 4-part harmony, and the learning is by stochastic gradient descent with a cross-entropy loss function. We find critical slowing down of the learning approaching the transition from the overparametrized side: For a given network depth, learning times to reach small training loss values appear to diverge proportional to $1/(w - w_c)$ as the width w approaches a (loss-dependent) critical value $w_c$. We identify the zero-loss limit of this value with the interpolation transition. We also study aging (the slowing down of fluctuations as the time since the beginning of learning increases). Taking a system that has been learning for a time $\tau_w$, we measure the subsequent mean-square fluctuations of the weight values at times $\tau > \tau_w$. In the underparametrized phase, we find that they are well-described by a single function of $\tau/\tau_w$. While this scaling holds approximately at short times at the transition and in the overparametrized phase, it breaks down at longer times when the training loss gets close to the lower limit imposed by the stochastic gradient descent dynamics. Both this kind of aging and the critical slowing down are also found in certain spin glass models, suggesting that those models contain the most essential features of the learning dynamics.