The Dynamics of RMSProp Beyond the Stability Threshold

Abstract

We study what happens to RMSProp when the learning rate is too large for convergence to the minimizer. For positive-definite quadratic loss functions, we show that the minimizer is Lyapunov un- stable above an explicit critical learning rate. Nevertheless, every trajectory eventually enters a compact trapping region. We obtain explicit bounds proportional to h in the spatial variables and to h2 in the memory variables, where h is the learning rate. We also prove that nontrivial period-two orbits exist precisely above the critical learning rate, and under additional spectral assumptions, an attracting branch of these orbits emerges from the minimizer. For sufficiently large learning rates, we construct branches near admissible corners of a cube. We then explore the dynamics inside the trapping region using the parameterization method and numerical continuation, exploring invariant circles and tori, coexistence of attracting sets, circle-doubling sequences, and chaotic motions. We examine how the dynamics depend on the loss Hessian, memory decay, and dimension. Finally, we test a learning-rate reduction schedule and continue selected invariant circles into an autonomous variant of the Adam optimizer.