A Bifurcation Theory for the Equilibria of Modern Hopfield Networks

Abstract

Modern Hopfield networks provide a unifying framework for associative memory, transformer attention, diffusion-based generative models, and biological attractor dynamics, linking these systems through a common energy-based dynamics in which states are updated toward weighted combinations of stored patterns. Across these settings, network dynamics is determined by the organization of the energy landscape and the bifurcations of its fixed points. Despite their central role, a general theory of these bifurcations has remained unavailable beyond specific architectures and idealized pattern ensembles. Here we derive stability and bifurcation criteria for the fixed points of general convex-dual Modern Hopfield networks for a general statistics of the stored patterns. Applying this framework to random, block-correlated, and infinitely hierarchical pattern ensembles, we show how memory correlations systematically organize the emergence of hierarchical attractors through successive bifurcations. We further demonstrate that these predictions quantitatively describe retrieval bifurcations in MHNs storing patterns sampled from MNIST, and recapitulate the hierarchical organization of hematopoietic cell identities. Our results establish a general bifurcation theory for Modern Hopfield networks and identify the organization of fixed points as a unifying principle underlying their computational and biological behavior.