Stochastic Thermodynamics of Associative Memory

Abstract

Dense Associative Memory networks (DenseAMs) unify several popular paradigms in Artificial Intelligence (AI), such as Hopfield Networks, transformers, and diffusion models, while casting their computational properties into the language of dynamical systems and energy landscapes. This formulation provides a natural setting for studying thermodynamics and computation in neural systems, because DenseAMs are simultaneously simple enough to admit analytic treatment and rich enough to implement nontrivial computational function. Aspects of these networks have been studied at equilibrium and at zero temperature, but the thermodynamic costs associated with their operation out of equilibrium are largely unexplored. Here, we define the thermodynamic entropy production associated with the operation of such networks, and study polynomial DenseAMs at intermediate memory load. At large system sizes and intermediate and low load, we use dynamical mean field theory to characterize out-of-equilibrium properties, work requirements, and memory transition times when driving the system with corrupted memories. We characterize a failure mode of higher order networks not observed at zero temperature. Further, we develop a method for calculating work and power costs in the mean field limit. Finally, we find tradeoffs between entropy production, memory retrieval accuracy, and operation speed.