Linear equivalence of nonlinear recurrent neural networks

Abstract

Large nonlinear recurrent neural networks with random couplings generate rich, potentially chaotic activity and are of interest in neuroscience and other fields. A key object encoding the structure of activity is the $N \times N$ covariance matrix. Recent work proposed an ansatz in which, at large $N$ and for typical quenched couplings, this covariance matrix matches that of a linear network with the same couplings, driven by independent noise. We derive this ansatz using a two-site cavity method that gives access to the joint statistics of activities at a pair of sites without disorder averaging. Specifically, we decompose each unit's activity into a linear response to its local field and a nonlinear residual; using the cavity method, we show that cross covariances of residuals at distinct sites are strongly suppressed, so that the residuals act as independent noise driving a linear network. In an alternative derivation, we construct a self-consistent equation for the covariance matrix in which non-Gaussian contributions supply cross terms that, in a linear network, would correspond to an external drive. Higher-order cross-site moments admit a Wick decomposition into pairwise covariances at leading order, reducing them to the linear-equivalent ansatz. We confirm the results in simulations and discuss their neuroscience implications.