Efficiency and robustness partition the solution space for memory in recurrent networks

Abstract

In computational neuroscience, task-trained recurrent neural networks (RNNs) are commonly used as a testbed for exploring the space of recurrent circuit solutions compatible with a particular function. However, these networks are subject to inductive biases that may be misaligned with those of biological circuits, which operate under various efficiency and robustness constraints. Here, using a minimal stimulus recall task, we systematically characterize how the solution space of recurrent neural networks is shaped by the desiderata of weight efficiency, activity efficiency, and robustness to noise. We show that weight efficient solutions exhibit a low-rank bias that aligns with the inductive biases of task-trained networks and favors dynamical motifs involving slow, persistent dynamics, whereas activity efficiency encourages high-rank solutions that rely on transient amplification. Moreover, we find that activity efficient solutions exhibit a strong dissociation between high-variance modes of activity and those that causally influence network output. Unlike in static feedforward settings, we demonstrate that noise robustness and activity efficiency are generally in opposition, rather than synergistic. Using these findings, we propose a normative account of misaligned coding phenomena recently identified in premotor cortex.