Learning Dynamics Reveal a Hierarchy of Weight-Induced Layerwise Gram Metrics

Abstract

We study feed-forward ReLU networks with fixed readout and quadratic loss, and rewrite gradient descent as a collective dynamics of activation fields and conjugate fields on the training set. Working to first order in the learning rate inside a fixed activation chamber, we derive explicitly the one-, two- and three-hidden-layer cases, and then give the arbitrary-depth recursion. For one hidden layer the activation dynamics closes directly and the residual update is governed by the product of an input Gram matrix and a co-activation/backpropagation Gram matrix. For two hidden layers a conjugate field is required, but no nontrivial pullback Gram metric has yet appeared. For three hidden layers the first weight-induced pullback Gram metric enters the conjugate-field dynamics. At arbitrary depth, activation variations propagate forward through a recursive response operator \(\cU_\ell^{\alpha\beta}\), while conjugate-field variations propagate backward through an effective transport operator \(\cM_\ell^{\alpha\beta}\). Their contractions reconstruct a layerwise residual kernel \[ K_{\alpha\beta}^{(L)}=\sum_{\ell=1}^{L}Q_{\alpha\beta}^{(\ell-1)}S_{\alpha\beta}^{(\ell)}. \] The resulting description exposes a duality between push-forward and pullback transport across every layer cut, and identifies the first Gram metrics as the lowest nontrivial terms in a broader hierarchy of activation-conditioned transport operators. We deliberately stop at the level of collective fields, conjugate fields, residual kernels and cut-wise transport metrics, leaving the later tensorial geometric formulation outside the scope of this paper.