Backpropagation as a Nilpotent Linear System

Abstract

Backpropagation is the computational engine of deep learning, yet its mathematical structure is typically treated as a procedural traversal of computational graphs. We present a global operator theory of the \emph{F-adjoint} framework, which reformulates the layerwise backward recursion of an $L$-depth feedforward network into a single linear system $(I-\cB)\Xs=\bG$, where $\bG$ is a source vector. We prove that the global backward operator $\cB$ is strictly block upper-triangular and nilpotent of index at most $L$. This nilpotency guarantees the exact termination of the Neumann series solution after at most $L$ terms, revealing classical backpropagation to be mathematically equivalent to block back-substitution on an upper bidiagonal system. We formalise \emph{F-symmetry} -- the condition in which the backward pass perfectly mirrors the forward pass -- identifying orthogonal weight matrices as canonical examples. Through worked numerical examples, we demonstrate how this operator perspective exposes the single-path collapse of strictly feedforward networks and its breakdown in residual architectures. Finally, we leverage this compositional structure to rigorously derive the mechanics of residual networks (gradient highways) and transfer learning (gradient truncation). This framework elevates backpropagation from an algorithmic recipe to a global nilpotent-operator formulation.