A Nonlinear Singular Value Theory for Neural Networks
Recently Brown et al. [2025] established a singular value decomposition (SVD) for maps (especially nonlinear) satisfying certain norm conditions. We prove that most modern neural architectures admit this nonlinear SVD (NLSVD) representation---with no change in input--output behavior---and enumerate the classes covered. In this factorization the network is a left-invertible nonlinear map followed by a final linear layer. Moreover, the left-invertible factor is norm-preserving, so distances in the embedding (activations before the final linear layer) calibrate directly to distances in input space. We introduce a flexible architecture that yields an explicit decomposition at training time, a data-driven algorithm for estimating the representation from trained models, and the mathematical foundations for nonlinear analogues of row and null spaces in neural networks. Empirical case studies illustrate uses of the theory for latent-space pullback (visualization and data generation), bias detection, and membership-inference robustness under training. Altogether, these foundations support new approaches to core problems in neural-network analysis.