arXiv · 2611.14855
Splitting and Alternating Minimization Methods for Training Orthogonal Deep Feedforward Networks
Abstract
Training deep feedforward networks with weight matrices constrained to layer-wise Stiefel manifolds serves as an effective regularization technique to improve generalization. However, the simultaneous presence of orthogonality constraints and nonsmooth activation functions poses severe theoretical and computational challenges for standard stochastic gradient and Riemannian optimization methods. In this paper, we propose two splitting and alternating-based algorithms to train orthogonal deep feedforward networks (OrthDFNs). By introducing auxiliary variables, we decouple the nested nonsmooth compositions and manifold constraints into a structured equality-constrained optimization problem. To solve this model, we develop OrthALM, an inexact augmented Lagrangian method (ALM) that employs a proximal alternating minimization scheme to solve its subproblems. Crucially, every block coordinate subproblem admits an exact, closed-form solution. We establish a rigorous convergence theory for OrthALM, proving that every accumulation point of the generated sequence is a Karush--Kuhn--Tucker (KKT) point of the regularized, reformulated problem. Furthermore, we introduce OrthADMM as a practical, proximal-free variant that executes a single inner alternating sweep per multiplier update to accelerate computation. Extensive numerical experiments show that OrthALM attains lower training error and constraint residuals than OrthADMM in most configurations. OrthADMM achieves competitive test accuracy and the lowest reported runtime in most comparisons with the selected baselines.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Jiawei Wang. 2026-11-11. Splitting and Alternating Minimization Methods for Training Orthogonal Deep Feedforward Networks. https://searcharxiv.com/abs/2611.14855
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.