arXiv · 2611.19382
Generalization Bounds for Neural Networks with Sparse Connectivity
Abstract
Empirical evidence suggests that large neural networks rarely make effective use of all available parameters, with learned solutions exhibiting pronounced sparsity. Despite this ubiquity, existing generalization theory only partially explains how such sparsity influences statistical performance. In this paper, we derive non-asymptotic generalization gap bounds for deep neural networks whose layer-wise weight matrices are restricted by $\ell_p$ quasi-norm ($0<p\leq 1$) constraints. Our analysis gives rise to a novel result that interpolates between classical norm-based generalization guarantees and parameter-counting bounds. Remarkably, in the extreme sparsity limit as the quasi-norm order $p$ approaches zero, our analysis reveals a transition to a square-root dependence on the number of \emph{active} (non-zero) parameters, resulting in bounds that are strictly tighter than existing parameter-counting approaches. Experiments demonstrate that our bounds exhibit a milder dependence on initial architectural choice at the over-parametrization regime, effectively capturing the generalization benefit of the sparsity arising from training on natural data.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Nong Minh Hieu, Antoine Ledent. 2026-11-14. Generalization Bounds for Neural Networks with Sparse Connectivity. https://searcharxiv.com/abs/2611.19382
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.