arXiv subjects

Alireza Naderi

Publications and source records attributed to Alireza Naderi.

7 recordsLinked to original sources

arXiv2026

Data-Driven Reachability of Nonlinear Lipschitz Systems via Koopman Operator Embeddings

Data-driven safety verification of robotic systems often relies on zonotopic reachability analysis due to its scalability and computational efficiency. However, for nonlinear systems, these methods can become overly conservative, especially over long prediction horizons and under measurement noise. We propose a data-driven reachability framework based on the Koopman operator and zonotopic set representations that lifts the nonlinear system into a finite-dimensional, linear, state-input-dependent model. Reachable sets are then computed in the lifted space and projected back to the original state space to obtain guaranteed over-approximations of the true dynamics. The proposed method reduces conservatism while preserving formal safety guarantees, and we prove that the resulting reachable sets over-approximate the true reachable sets. Numerical simulations and real-world experiments on an autonomous vehicle show that the proposed approach yields substantially tighter reachable set over-approximations than both model-based and linear data-driven methods, particularly over long horizons.

eess.SY↗

arXiv2025

Online Data-Driven Reachability Analysis using Zonotopic Recursive Least Squares

Reachability analysis is a key formal verification technique for ensuring the safety of modern cyber physical systems subject to uncertainties in measurements, system models (parameters), and inputs. Classical model-based approaches rely on accurate prior knowledge of system dynamics, which may not always be available or reliable. To address this, we present a data-driven reachability analysis framework that computes over-approximations of reachable sets directly from online state measurements. The method estimates time-varying unknown models using an Exponentially Forgetting Zonotopic Recursive Least Squares (EF ZRLS) method, which processes data corrupted by bounded noise. Specifically, a time-varying set of models that contains the true model of the system is estimated recursively, and then used to compute the forward reachable sets under process noise and uncertain inputs. Our approach applies to both discrete-time Linear Time Varying (LTV) and nonlinear Lipschitz systems. Compared to existing techniques, it produces less conservative reachable set over approximations, remains robust under slowly varying dynamics, and operates solely on real-time data without requiring any pre-recorded offline experiments. Numerical simulations and real-world experiments validate the effectiveness and practical applicability of the proposed algorithms.

eess.SY↗

arXiv2025

Mind the Gap: a Spectral Analysis of Rank Collapse and Signal Propagation in Attention Layers

Attention layers are the core component of transformers, the current state-of-the-art neural network architecture. Alternatives to softmax-based attention are being explored due to its tendency to hinder effective information flow. Even at initialisation, it remains poorly understood why the propagation of signals and gradients through these random networks can be pathological, resulting in issues known as (i) vanishing/exploding gradients and (ii) rank collapse $\textit{in depth}$, i.e. when all tokens converge to a single representation along layers. While rank collapse in depth naturally arises from repeated matrix multiplications$\unicode{x2013}$a common pattern across various architectures$\unicode{x2013}$we identify an additional and previously unknown challenge unique to softmax attention layers: (iii) rank collapse $\textit{in width}$, which occurs as the context length increases. Using Random Matrix Theory, we conduct a rigorous analysis that uncovers a spectral gap between the two largest singular values of the attention matrix as the cause of (iii), which in turn exacerbates (i) and (ii). Building on this insight, we propose a novel yet simple practical solution to mitigate rank collapse in width by removing the outlier eigenvalue(s). Our theoretical framework offers a fresh perspective on recent practical studies, such as (Ye et al., 2024; Ali et al., 2023), whose ad hoc solutions can now be interpreted as implicit efforts to address the spectral gap issue. This work provides valuable theoretical support for ongoing large-scale empirical research, bringing theory and practice one step closer in the understanding of transformers.

cs.LG↗

arXiv2025

A simple proof of almost sure convergence for the largest singular value of a product of Gaussian matrices

Let $m \geq 1$ and consider the product of $m$ independent $n \times n$ matrices $\mathbf{W} = \mathbf{W}_1 \dots \mathbf{W}_m$, each $\mathbf{W}_{i}$ with i.i.d. normalised $\mathcal{N}(0, n^{-1/2})$ entries. It is shown in Penson et al. (2011) that the empirical distribution of the squared singular values of $\mathbf{W}$ converges to a deterministic distribution compactly supported on $[0, u_m]$, where $u_m = \frac{{(m+1)}^{m+1}}{m^m}$. This generalises the well-known case of $m=1$, corresponding to the Marchenko-Pastur distribution for square matrices. Moreover, for $m=1$, it was first shown by Geman (1980) that the largest squared singular value almost surely converges to the right endpoint (the so-called ``soft edge'') of the support, i.e. $s_1^2(\mathbf{W}) \xrightarrow{a.s.} u_1$. Herein, we present a proof for the general case $s_1^2(\mathbf{W}) \xrightarrow{a.s.} u_m$ for $m\geq 1$. Although we do not claim novelty for our result, the proof is simple and does not require familiarity with modern techniques of free probability.

math.PR↗

arXiv2024

Beyond IID weights: sparse and low-rank deep Neural Networks are also Gaussian Processes

The infinitely wide neural network has been proven a useful and manageable mathematical model that enables the understanding of many phenomena appearing in deep learning. One example is the convergence of random deep networks to Gaussian processes that allows a rigorous analysis of the way the choice of activation function and network weights impacts the training dynamics. In this paper, we extend the seminal proof of Matthews et al. (2018) to a larger class of initial weight distributions (which we call PSEUDO-IID), including the established cases of IID and orthogonal weights, as well as the emerging low-rank and structured sparse settings celebrated for their computational speed-up benefits. We show that fully-connected and convolutional networks initialized with PSEUDO-IID distributions are all effectively equivalent up to their variance. Using our results, one can identify the Edge-of-Chaos for a broader class of neural networks and tune them at criticality in order to enhance their training. Moreover, they enable the posterior distribution of Bayesian Neural Networks to be tractable across these various initialization schemes.

stat.ML↗

arXiv2021

Beyond Independent Measurements: General Compressed Sensing with GNN Application

We consider the problem of recovering a structured signal $\mathbf{x} \in \mathbb{R}^{n}$ from noisy linear observations $\mathbf{y} =\mathbf{M} \mathbf{x}+\mathbf{w}$. The measurement matrix is modeled as $\mathbf{M} = \mathbf{B}\mathbf{A}$, where $\mathbf{B} \in \mathbb{R}^{l \times m}$ is arbitrary and $\mathbf{A} \in \mathbb{R}^{m \times n}$ has independent sub-gaussian rows. By varying $\mathbf{B}$, and the sub-gaussian distribution of $\mathbf{A}$, this gives a family of measurement matrices which may have heavy tails, dependent rows and columns, and singular values with a large dynamic range. When the structure is given as a possibly non-convex cone $T \subset \mathbb{R}^{n}$, an approximate empirical risk minimizer is proven to be a robust estimator if the effective number of measurements is sufficient, even in the presence of a model mismatch. In classical compressed sensing with independent (sub-)gaussian measurements, one asks how many measurements are needed to recover $\mathbf{x}$? In our setting, however, the effective number of measurements depends on the properties of $\mathbf{B}$. We show that the effective rank of $\mathbf{B}$ may be used as a surrogate for the number of measurements, and if this exceeds the squared Gaussian mean width of $(T-T) \cap \mathbb{S}^{n-1}$, then accurate recovery is guaranteed. Furthermore, we examine the special case of generative priors in detail, that is when $\mathbf{x}$ lies close to $T = \mathrm{ran}(G)$ and $G: \mathbb{R}^k \rightarrow \mathbb{R}^n$ is a Generative Neural Network (GNN) with ReLU activation functions. Our work relies on a recent result in random matrix theory by Jeong, Li, Plan, and Yilmaz arXiv:2001.10631. .

cs.IT↗

arXiv2021

Shortest-support Multi-Spline Bases for Generalized Sampling

Generalized sampling consists in the recovery of a function $f$, from the samples of the responses of a collection of linear shift-invariant systems to the input $f$. The reconstructed function is typically a member of a finitely generated integer-shift-invariant space that can reproduce polynomials up to a given degree $M$. While this property allows for an approximation power of order $(M+1)$, it comes with a tradeoff on the length of the support of the basis functions. Specifically, we prove that the sum of the length of the support of the generators is at least $(M+1)$. Following this result, we introduce the notion of shortest basis of degree $M$, which is motivated by our desire to minimize the computational costs. We then demonstrate that any basis of shortest support generates a Riesz basis. Finally, we introduce a recursive algorithm to construct the shortest-support basis for any multi-spline space. It provides a generalization of both polynomial and Hermite B-splines. This framework paves the way for novel applications such as fast derivative sampling with arbitrarily high approximation power.

math.NA↗