arXiv subjects

Babak Hassibi

Publications and source records attributed to Babak Hassibi.

At least 19 recordsLinked to original sources

arXiv2026

Deriving Approximate Message Passing from the Convex Gaussian Min-Max Theorem

Approximate message passing (AMP) provides fast iterative algorithms for high-dimensional signal recovery with Gaussian design matrices, while the Convex Gaussian Min-max Theorem (CGMT) gives a static optimization framework for obtaining sharp asymptotic characterizations of convex estimators. Although these two frameworks often lead to the same scalar state-evolution equations, their connection is usually indirect. In this paper, we establish a direct connection between the two for regularized linear regression in the proportional high-dimensional regime. When the CGMT Auxiliary Optimization (AO) and Primary Optimization (PO) give the same primal-dual solution, we show that the CGMT framework recovers the AMP fixed-point equations, including the Onsager correction. We further identify the AO Gaussian vectors with the Gaussian perturbations in the primal and residual AMP channels. For regularized M-estimation, the same viewpoint recovers the fixed point of scalar-variance max-sum Generalized AMP (GAMP). These results show that the AMP (and GAMP) iterations are suggested, and can be derived, from the CGMT framework, and may further suggest a way to derive AMP-like algorithms in settings where CGMT applies but standard AMP derivations are unavailable.

cs.IT↗

arXiv2026

On Codes with Support-Constrained Parity Checks

We study linear codes that maximize minimum distance subject to arbitrary support constraints on the parity-check matrix. Such constraints arise naturally in the design of LDPC codes, locally repairable codes, and hardware-constrained systems where each parity check must involve only a limited number of code symbols. They are also essential in quantum error correction, where sparse stabilizers reduce measurement noise and respect the connectivity constraints of physical qubit architectures. We derive the optimal minimum distance possible given support constraints on the parity-check matrix and show it is achievable over sufficiently large fields. When this maximum distance coincides with the Singleton bound for unconstrained parity check matrices, the dual GM-MDS construction yields generalized Reed--Solomon codes obeying the mask. In the generator-matrix setting, the GM-MDS theorem guarantees that the optimal distance can always be achieved by a subcode of a generalized Reed--Solomon code while satisfying arbitrary support constraints. We show that this is not true for the parity-check setting. We exhibit a set of support constraints, derived from the vertex-edge incidence of $K_{6,6}$, for which the optimal minimum distance cannot be realized by any subcode of a generalized Reed--Solomon code over any field. We also analyze structured constraint families -- regular, balanced, and cyclic masks -- through numerical optimization, providing design guidance for practical code constructions.

cs.IT↗

arXiv2026

Distributionally Robust K-Means Clustering

K-means clustering is a workhorse of unsupervised learning, but it is notoriously brittle to outliers, distribution shifts, and limited sample sizes. Viewing k-means as Lloyd--Max quantization of the empirical distribution, we develop a distributionally robust variant that protects against such pathologies. We posit that the unknown population distribution lies within a Wasserstein-2 ball around the empirical distribution. In this setting, one seeks cluster centers that minimize the worst-case expected squared distance over this ambiguity set, leading to a minimax formulation. A tractable dual yields a soft-clustering scheme that replaces hard assignments with smoothly weighted ones. We propose an efficient block coordinate descent algorithm with provable monotonic decrease and local linear convergence. Experiments on standard benchmarks and large-scale synthetic data demonstrate substantial gains in outlier detection and robustness to noise.

cs.LG↗

arXiv2026

Quantized Online LQR

We study online linear-quadratic regulation (LQR) with unknown dynamics under communication rate constraints. Classical networked control quantizes the plant state at every time step, requiring $O(T)$ total bits while injecting persistent quantization noise that limits control performance. We consider a setting where the plant observes its state locally and can estimate system dynamics via ordinary least squares, while a remote controller possesses knowledge of the control cost. Rather than quantizing the raw state, the plant transmits learned dynamics estimates over a rate-limited uplink, and the controller returns the optimal control policy so that the plant can compute actions locally using its superior state knowledge. We first prove a fundamental information-theoretic lower bound: any scheme achieving $O(T^\alpha)$ regret for $\alpha \in [1/2,1)$ compared to the optimal infinite horizon LQR controller that knows the true system dynamics must transmit at least $\Omega(\log T)$ bits. We then design the \textbf{Quantized Certainty Equivalent (QCE-LQR)} algorithm, which matches this bound. The resulting regret bound contains inflation factors $Q_{\mathrm{slow}}(\varrho)$ and $Q_{\mathrm{fast}}(\varrho)$ that vanish as the codebook resolution increases, smoothly recovering the unquantized baseline regret. Numerical experiments on four benchmark systems -- from a scalar unstable plant to a 24-parameter Boeing 747 lateral model -- confirm that a variant of QCE-LQR achieves regret comparable to an unquantized certainty equivalent controller over a horizon of $T=10{,}000$ steps.

eess.SY↗

arXiv2026

Precise Performance of Linear Denoisers in the Proportional Regime

In the present paper we study the performance of linear denoisers for noisy data of the form $\mathbf{x} + \mathbf{z}$, where $\mathbf{x} \in \mathbb{R}^d$ is the desired data with zero mean and unknown covariance $\mathbf{\Sigma}$, and $\mathbf{z} \sim \mathcal{N}(0, \mathbf{\Sigma}_{\mathbf{z}})$ is additive noise. Since the covariance $\mathbf{\Sigma}$ is not known, the standard Wiener filter cannot be employed for denoising. Instead we assume we are given samples $\mathbf{x}_1,\dots,\mathbf{x}_n \in \mathbb{R}^d$ from the true distribution. A standard approach would then be to estimate $\mathbf{\Sigma}$ from the samples and use it to construct an ``empirical" Wiener filter. However, in this paper, motivated by the denoising step in diffusion models, we take a different approach whereby we train a linear denoiser $\mathbf{W}$ from the data itself. In particular, we synthetically construct noisy samples $\hat{\mathbf{x}}_i$ of the data by injecting the samples with Gaussian noise with covariance $\mathbf{\Sigma}_1 \neq \mathbf{\Sigma}_{\mathbf{z}}$ and find the best $\mathbf{W}$ that approximates $\mathbf{W}\hat{\mathbf{x}}_i \approx \mathbf{x}_i$ in a least-squares sense. In the proportional regime $\frac{n}{d} \rightarrow \kappa > 1$ we use the {\it Convex Gaussian Min-Max Theorem (CGMT)} to analytically find the closed form expression for the generalization error of the denoiser obtained from this process. Using this expression one can optimize over $\mathbf{\Sigma}_1$ to find the best possible denoiser. Our numerical simulations show that our denoiser outperforms the ``empirical" Wiener filter in many scenarios and approaches the optimal Wiener filter as $\kappa\rightarrow\infty$.

stat.ML↗

arXiv2026

Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime

In this work, we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent and Gradient Clipping. We consider preconditioners of the form $\nabla K$, where $K: \mathbb{R}^{d \times k} \to \mathbb{R}$ is convex and apply $\nabla K(\cdot)$ to train an over-parameterized linear model with a convex loss of the form $\ell(X W - Y)$, for weights $W \in \mathbb{R}^{d \times k}$, labels $Y \in \mathbb{R}^{n \times k}$ and data $X \in \mathbb{R}^{n \times d}$. Under the aforementioned assumptions, we prove that the iterates of the full-batch preconditioned gradient descent converge at an exponential rate to a point $W_{\infty} \in \mathbb{R}^{d \times k}$ satisfying $XW_{\infty} = Y$. We also study the implicit bias of Dual Space Preconditioned Gradient Descent. First, we demonstrate analytically and empirically that, for general $K(\cdot)$, $W_\infty$ depends on the chosen constant step size, hindering a precise characterization of the implicit bias. We also provide an approximate implicit bias property for general preconditioners, namely, $\|W_0 - W_{\infty}\|_F \le c \|W_0 - W_{\text{GD}, \infty}\|_F$ for a constant $c>0$ and $W_{\text{GD}, \infty}$ denoting the convergence point of GD initialized at $W_0$. Furthermore, for preconditioners of the form $K(G) = h(\|G\|_F)$, known as {\it isotropic preconditioners}, and for the stochastic variation of the algorithm with arbitrary batch-size, we prove linear convergence to $W_{\text{GD}, \infty}$. Finally, in the experiments, we demonstrate faster convergence on a nonlinear model obtained using the smoothed matrix elastic-net as a preconditioner.

stat.ML↗

arXiv2026

Implicit Bias and Convergence of Matrix Stochastic Mirror Descent

We investigate Stochastic Mirror Descent (SMD) with matrix parameters and vector-valued predictions, a framework relevant to multi-class classification and matrix completion problems. Focusing on the overparameterized regime, where the total number of parameters exceeds the number of training samples, we prove that SMD with matrix mirror functions $\psi(\cdot)$ converges exponentially to a global interpolator. Furthermore, we generalize classical implicit bias results of vector SMD by demonstrating that the matrix SMD algorithm converges to the unique solution minimizing the Bregman divergence induced by $\psi(\cdot)$ from initialization subject to interpolating the data. These findings reveal how matrix mirror maps dictate inductive bias in high-dimensional, multi-output problems.

stat.ML↗

arXiv2025

One-Bit Quantization for Random Features Models

Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.

cs.LG↗

arXiv2025

Learn to Change the World: Multi-level Reinforcement Learning with Model-Changing Actions

Reinforcement learning usually assumes a given or sometimes even fixed environment in which an agent seeks an optimal policy to maximize its long-term discounted reward. In contrast, we consider agents that are not limited to passive adaptations: they instead have model-changing actions that actively modify the RL model of world dynamics itself. Reconfiguring the underlying transition processes can potentially increase the agents' rewards. Motivated by this setting, we introduce the multi-layer configurable time-varying Markov decision process (MCTVMDP). In an MCTVMDP, the lower-level MDP has a non-stationary transition function that is configurable through upper-level model-changing actions. The agent's objective consists of two parts: Optimize the configuration policies in the upper-level MDP and optimize the primitive action policies in the lower-level MDP to jointly improve its expected long-term reward.

cs.LG↗

arXiv2025

A Precise Performance Analysis of the Randomized Singular Value Decomposition

The Randomized Singular Value Decomposition (RSVD) is a widely used algorithm for efficiently computing low-rank approximations of large matrices, without the need to construct a full-blown SVD. Of interest, of course, is the approximation error of RSVD compared to the optimal low-rank approximation error obtained from the SVD. While the literature provides various upper and lower error bounds for RSVD, in this paper we derive precise asymptotic expressions that characterize its approximation error as the matrix dimensions grow to infinity. Our expressions depend only on the singular values of the matrix, and we evaluate them for two important matrix ensembles: those with power law and bilevel singular value distributions. Our results aim to quantify the gap between the existing theoretical bounds and the actual performance of RSVD. Furthermore, we extend our analysis to polynomial-filtered RSVD, deriving performance characterizations that provide insights into optimal filter selection.

math.NA↗

arXiv2025

Optimal Implicit Bias in Linear Regression

Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, ``What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter.

cs.LG↗

arXiv2025

Beyond Quadratic Costs in LQR: Bregman Divergence Control

In the past couple of decades, the use of ``non-quadratic" convex cost functions has revolutionized signal processing, machine learning, and statistics, allowing one to customize solutions to have desired structures and properties. However, the situation is not the same in control where the use of quadratic costs still dominates, ostensibly because determining the ``value function", i.e., the optimal expected cost-to-go, which is critical to the construction of the optimal controller, becomes computationally intractable as soon as one considers general convex costs. As a result, practitioners often resort to heuristics and approximations, such as model predictive control that only looks a few steps into the future. In the quadratic case, the value function is easily determined by solving Riccati equations. In this work, we consider a special class of convex cost functions constructed from Bregman divergence and show how, with appropriate choices, they can be used to fully extend the framework developed for the quadratic case. The resulting optimal controllers are infinite horizon, come with stability guarantees, and have state-feedback, or estimated state-feedback, laws. They exhibit a much wider range of behavior than their quadratic counterparts since the feedback laws are nonlinear. The approach can be applied to several cases of interest, including safety control, sparse control, and bang-bang control.

eess.SY↗

arXiv2025

Beyond Quadratic Costs: A Bregman Divergence Approach to H$_\infty$ Control

In the past couple of decades, non-quadratic convex penalties have reshaped signal processing and machine learning; in robust control, however, general convex costs break the Riccati and storage function structure that make the design tractable. Practitioners thus default to approximations, heuristics or robust model predictive control that are solved online for short horizons. We close this gap by extending $H_\infty$ control of discrete-time linear systems to strictly convex penalties on state, input, and disturbance, recasting the objective with Bregman divergences that admit a completion-of-squares decomposition. The result is a closed-form, time-invariant, full-information stabilizing controller that minimizes a worst-case performance ratio over the infinite horizon. Necessary and sufficient existence/optimality conditions are given by a Riccati-like identity together with a concavity requirement; with quadratic costs, these collapse to the classical $H_\infty$ algebraic Riccati equation and the associated negative-semidefinite condition, recovering the linear central controller. Otherwise, the optimal controller is nonlinear and can enable safety envelopes, sparse actuation, and bang-bang policies with rigorous $H_\infty$ guarantees.

eess.SY↗

arXiv2025

Robust Mean Estimation With Auxiliary Samples

In data-driven learning and inference tasks, the high cost of acquiring samples from the target distribution often limits performance. A common strategy to mitigate this challenge is to augment the limited target samples with data from a more accessible "auxiliary" distribution. This paper establishes fundamental limits of this approach by analyzing the improvement in the mean square error (MSE) when estimating the mean of the target distribution. Using the Wasserstein-2 metric to quantify the distance between distributions, we derive expressions for the worst-case MSE when samples are drawn (with labels) from both a target distribution and an auxiliary distribution within a specified Wasserstein-2 distance from the target distribution. We explicitly characterize the achievable MSE and the optimal estimator in terms of the problem dimension, the number of samples from the target and auxiliary distributions, the Wasserstein-2 distance, and the covariance of the target distribution. We note that utilizing samples from the auxiliary distribution effectively improves the MSE when the squared radius of the Wasserstein-2 uncertainty ball is small compared to the variance of the true distribution and the number of samples from the true distribution is limited. Numerical simulations in the Gaussian location model illustrate the theoretical findings.

math.ST↗

arXiv2025

Gaussian Universality for Diffusion Models

We investigate Gaussian Universality for data distributions generated via diffusion models. By Gaussian Universality we mean that the test error of a generalized linear model $f(\mathbf{W})$ trained for a classification task on the diffusion data matches the test error of $f(\mathbf{W})$ trained on the Gaussian Mixture with matching means and covariances per class.In other words, the test error depends only on the first and second order statistics of the diffusion-generated data in the linear setting. As a corollary, the analysis of the test error for linear classifiers can be reduced to Gaussian data from diffusion-generated data. Analysing the performance of models trained on synthetic data is a pertinent problem due to the surge of methods such as \cite{sehwag2024stretchingdollardiffusiontraining}. Moreover, we show that, for any $1$- Lipschitz scalar function $\phi$, $\phi(\mathbf{x})$ is close to $\mathbb{E} \phi(\mathbf{x})$ with high probability for $\mathbf{x}$ sampled from the conditional diffusion model corresponding to each class. Finally, we note that current approaches for proving universality do not apply to diffusion-generated data as the covariance matrices of the data tend to have vanishing minimum singular values, contrary to the assumption made in the literature. This leaves extending previous mathematical universality results as an intriguing open question.

stat.ML↗

arXiv2024

Universality in Transfer Learning for Linear Models

We study the problem of transfer learning and fine-tuning in linear models for both regression and binary classification. In particular, we consider the use of stochastic gradient descent (SGD) on a linear model initialized with pretrained weights and using a small training data set from the target distribution. In the asymptotic regime of large models, we provide an exact and rigorous analysis and relate the generalization errors (in regression) and classification errors (in binary classification) for the pretrained and fine-tuned models. In particular, we give conditions under which the fine-tuned model outperforms the pretrained one. An important aspect of our work is that all the results are "universal", in the sense that they depend only on the first and second order statistics of the target distribution. They thus extend well beyond the standard Gaussian assumptions commonly made in the literature. Furthermore, our universality results extend beyond standard SGD training to the test error of a classification task trained using a ridge regression.

cs.LG↗

arXiv2024

Optimal Infinite-Horizon Mixed $\mathit{H}_2/\mathit{H}_\infty$ Control

We study the problem of mixed $\mathit{H}_2/\mathit{H}_\infty$ control in the infinite-horizon setting. We identify the optimal causal controller that minimizes the $\mathit{H}_2$ cost of the closed-loop system subject to an $\mathit{H}_\infty$ constraint. Megretski proved that the optimal mixed $\mathit{H}_2/\mathit{H}_\infty$ controller is non-rational whenever the constraint is active without giving an explicit construction of the controller. In this work, we provide the first exact closed-form solution to the infinite-horizon mixed $\mathit{H}_2/\mathit{H}_\infty$ control in the frequency domain. While the optimal controller is non-rational, our formulation provides a finite-dimensional parameterization of the optimal controller. Leveraging this fact, we introduce an efficient iterative algorithm that finds the optimal causal controller in the frequency domain. We show that this algorithm is convergent when the system is scalar and present numerical evidence for exponential convergence of the proposed algorithm. Finally, we show how to find the best (in $\mathit{H}_\infty$ norm) fixed-order rational approximations of the optimal mixed $\mathit{H}_2/\mathit{H}_\infty$ controller and study its performance.

math.OC↗

arXiv2024

The Distributionally Robust Infinite-Horizon LQR

We explore the infinite-horizon Distributionally Robust (DR) linear-quadratic control. While the probability distribution of disturbances is unknown and potentially correlated over time, it is confined within a Wasserstein-2 ball of a radius $r$ around a known nominal distribution. Our goal is to devise a control policy that minimizes the worst-case expected Linear-Quadratic Regulator (LQR) cost among all probability distributions of disturbances lying in the Wasserstein ambiguity set. We obtain the optimality conditions for the optimal DR controller and show that it is non-rational. Despite lacking a finite-order state-space representation, we introduce a computationally tractable fixed-point iteration algorithm. Our proposed method computes the optimal controller in the frequency domain to any desired fidelity. Moreover, for any given finite order, we use a convex numerical method to compute the best rational approximation (in $H_\infty$-norm) to the optimal non-rational DR controller. This enables efficient time-domain implementation by finite-order state-space controllers and addresses the computational hurdles associated with the finite-horizon approaches to DR-LQR problems, which typically necessitate solving a Semi-Definite Program (SDP) with a dimension scaling with the time horizon. We provide numerical simulations to showcase the effectiveness of our approach.

math.OC↗