arXiv subjects

Michael J. Feldman

Publications and source records attributed to Michael J. Feldman.

5 recordsLinked to original sources

arXiv2026

Sparse PCA: Phase Transitions in the Critical Regime

This work studies estimation of sparse principal components in high dimensions. Specifically, we consider a class of estimators based on kernel PCA, generalizing the covariance thresholding algorithm proposed by Krauthgamer et al. (2015). Focusing on Johnstone's spiked covariance model, we investigate the "critical" sparsity regime, where the sparsity level $m$, sample size $n$, and dimension $p$ each diverge and $m/\sqrt{n} \rightarrow β$, $p/n \rightarrow γ$. Within this framework, we develop a fine-grained understanding of signal detection and recovery. Our results establish a detectability phase transition, analogous to the Baik--Ben Arous--Péché (BBP) transition: above a certain threshold -- depending on the kernel function, $γ$, and $β$ -- kernel PCA is informative. Conversely, below the threshold, kernel principal components are asymptotically orthogonal to the signal. Notably, above this detection threshold, we find that consistent support recovery is possible with high probability. Sparsity plays a key role in our analysis, and results in more nuanced phenomena than in related studies of kernel PCA with delocalized (dense) components. Finally, we identify optimal kernel functions for detection -- and consequently, support recovery -- and numerical calculations suggest that soft thresholding is nearly optimal.

math.ST↗

arXiv2026

Robustness of OLS to sample removals: Theoretical analysis and implications

For learned models to be trustworthy, it is essential to verify their robustness to perturbations in the training data. Classical approaches involve uncertainty quantification via confidence intervals and bootstrap methods. In contrast, recent work proposes a more stringent form of robustness: stability to the removal of any subset of $k$ samples from the training set. In this paper, we present a theoretical study of this criterion for ordinary least squares (OLS). Our contributions are as follows: (1) Given $n$ i.i.d. training samples from a general misspecified model, we prove that with high probability, OLS is robust to the removal of any $k \ll n $ samples. (2) For data of dimension $p$, OLS can withstand up to ${k\ll \sqrt{np}/\log n}$ sample removals while remaining robust and achieving the same error rate as OLS applied to the full dataset. Conversely, if $k$ is proportional to $n$, OLS is provably non-robust. (3) We revisit prior analyses that found several econometric datasets to be highly non-robust to sample removals. While this appears to contradict our results in (1), we demonstrate that the sensitivity is due to either heavy-tailed responses or correlated samples. Empirically, this sensitivity is considerably attenuated by classical robust methods, such as linear regression with a Huber loss.

math.ST↗

arXiv2025

Spectral Properties of Elementwise-Transformed Spiked Matrices

This work concerns elementwise-transformations of spiked matrices: $Y_n = n^{-1/2} f( \sqrt{n} X_n + Z_n)$. Here, $f$ is a function applied elementwise, $X_n$ is a low-rank signal matrix, and $Z_n$ is white noise. We find that principal component analysis is powerful for recovering signal under highly nonlinear or discontinuous transformations. Specifically, in the high-dimensional setting where $Y_n$ is of size $n \times p$ with $n,p \rightarrow \infty$ and $p/n \rightarrow γ> 0$, we uncover a phase transition: for signal-to-noise ratios above a sharp threshold -- depending on $f$, the distribution of elements of $Z_n$, and the limiting aspect ratio $γ$ -- the principal components of $Y_n$ (partially) recover those of $X_n$. Below this threshold, the principal components of $Y_n$ are asymptotically orthogonal to the signal. In contrast, in the standard setting where $X_n + n^{-1/2}Z_n$ is observed directly, the analogous phase transition depends only on $γ$. A similar phenomenon occurs with $X_n$ square and symmetric and $Z_n$ a generalized Wigner matrix.

math.ST↗

arXiv2023

Spiked Singular Values and Vectors under Extreme Aspect Ratios

The behavior of the leading singular values and vectors of noisy low-rank matrices is fundamental to many statistical and scientific problems. Theoretical understanding currently derives from asymptotic analysis under one of two regimes: (1) the classical regime, with a fixed number of rows and large number of columns, or vice versa, and (2) the proportional regime, with large numbers of rows and columns, proportional to one another. This paper is concerned with the disproportional regime, where the matrix is either ``tall and narrow'' or ``short and wide'': we study sequences of matrices of size $n \times m_n$ with aspect ratio $ n/m_n \rightarrow 0$ or $n/m_n \rightarrow \infty$ as $n \rightarrow \infty$. This regime has important ``big data'' applications. Theory derived here shows that the displacement of the empirical singular values and vectors from their noise-free counterparts and the associated phase transitions -- well-known under proportional growth asymptotics -- still occur in the disproportionate setting. They must be quantified, however, on a novel scale of measurement that adjusts with the changing aspect ratio as the matrix size increases. In this setting, the top singular vectors corresponding to the longer of the two matrix dimensions are asymptotically uncorrelated with the noise-free signal.

math.ST↗

arXiv2023

Optimal Eigenvalue Shrinkage in the Semicircle Limit

Modern datasets are trending towards ever higher dimension. In response, recent theoretical studies of covariance estimation often assume the proportional-growth asymptotic framework, where the sample size $n$ and dimension $p$ are comparable, with $n, p \rightarrow \infty $ and $γ_n = p/n \rightarrow γ> 0$. Yet, many datasets -- perhaps most -- have very different numbers of rows and columns. We consider instead the disproportional-growth asymptotic framework, where $n, p \rightarrow \infty$ and $γ_n \rightarrow 0$ or $γ_n \rightarrow \infty$. Either disproportional limit induces novel behavior unseen within previous proportional and fixed-$p$ analyses. We study the spiked covariance model, with theoretical covariance a low-rank perturbation of the identity. For each of 15 different loss functions, we exhibit in closed form new optimal shrinkage and thresholding rules. Our optimal procedures demand extensive eigenvalue shrinkage and offer substantial performance benefits over the standard empirical covariance estimator. Practitioners may ask whether to view their data as arising within (and apply the procedures of) the proportional or disproportional frameworks. Conveniently, it is possible to remain {\it framework agnostic}: one unified set of closed-form shrinkage rules (depending only on the aspect ratio $γ_n$ of the given data) offers full asymptotic optimality under either framework. At the heart of the phenomena we explore is the spiked Wigner model, in which a low-rank matrix is perturbed by symmetric noise. Exploiting a connection to the spiked covariance model as $γ_n \rightarrow 0$, we derive optimal eigenvalue shrinkage rules for estimation of the low-rank component, of independent and fundamental interest.

math.ST↗