Searcharxiv⌕ Search

arXiv subjects

Yandi Shen

Publications and source records attributed to Yandi Shen.

At least 19 recordsLinked to original sources
arXiv2026

Empirical Bayes linear regression in high dimensions: Method of moments and sub-linear sample complexity

We study empirical Bayes estimation of the prior in high-dimensional linear regression $\mathbf{y}=\mathbf{X}\mathbf{\beta}+\mathbf{\varepsilon}$, where the regression coefficients are drawn independently from an unknown sub-Gaussian prior. In contrast to the sequence model, the design matrix couples the latent coefficients, so that recovering the prior requires deconvolving it from both the noise and copies of itself. We introduce the \emph{Empirical Bayes Method of Moments} (EBMoM), a computationally efficient procedure for general designs that recursively estimates the prior moments through a lower-triangular system of estimating equations and runs in time $O(np^2)$. Under mild design conditions, satisfied in particular by a broad class of correlated random designs, we show that EBMoM consistently estimates a growing number of moments and hence the prior itself, provided that $n\geq p^{1-o(1)}$. A matching information-theoretic lower bound, valid for a broad class of designs, shows that this sub-linear sample complexity is optimal for nonparametric prior estimation. This improves on existing results for likelihood-based methods whose consistency requires a linear sample size $n=\Omega(p)$.

math.ST↗
arXiv2026

Approximate Message Passing with Random Initialization for Phase Retrieval

We analyze approximate message passing (AMP) with an independent Gaussian initialization for noiseless phase retrieval in the proportional asymptotic regime. A random initialization has overlap of order $d^{-1/2}$ with the signal, and AMP requires a growing number of iterations to attain non-vanishing overlap. Thus, its precise behavior cannot be characterized by classical fixed-time state evolution. We prove a Gaussian decomposition of the AMP trajectory and control its error over the horizons required for recovery. The resulting analysis shows that random initialization attains the weak-recovery threshold $\delta_{\rm weak}=1/2$. For $\delta\in(\delta_{\rm weak},\delta_{\rm str})$, where $\delta_{\rm str}\approx1.13$, the signal strength follows state evolution and approaches its stable finite fixed point uniformly for \(n^{1/3}/\operatorname{polylog}(n)\) iterations. For $\delta>\delta_{\rm str}$, AMP reaches any prescribed fixed recovery accuracy within $O_{\delta,\varepsilon}(\log n)$ iterations. The majority of our analysis applies more generally to generalized AMP for single-index models.

math.ST↗
arXiv2025

Learning single index model with gradient descent: spectral initialization and precise asymptotics

Non-convex optimization plays a central role in many statistics and machine learning problems. Despite the landscape irregularities for general non-convex functions, some recent work showed that for many learning problems with random data and large enough sample size, there exists a region around the true signal with benign landscape. Motivated by this observation, a widely used strategy is a two-stage algorithm, where we first apply a spectral initialization to plunge into the region, and then run gradient descent for further refinement. While this two-stage algorithm has been extensively analyzed for many non-convex problems, the precise distributional property of both its transient and long-time behavior remains to be understood. In this work, we study this two-stage algorithm in the context of single index models under the proportional asymptotics regime. We derive a set of dynamical mean field equations, which describe the precise behavior of the trajectory of spectral initialized gradient descent in the large system limit. We further show that when the spectral initialization successfully lands in a region of benign landscape, the above equation system is asymptotically time translation invariant and exponential converging, and thus admits a set of long-time fixed points that represents the mean field characterization of the limiting point of the gradient descent dynamic. As a proof of concept, we demonstrate our general theory in the example of regularized Wirtinger flow for phase retrieval.

math.ST↗
arXiv2025

Besting Good--Turing: Optimality of Non-Parametric Maximum Likelihood for Distribution Estimation

When faced with a small sample from a large universe of possible outcomes, scientists often turn to the venerable Good--Turing estimator. Despite its pedigree, however, this estimator comes with considerable drawbacks, such as the need to hand-tune smoothing parameters and the lack of a precise optimality guarantee. We introduce a parameter-free estimator that bests Good--Turing in both theory and practice. Our method marries two classic ideas, namely Robbins's empirical Bayes and Kiefer--Wolfowitz non-parametric maximum likelihood estimation (NPMLE), to learn an implicit prior from data and then convert it into probability estimates. We prove that the resulting estimator attains the optimal instance-wise risk up to logarithmic factors in the competitive framework of Orlitsky and Suresh, and that the Good--Turing estimator is strictly suboptimal in the same framework. Our simulations on synthetic data and experiments with English corpora and U.S. Census data show that our estimator consistently outperforms both the Good--Turing estimator and explicit Bayes procedures.

math.ST↗
arXiv2025

Dynamical mean-field analysis of adaptive Langevin diffusions: Propagation-of-chaos and convergence of the linear response

Motivated by an application to empirical Bayes learning in high-dimensional regression, we study a class of Langevin diffusions in a system with random disorder, where the drift coefficient is driven by a parameter that continuously adapts to the empirical distribution of the realized process up to the current time. The resulting dynamics take the form of a stochastic interacting particle system having both a McKean-Vlasov type interaction and a pairwise interaction defined by the random disorder. We prove a propagation-of-chaos result, showing that in the large system limit over dimension-independent time horizons, the empirical distribution of sample paths of the Langevin process converges to a deterministic limit law that is described by dynamical mean-field theory. This law is characterized by a system of dynamical fixed-point equations for the limit of the drift parameter and for the correlation and response kernels of the limiting dynamics. Using a dynamical cavity argument, we verify that these correlation and response kernels arise as the asymptotic limits of the averaged correlation and linear response functions of single coordinates of the system. These results enable an asymptotic analysis of an empirical Bayes Langevin dynamics procedure for learning an unknown prior parameter in a linear regression model, which we develop in a companion paper.

math.ST↗
arXiv2025

Dynamical mean-field analysis of adaptive Langevin diffusions: Replica-symmetric fixed point and empirical Bayes

In many applications of statistical estimation via sampling, one may wish to sample from a high-dimensional target distribution that is adaptively evolving to the samples already seen. We study an example of such dynamics, given by a Langevin diffusion for posterior sampling in a Bayesian linear regression model with i.i.d. regression design, whose prior continuously adapts to the Langevin trajectory via a maximum marginal-likelihood scheme. Results of dynamical mean-field theory (DMFT) developed in our companion paper establish a precise high-dimensional asymptotic limit for the joint evolution of the prior parameter and law of the Langevin sample. In this work, we carry out an analysis of the equations that describe this DMFT limit, under conditions of approximate time-translation-invariance which include, in particular, settings where the posterior law satisfies a log-Sobolev inequality. In such settings, we show that this adaptive Langevin trajectory converges on a dimension-independent time horizon to an equilibrium state that is characterized by a system of scalar fixed-point equations, and the associated prior parameter converges to a critical point of a replica-symmetric limit for the model free energy. As a by-product of our analyses, we obtain a new dynamical proof that this replica-symmetric limit for the free energy is exact, in models having a possibly misspecified prior and where a log-Sobolev inequality holds for the posterior law.

math.ST↗
arXiv2023

Gradient flows for empirical Bayes in high-dimensional linear models

Empirical Bayes provides a powerful framework for learning and adapting to latent structure in data. In sequence models where an independent observation is associated to each latent parameter, theory and methods around empirical Bayes are well-developed. However, in models where latent parameters and observed data interact through more complex designs, many statistical and algorithmic questions remain unanswered. In this work, we study a canonical setting of empirical Bayes estimation for the distribution of regression coefficients in a high-dimensional Bayesian or random effects linear model. Computationally, we propose a new system of gradient flow equations for computing a nonparametric maximum likelihood estimator (NPMLE), which jointly optimizes over the prior and posterior distributions of the regression coefficients in a Gibbs variational representation of the marginal log-likelihood. A diffusion-based implementation yields an adaptive Langevin dynamics algorithm in which the prior evolves continuously to optimize a sequence model log-likelihood defined by the coordinates of the Langevin sample. Theoretically, we show polynomial-time convergence of the proposed gradient flow to a near-NPMLE from any initialization within a convex sub-level set of the marginal log-likelihood, by developing a high-temperature log-Sobolev inequality for the posterior law. We establish the statistical consistency of any near-NPMLE under deterministic conditions for the regression design as $n,p\rightarrow\infty$.

math.ST↗
arXiv2023

The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing

We propose $\textsf{ScaledGD($\lambda$)}$, a preconditioned gradient descent method to tackle the low-rank matrix sensing problem when the true rank is unknown, and when the matrix is possibly ill-conditioned. Using overparametrized factor representations, $\textsf{ScaledGD($\lambda$)}$ starts from a small random initialization, and proceeds by gradient descent with a specific form of damped preconditioning to combat bad curvatures induced by overparameterization and ill-conditioning. At the expense of light computational overhead incurred by preconditioners, $\textsf{ScaledGD($\lambda$)}$ is remarkably robust to ill-conditioning compared to vanilla gradient descent ($\textsf{GD}$) even with overprameterization. Specifically, we show that, under the Gaussian design, $\textsf{ScaledGD($\lambda$)}$ converges to the true low-rank matrix at a constant linear rate after a small number of iterations that scales only logarithmically with respect to the condition number and the problem dimension. This significantly improves over the convergence rate of vanilla $\textsf{GD}$ which suffers from a polynomial dependency on the condition number. Our work provides evidence on the power of preconditioning in accelerating the convergence without hurting generalization in overparameterized learning.

cs.LG↗
arXiv2022

Empirical Bayes estimation: When does $g$-modeling beat $f$-modeling in theory (and in practice)?

Empirical Bayes (EB) is a popular framework for large-scale inference that aims to find data-driven estimators to compete with the Bayesian oracle that knows the true prior. Two principled approaches to EB estimation have emerged over the years: $f$-modeling, which constructs an approximate Bayes rule by estimating the marginal distribution of the data, and $g$-modeling, which estimates the prior from data and then applies the learned Bayes rule. For the Poisson model, the prototypical examples are the celebrated Robbins estimator and the nonparametric MLE (NPMLE), respectively. It has long been recognized in practice that the Robbins estimator, while being conceptually appealing and computationally simple, lacks robustness and can be easily derailed by ``outliers'', unlike the NPMLE which provides more stable and interpretable fit thanks to its Bayes form. On the other hand, not only do the existing theories shed little light on this phenomenon, but they all point to the opposite, as both methods have recently been shown optimal in terms of regret (excess over the Bayes risk) for compactly supported and subexponential priors. In this paper we provide a theoretical justification for the superiority of $g$-modeling over $f$-modeling for heavy-tailed data by considering priors with bounded $p>1$th moment. We show that with mild regularization, any $g$-modeling method that is Hellinger rate-optimal in density estimation achieves an optimal total regret $\tilde \Theta(n^{\frac{3}{2p+1}})$; in particular, the special case of NPMLE succeeds without regularization. In contrast, there exists an $f$-modeling estimator whose density estimation rate is optimal but whose EB regret is suboptimal by a polynomial factor. These results show that the proper Bayes form provides a ``general recipe of success'' for optimal EB estimation that applies to all $g$-modeling (but not $f$-modeling) methods.

math.ST↗
arXiv2022

Contiguity under high dimensional Gaussianity with applications to covariance testing

Le Cam's third/contiguity lemma is a fundamental probabilistic tool to compute the limiting distribution of a given statistic $T_n$ under a non-null sequence of probability measures $\{Q_n\}$, provided its limiting distribution under a null sequence $\{P_n\}$ is available, and the log likelihood ratio $\{\log (dQ_n/dP_n)\}$ has a distributional limit. Despite its wide-spread applications to low-dimensional statistical problems, the stringent requirement of Le Cam's third/contiguity lemma on the distributional limit of the log likelihood ratio makes it challenging, or even impossible to use in many modern high-dimensional statistical problems. This paper provides a non-asymptotic analogue of Le Cam's third/contiguity lemma under high dimensional normal populations. Our contiguity method is particularly compatible with sufficiently regular statistics $T_n$: the regularity of $T_n$ effectively reduces both the problems of (i) obtaining a null (Gaussian) limit distribution and of (ii) verifying our new quantitative contiguity condition, to those of derivative calculations and moment bounding exercises. More important, our method bypasses the need to understand the precise behavior of the log likelihood ratio, and therefore possibly works even when it necessarily fails to stabilize -- a regime beyond the reach of classical contiguity methods. As a demonstration of the scope of our new contiguity method, we obtain asymptotically exact power formulae for a number of widely used high-dimensional covariance tests, including the likelihood ratio tests and trace tests, that hold uniformly over all possible alternative covariance under mild growth conditions on the dimension-to-sample ratio. These new results go much beyond the scope of previous available case-specific techniques, and exhibit new phenomenon regarding the behavior of these important class of covariance tests.

math.ST↗
arXiv2022

Uncertainty quantification in the Bradley-Terry-Luce model

The Bradley-Terry-Luce (BTL) model is a benchmark model for pairwise comparisons between individuals. Despite recent progress on the first-order asymptotics of several popular procedures, the understanding of uncertainty quantification in the BTL model remains largely incomplete, especially when the underlying comparison graph is sparse. In this paper, we fill this gap by focusing on two estimators that have received much recent attention: the maximum likelihood estimator (MLE) and the spectral estimator. Using a unified proof strategy, we derive sharp and uniform non-asymptotic expansions for both estimators in the sparsest possible regime (up to some poly-logarithmic factors) of the underlying comparison graph. These expansions allow us to obtain: (i) finite-dimensional central limit theorems for both estimators; (ii) construction of confidence intervals for individual ranks; (iii) optimal constant of $\ell_2$ estimation, which is achieved by the MLE but not by the spectral estimator. Our proof is based on a self-consistent equation of the second-order remainder vector and a novel leave-two-out analysis.

math.ST↗
arXiv2022

Universality of regularized regression estimators in high dimensions

The Convex Gaussian Min-Max Theorem (CGMT) has emerged as a prominent theoretical tool for analyzing the precise stochastic behavior of various statistical estimators in the so-called high dimensional proportional regime, where the sample size and the signal dimension are of the same order. However, a well recognized limitation of the existing CGMT machinery rests in its stringent requirement on the exact Gaussianity of the design matrix, therefore rendering the obtained precise high dimensional asymptotics largely a specific Gaussian theory in various important statistical models. This paper provides a structural universality framework for a broad class of regularized regression estimators that is particularly compatible with the CGMT machinery. In particular, we show that with a good enough $\ell_\infty$ bound for the regression estimator $\hatμ_A$, any `structural property' that can be detected via the CGMT for $\hatμ_G$ (under a standard Gaussian design $G$) also holds for $\hatμ_A$ under a general design $A$ with independent entries. As a proof of concept, we demonstrate our new universality framework in three key examples of regularized regression estimators: the Ridge, Lasso and regularized robust regression estimators, where new universality properties of risk asymptotics and/or distributions of regression estimators and other related quantities are proved. As a major statistical implication of the Lasso universality results, we validate inference procedures using the degrees-of-freedom adjusted debiased Lasso under general design and error distributions. We also provide a counterexample, showing that universality properties for regularized regression estimators do not extend to general isotropic designs.

math.ST↗
arXiv2021

Nonparametric mixture MLEs under Gaussian-smoothed optimal transport distance

The Gaussian-smoothed optimal transport (GOT) framework, pioneered in Goldfeld et al. (2020) and followed up by a series of subsequent papers, has quickly caught attention among researchers in statistics, machine learning, information theory, and related fields. One key observation made therein is that, by adapting to the GOT framework instead of its unsmoothed counterpart, the curse of dimensionality for using the empirical measure to approximate the true data generating distribution can be lifted. The current paper shows that a related observation applies to the estimation of nonparametric mixing distributions in discrete exponential family models, where under the GOT cost the estimation accuracy of the nonparametric MLE can be accelerated to a polynomial rate. This is in sharp contrast to the classical sub-polynomial rates based on unsmoothed metrics, which cannot be improved from an information-theoretical perspective. A key step in our analysis is the establishment of a new Jackson-type approximation bound of Gaussian-convoluted Lipschitz functions. This insight bridges existing techniques of analyzing the nonparametric MLEs and the new GOT framework.

math.ST↗
arXiv2021

High dimensional asymptotics of likelihood ratio tests in the Gaussian sequence model under convex constraints

In the Gaussian sequence model $Y=μ+ξ$, we study the likelihood ratio test (LRT) for testing $H_0: μ=μ_0$ versus $H_1: μ\in K$, where $μ_0 \in K$, and $K$ is a closed convex set in $\mathbb{R}^n$. In particular, we show that under the null hypothesis, normal approximation holds for the log-likelihood ratio statistic for a general pair $(μ_0,K)$, in the high dimensional regime where the estimation error of the associated least squares estimator diverges in an appropriate sense. The normal approximation further leads to a precise characterization of the power behavior of the LRT in the high dimensional regime. These characterizations show that the power behavior of the LRT is in general non-uniform with respect to the Euclidean metric, and illustrate the conservative nature of existing minimax optimality and sub-optimality results for the LRT. A variety of examples, including testing in the orthant/circular cone, isotonic regression, Lasso, and testing parametric assumptions versus shape-constrained alternatives, are worked out to demonstrate the versatility of the developed theory.

math.ST↗
arXiv2021

Generalized kernel distance covariance in high dimensions: non-null CLTs and power universality

Distance covariance is a popular dependence measure for two random vectors $X$ and $Y$ of possibly different dimensions and types. Recent years have witnessed concentrated efforts in the literature to understand the distributional properties of the sample distance covariance in a high-dimensional setting, with an exclusive emphasis on the null case that $X$ and $Y$ are independent. This paper derives the first non-null central limit theorem for the sample distance covariance, and the more general sample (Hilbert-Schmidt) kernel distance covariance in high dimensions, primarily in the Gaussian case. The new non-null central limit theorem yields an asymptotically exact first-order power formula for the widely used generalized kernel distance correlation test of independence between $X$ and $Y$. The power formula in particular unveils an interesting universality phenomenon: the power of the generalized kernel distance correlation test is completely determined by $n\cdot \text{dcor}^2(X,Y)/\sqrt{2}$ in the high dimensional limit, regardless of a wide range of choices of the kernels and bandwidth parameters. Furthermore, this separation rate is also shown to be optimal in a minimax sense. The key step in the proof of the non-null central limit theorem is a precise expansion of the mean and variance of the sample distance covariance in high dimensions, which shows, among other things, that the non-null Gaussian approximation of the sample distance covariance involves a rather subtle interplay between the dimension-to-sample ratio and the dependence between $X$ and $Y$.

math.ST↗
arXiv2020

On a phase transition in general order spline regression

In the Gaussian sequence model $Y= θ_0 + \varepsilon$ in $\mathbb{R}^n$, we study the fundamental limit of approximating the signal $θ_0$ by a class $Θ(d,d_0,k)$ of (generalized) splines with free knots. Here $d$ is the degree of the spline, $d_0$ is the order of differentiability at each inner knot, and $k$ is the maximal number of pieces. We show that, given any integer $d\geq 0$ and $d_0\in\{-1,0,\ldots,d-1\}$, the minimax rate of estimation over $Θ(d,d_0,k)$ exhibits the following phase transition: \begin{equation*} \begin{aligned} \inf_{\widetildeθ}\sup_{θ\inΘ(d,d_0, k)}\mathbb{E}_θ\|\widetildeθ - θ\|^2 \asymp_d \begin{cases} k\log\log(16n/k), & 2\leq k\leq k_0,\\ k\log(en/k), & k \geq k_0+1. \end{cases} \end{aligned} \end{equation*} The transition boundary $k_0$, which takes the form $\lfloor{(d+1)/(d-d_0)\rfloor} + 1$, demonstrates the critical role of the regularity parameter $d_0$ in the separation between a faster $\log \log(16n)$ and a slower $\log(en)$ rate. We further show that, once encouraging an additional '$d$-monotonicity' shape constraint (including monotonicity for $d = 0$ and convexity for $d=1$), the above phase transition is eliminated and the faster $k\log\log(16n/k)$ rate can be achieved for all $k$. These results provide theoretical support for developing $\ell_0$-penalized (shape-constrained) spline regression procedures as useful alternatives to $\ell_1$- and $\ell_2$-penalized ones.

math.ST↗
arXiv2020

Optimal estimation of variance in nonparametric regression with random design

Consider the heteroscedastic nonparametric regression model with random design \begin{align*} Y_i = f(X_i) + V^{1/2}(X_i)\varepsilon_i, \quad i=1,2,\ldots,n, \end{align*} with $f(\cdot)$ and $V(\cdot)$ $α$- and $β$-Hölder smooth, respectively. We show that the minimax rate of estimating $V(\cdot)$ under both local and global squared risks is of the order \begin{align*} n^{-\frac{8αβ}{4αβ+ 2α+ β}} \vee n^{-\frac{2β}{2β+1}}, \end{align*} where $a\vee b := \max\{a,b\}$ for any two real numbers $a,b$. This result extends the fixed design rate $n^{-4α} \vee n^{-2β/(2β+1)}$ derived in Wang et al. [2008] in a non-trivial manner, as indicated by the appearances of both $α$ and $β$ in the first term. In the special case of constant variance, we show that the minimax rate is $n^{-8α/(4α+1)}\vee n^{-1}$ for variance estimation, which further implies the same rate for quadratic functional estimation and thus unifies the minimax rate under the nonparametric regression model with those under the density model and the white noise model. To achieve the minimax rate, we develop a U-statistic-based local polynomial estimator and a lower bound that is constructed over a specified distribution family of randomness designed for both $\varepsilon_i$ and $X_i$.

math.ST↗