arXiv subjects

Dongming Huang

Publications and source records attributed to Dongming Huang.

15 recordsLinked to original sources

arXiv2026

Linear Functional Testing with General Loadings in Sparse Regression: Separation Rates and Computational Barriers

We study the problem of testing $H_0: \xi^\top\beta=t_0$ in high-dimensional sparse linear regression with Gaussian random design and unknown design covariance. The loading vector $\xi$ is arbitrary, and the exact sparsity level $k$ is unknown but bounded by a known value $k_u$. Tests are required to control Type I error uniformly over the $k_u$-sparse null, while power is evaluated against $k$-sparse alternatives. We construct a computationally efficient mixed test that gives an upper bound on the adaptive separation distance and establish an information-theoretic lower bound calibrated to the magnitude profile of $\xi$. In the ultra-sparse regime $k_u\lesssim \sqrt n/\log p$, these bounds characterize the adaptive separation rate up to logarithmic factors for arbitrary $\xi$. In the moderately sparse regime $\sqrt n/\log p\ll k_u\lesssim n/\log p$, these bounds match for several classes of loading vectors but may differ in general. In this regime, we further prove a low-degree lower bound that matches the upper bound up to logarithmic factors. This provides evidence that improving on the rate of the mixed test, if statistically possible, may be computationally hard. For flat sparse loadings, we complement this evidence with a polynomial-time reduction from sparse CCA. Finally, we examine how information about the design covariance affects the adaptive separation rate in two settings. Under a sparse signed-spiked covariance model, the information-theoretic lower bound is attainable up to logarithmic factors by a computationally inefficient procedure, while the low-degree lower bound and sparse-CCA reduction continue to apply, providing evidence for a statistical-computational gap. When the design covariance is known and diagonal, the adaptive separation rate takes the same form as in the ultra-sparse regime.

math.ST↗

arXiv2026

Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions

Existing large-dimensional theory for spectral algorithms resolves either the optimally tuned point or the interpolation limit, but leaves the under-regularized regime unexplored. We study the learning curve and benign overfitting of spectral algorithms in the large-dimensional setting where the sample size and dimension are of comparable order, i.e., $n \asymp d^{\gamma}$ for some $\gamma>0$. We first consider inner-product kernels on the sphere $\mathbb{S}^{d-1}$ and establish a sharp asymptotic characterization of the excess risk across the full regularization path under various source conditions $s \geq 0$, where $s$ measures the relative smoothness of the regression function. Our results reveal that the learning curve is not simply U-shaped but instead consists of three distinct regimes: over-regularized, under-regularized, and interpolation regimes. This characterization allows us to fully capture the benign overfitting phenomenon, demonstrating that benign overfitting arises consistently across both the under-regularized and interpolation regimes whenever $s$ is positive but no larger than a critical threshold. We further show that, in the sufficiently regularized regime, the kernel learning curve is recovered by an associated sequence model. Finally, we extend the learning-curve analysis to large-dimensional KRR for a class of kernels on general domains in $\mathbb{R}^d$ whose low-degree eigenspaces satisfy spectral-scaling and hyper-contractivity conditions.

stat.ML↗

arXiv2025

Minimax and adaptive estimation of general linear functionals under sparsity

We study nonasymptotic minimax estimation of the linear functional $L(\theta)=\eta^\top \theta$ for a high-dimensional $s$-sparse mean vector with an arbitrary loading vector $\eta$. For symmetric noise with exponentially decaying tails, we derive the sharp minimax rate, explicit in $s$, $\eta$, the tail parameter, and the noise level. The proposed estimator combines plug-in estimation for coordinates with large loadings and thresholding for coordinates with small loadings, and the matching lower bound is obtained via a loading-dependent sparse prior. For unknown sparsity, we construct an $\eta$-dependent Lepski-type procedure and show that, for a broad verifiable class of loading vectors, its risk matches the oracle rate up to the optimal logarithmic factor. Explicit examples illustrate how heterogeneity in $\eta$ changes both the minimax and adaptive rates. We also extend the analysis to non-symmetric noise, hypothesis testing, and estimation with unknown noise variance, where we show that asymmetry can increase the minimax rate in certain examples of $\eta$. Among these results, the two main technical novelties are the following. First, we extend the sharp lower-bound theory beyond the Gaussian setting via a new $\chi^2$ bound for generalized Gaussian distributions. Second, for possibly non-symmetric noise, we derive new lower bounds through a worst-case asymmetric construction.

math.ST↗

arXiv2025

Alignment-Sensitive Minimax Rates for Spectral Algorithms with Learned Kernels

We study spectral algorithms in the setting where kernels are learned from data. We introduce the effective span dimension (ESD), an alignment-sensitive complexity measure that depends jointly on the signal, spectrum, and noise level $\sigma^2$. The ESD is well-defined for arbitrary kernels and signals without requiring eigen-decay conditions or source conditions. We prove that for sequence models whose ESD is at most $K$, the minimax excess risk scales as $\sigma^2 K$. Furthermore, we analyze over-parameterized gradient flow and prove that it can reduce the ESD. This finding establishes a connection between adaptive feature learning and provable improvements in generalization of spectral algorithms. We demonstrate the generality of the ESD framework by extending it to linear models and RKHS regression, and we support the theory with numerical experiments. This framework provides a novel perspective on generalization beyond traditional fixed-kernel theories.

cs.LG↗

arXiv2025

A Generalized Framework for Approximate Co-Sufficient Sampling

Approximate co-sufficient sampling (aCSS) offers a principled route to hypothesis testing when null distributions are unknown, yet current implementations are confined to maximum likelihood estimators with smooth or linear regularization and provide little theoretical insight into power. We present a generalized framework that widens the scope of the aCSS method to embrace nonlinear regularization, such as group lasso and nonconvex penalties, as well as robust and nonparametric estimators. Moreover, we introduce a weighted sampling scheme for enhanced flexibility and propose a generalized aCSS framework that unifies existing conditional sampling methods. Our theoretical analysis rigorously establishes validity and, for the first time, characterizes the power optimality of aCSS procedures in certain high-dimensional settings.

stat.ME↗

arXiv2025

On the Optimality of Functional Sliced Inverse Regression

In this paper, we prove that functional sliced inverse regression (FSIR) achieves the optimal (minimax) rate for estimating the central space in functional sufficient dimension reduction problems. First, we provide a concentration inequality for the FSIR estimator of the covariance of the conditional mean. Based on this inequality, we establish the root-$n$ consistency of the FSIR estimator of the image of covariance of the conditional mean. Second, we apply the most widely used truncated scheme to estimate the inverse of the covariance operator and identify the truncation parameter that ensures that FSIR can achieve the optimal minimax convergence rate for estimating the central space. Finally, we conduct simulations to demonstrate the optimal choice of truncation parameter and the estimation efficiency of FSIR. To the best of our knowledge, this is the first paper to rigorously prove the minimax optimality of FSIR in estimating the central space for multiple-index models and general $Y$ (not necessarily discrete).

math.ST↗

arXiv2025

Testing Multivariate Conditional Independence Using Exchangeable Sampling and Sufficient Statistics

We consider testing multivariate conditional independence between a response Y and a covariate vector X given additional variables Z. We introduce the Multivariate Sufficient Statistic Conditional Randomization Test (MS-CRT), which generates exchangeable copies of X by conditioning on sufficient statistics of P(X|Z). MS-CRT requires no modelling assumption on Y and accommodates any test statistics, including those derived from complex predictive models. It relaxes the assumptions of standard conditional randomization tests by allowing more unknown parameters in P(X|Z) than the sample size. MS-CRT avoids multiplicity corrections and effectively detects joint signals, even when individual components of X have only weak effects on Y . Our method extends to group selection with false discovery rate control. We develop efficient implementations for two important cases where P(X,Z) is either multivariate normal or belongs to a graphical model. For normal models, we establish the minimax rate optimality. For graphical models, we demonstrate the superior performance of our method compared to existing methods through comprehensive simulations and real-data examples.

stat.ME↗

arXiv2025

Goodness-of-Fit Tests for High-Dimensional Gaussian Graphical Models via Exchangeable Sampling

We introduce a general framework for testing goodness-of-fit for Gaussian graphical models in both the low- and high-dimensional settings. This framework is based on a novel algorithm for generating exchangeable copies by conditioning on sufficient statistics. This framework provides exact finite-sample error control regardless of the dimension and allows flexible choices of test statistics to improve power. We explore several candidate test statistics and conduct extensive simulation studies to demonstrate their finite-sample performance compared to existing methods. The proposed tests exhibit superior power, particularly in cases where the true precision matrix deviates from the null hypothesis due to many small nonzero entries. To justify theoretically, we consider a high-dimensional setting where the proposed test achieves rate-optimality under two distinct signal patterns in the precision matrix: (1) dense patterns with many small nonzero entries and (2) strong patterns with at least one large entry. Finally, we illustrate the usefulness of the proposed test through real-world applications.

stat.ME↗

arXiv2024

Bayesian inference on Cox regression models using catalytic prior distributions

The Cox proportional hazards model (Cox model) is a popular model for survival data analysis. When the sample size is small relative to the dimension of the model, the standard maximum partial likelihood inference is often problematic. In this work, we propose the Cox catalytic prior distributions for Bayesian inference on Cox models, which is an extension of a general class of prior distributions originally designed for stabilizing complex parametric models. The Cox catalytic prior is formulated as a weighted likelihood of the regression coefficients based on synthetic data and a surrogate baseline hazard constant. This surrogate hazard can be either provided by the user or estimated from the data, and the synthetic data are generated from the predictive distribution of a fitted simpler model. For point estimation, we derive an approximation of the marginal posterior mode, which can be computed conveniently as a regularized log partial likelihood estimator. We prove that our prior distribution is proper and the resulting estimator is consistent under mild conditions. In simulation studies, our proposed method outperforms standard maximum partial likelihood inference and is on par with existing shrinkage methods. We further illustrate the application of our method to a real dataset.

stat.ME↗

arXiv2024

On the Structural Dimension of Sliced Inverse Regression

In this work, we address the longstanding puzzle that Sliced Inverse Regression (SIR) often performs poorly for sufficient dimension reduction when the structural dimension $d$ (the dimension of the central space) exceeds 4. We first show that in the multiple index model $Y=f( \mathbf{P} \boldsymbol{X})+ε$ where $\boldsymbol{X}$ is a $p$-standard normal vector, $ε$ is an independent noise, and $\mathbf{P}$ is a projection operator from $\mathbb R^{p}$ to $\mathbb R^{d}$, if the link function $f$ follows the law of a Gaussian process, then with high probability, the $d$-th eigenvalue $λ_{d}$ of $\mathrm{Cov}\left[\mathbb{E}(\boldsymbol{X}\mid Y)\right]$ satisfies $λ_{d}\leq C e^{-θd}$ for some positive constants $C$ and $θ$. We then focus on the low signal regime where $λ_{d}$ can be arbitrarily small and not larger than $d^{-8.1}$, and prove that the minimax risk of estimating the central space is lower bounded by $\frac{dp}{nλ_{d}}$. Combining these two results, we provide a convincing explanation for the poor performance of SIR when $d$ is large, a phenomenon that has perplexed researchers for nearly three decades. The technical tools developed here may be of independent interest for studying other sufficient dimension reduction methods.

math.ST↗

arXiv2024

Regularization Using Synthetic Data for High-Dimensional Inference

To address the challenges of obtaining reliable inference in high-dimensional models, we introduce the Synthetic-data Regularized Estimator (SRE). Unlike traditional regularization methods, the SRE regularizes the complex target model via a weighted likelihood based on synthetic data generated from a simpler, more stable model. This method provides a theoretically sound and practically effective alternative to parameter penalization. We establish key theoretical properties of the SRE in generalized linear models, including existence, stability, consistency, and minimax rate optimality. We leverage the Convex Gaussian Min-max Theorem to derive precise asymptotic characterizations in high-dimensional linear regimes where $n/p \to \delta > 0$, both for noninformative synthetic data and for informative auxiliary data in a transfer learning setting. Our asymptotic results characterize how performance depends on the signal strength and the similarity between target and auxiliary data sources. Building upon these results, we develop practical methodologies for high-dimensional inference, including tuning parameter selection, confidence interval construction, and calibrated variable selection. The effectiveness of the SRE is demonstrated through simulation studies and real-data applications.

math.ST↗

arXiv2024

The Optimality of Kernel Classifiers in Sobolev Space

Kernel methods are widely used in machine learning, especially for classification problems. However, the theoretical analysis of kernel classification is still limited. This paper investigates the statistical performances of kernel classifiers. With some mild assumptions on the conditional probability $η(x)=\mathbb{P}(Y=1\mid X=x)$, we derive an upper bound on the classification excess risk of a kernel classifier using recent advances in the theory of kernel regression. We also obtain a minimax lower bound for Sobolev spaces, which shows the optimality of the proposed classifier. Our theoretical results can be extended to the generalization error of overparameterized neural network classifiers. To make our theoretical results more applicable in realistic settings, we also propose a simple method to estimate the interpolation smoothness of $2η(x)-1$ and apply the method to real datasets.

math.ST↗

arXiv2023

Catalytic Priors: Using Synthetic Data to Specify Prior Distributions in Bayesian Analysis

Catalytic prior distributions provide general, easy-to-use, and interpretable specifications of prior distributions for Bayesian analysis. They are particularly beneficial when the observed data are inadequate to stably estimate a complex target model. A catalytic prior distribution is constructed by augmenting the observed data with synthetic data that are sampled from the predictive distribution of a simpler model estimated from the observed data. We illustrate the usefulness of the catalytic prior approach using an example from labor economics. In the example, the resulting Bayesian inference reflects many important aspects of the observed data, and the estimation accuracy and predictive performance of the inference based on the catalytic prior are superior to, or comparable to, that of other commonly used prior distributions. We further explore the connection between the catalytic prior approach and a few popular regularization methods. We expect the catalytic prior approach to be useful in many applications.

stat.ME↗

arXiv2020

Relaxing the Assumptions of Knockoffs by Conditioning

The recent paper Candès et al. (2018) introduced model-X knockoffs, a method for variable selection that provably and non-asymptotically controls the false discovery rate with no restrictions or assumptions on the dimensionality of the data or the conditional distribution of the response given the covariates. The one requirement for the procedure is that the covariate samples are drawn independently and identically from a precisely-known (but arbitrary) distribution. The present paper shows that the exact same guarantees can be made without knowing the covariate distribution fully, but instead knowing it only up to a parametric model with as many as $Ω(n^{*}p)$ parameters, where $p$ is the dimension and $n^{*}$ is the number of covariate samples (which may exceed the usual sample size $n$ of labeled samples when unlabeled samples are also available). The key is to treat the covariates as if they are drawn conditionally on their observed value for a sufficient statistic of the model. Although this idea is simple, even in Gaussian models conditioning on a sufficient statistic leads to a distribution supported on a set of zero Lebesgue measure, requiring techniques from topological measure theory to establish valid algorithms. We demonstrate how to do this for three models of interest, with simulations showing the new approach remains powerful under the weaker assumptions.

stat.ME↗

arXiv2017

On the optimality of sliced inverse regression in high dimensions

The central subspace of a pair of random variables $(y,x) \in \mathbb{R}^{p+1}$ is the minimal subspace $\mathcal{S}$ such that $y \perp \hspace{-2mm} \perp x\mid P_{\mathcal{S}}x$. In this paper, we consider the minimax rate of estimating the central space of the multiple index models $y=f(β_{1}^τx,β_{2}^τx,...,β_{d}^τx,ε)$ with at most $s$ active predictors where $x \sim N(0,I_{p})$. We first introduce a large class of models depending on the smallest non-zero eigenvalue $λ$ of $var(\mathbb{E}[x|y])$, over which we show that an aggregated estimator based on the SIR procedure converges at rate $d\wedge((sd+s\log(ep/s))/(nλ))$. We then show that this rate is optimal in two scenarios: the single index models; and the multiple index models with fixed central dimension $d$ and fixed $λ$. By assuming a technical conjecture, we can show that this rate is also optimal for multiple index models with bounded dimension of the central space. We believe that these (conditional) optimal rate results bring us meaningful insights of general SDR problems in high dimensions.

math.ST↗