arXiv subjects

Haihan Zhang

Publications and source records attributed to Haihan Zhang.

11 recordsLinked to original sources

arXiv2026

Near-Optimal Deterministic Exact-Value Complexity for Smooth Convex Optimization

We study the deterministic oracle complexity of smooth convex optimization when the algorithm receives only exact function values. The objective is a globally $β$-smooth convex function, all queries and the final output are restricted to the Euclidean ball of radius $R$, and the unique minimizer lies in the ball of radius $R/2$. We establish an upper bound of $O(d\sqrt{βR^2/ε})$ using coordinate finite differences together with an error-robust accelerated projected method. Our main contribution is a matching lower bound, up to the high-accuracy saturation of the construction: any deterministic adaptive value-oracle algorithm requires $Ω\!\left(d\min\{\sqrt{βR^2/ε},(d/\log(ed))^{1/3}\}\right)$ queries. Consequently, the minimax oracle complexity is $Θ(d\sqrt{βR^2/ε})$ throughout the moderate-accuracy regime $βR^2(\log(ed)/d)^{2/3}\leqε\leq cβR^2$ for a universal constant $c>0$. The lower bound must account for the fact that a single exact real value can encode arbitrarily much information. To overcome this difficulty, we construct a single fixed smooth convex hard instance using a Moreau-smoothed biased max chain, an exact prefix-shielding mechanism, and batched delayed rotations. These techniques preserve consistency with the full adaptive transcript and establish the optimality of the square-root complexity branch for deterministic bounded-query algorithms.

math.OC↗

arXiv2026

Near-Optimal Exact-Value Zeroth-Order Complexity for Smooth Strongly Convex Optimization

We study deterministic adaptive optimization of globally $β$-smooth, $μ$-strongly convex functions using exact scalar function values. Queries and outputs lie in $B_2^d(R)$, and the minimizer lies in $B_2^d(R/2)$. Set $κ=β/μ$, $Q=βR^2/ε$, and $D_d=(d/\log(ed))^{1/3}$. For sufficiently large $d$ and $0<ε\le c_εβR^2$, the minimax value complexity $N_ε$ satisfies \[ \begin{aligned} N_ε&\ge c d\min\{\sqrt Q,\sqrtκ,D_d\},\\ N_ε&\le C d\min\left\{ \sqrt Q,\sqrtκ[1+\log_+(Q/κ)] \right\}, \end{aligned} \] where $c,C,c_ε>0$ are universal constants and $\log_+(t)=\max\{0,\log t\}$. The lower bound uses an exactly shielded smooth chain and batched delayed rotations; the upper bound combines finite differences, acceleration, and restart. When $\min\{Q,κ\}\le D_d^2$, these bounds match up to constants in the accuracy-dominated regime $Q\leκ$ and at constant relative accuracy $ε=Θ(μR^2)$. For arbitrarily higher accuracy in the range $κ\le D_d^2$, the bounds differ by at most $1+\log(μR^2/ε)$; the optimal accuracy dependence remains unresolved in general.

math.OC↗

arXiv2026

Matching Higher-Order Oracle Complexity for Smooth Monotone Variational Inequalities

We establish near-optimal higher-order oracle bounds for smooth monotone variational inequalities. For fixed $p\ge2$, let $F$ be monotone on a known compact convex set $X$ of diameter at most $D$, with $\operatorname{Lip}(D^{p-1}F)\le L_p$. Each feasible query returns the complete jet $(F,DF,\ldots,D^{p-1}F)$, and the goal is to find $x$ with tangent residual $\operatorname{dist}(0,F(x)+N_X(x))\le\varepsilon$. Writing $Q=L_pD^p/\varepsilon$, we improve the $\widetilde O_p(Q^{1/p})$ upper bound of Chen et al. to $\widetilde O_p(Q^{2/(3p-1)})$ via a dimension-independent deterministic algorithm that returns an explicit tangent-residual certificate. We prove a matching $Ω_p(Q^{2/(3p-1)})$ lower bound for arbitrary adaptive deterministic algorithms and randomized algorithms with per-instance success probability at least $2/3$, without span or tensor-update restrictions. Hence the high-dimensional worst-case oracle complexity is $\widetildeΘ_p((L_pD^p/\varepsilon)^{2/(3p-1)})$. The same method applies to smooth convex--concave minimax problems, improving the fixed-geometry accuracy exponent from $4/(3p+1)$ to $2/(3p-1)$ and matching the known lower-bound exponent.

math.OC↗

arXiv2026

Near-Optimal Lower Bounds for Randomized Algorithms in Exact Value Zeroth-Order Convex Optimization

Whether exact scalar feedback intrinsically incurs the additional dimension $d$ paid by known zeroth-order methods remains open even for Lipschitz convex optimization. For a universal Lipschitz scale, the value only bound $O(d^2\log(d+1)\log(1/ε))$ and two-point bound $O(dε^{-2})$ yield the upper bound $\widetilde O\left(d\min\{d,ε^{-2}\}\right)$. By contrast, prior lower bounds for arbitrary randomized algorithms give only $Ω(\min\{d,ε^{-2}\})$, leaving a factor $d$ unexplained. We close this gap, up to logarithmic factors, for arbitrary adaptive randomized algorithms minimizing a convex objective with a universal Lipschitz scale over the $d$-dimensional Euclidean unit ball, where each query returns only the exact scalar value. Let $T_ε$ denote the minimum number of queries required to return an $ε$-suboptimal point with probability at least $1/2$, uniformly over the function class. We prove that \[T_ε\ge c\,\frac{d\min\{d,ε^{-2}\}}{\log\!\bigl(\min\{d,ε^{-2}\}\bigr)},\] for $d\ge d_0$ and $0<ε\leε_0$, where $c,ε_0>0$ and $d_0\in\mathbb N$ are universal constants. This gives $Ω\left(\frac{d}{ε^2\log(1/ε)}\right)$ in the low-accuracy regime $ε\ge d^{-1/2}$ and $Ω\left(\frac{d^2}{\log d}\right)$ in the high-accuracy regime $ε\le d^{-1/2}$ with the latter independent of $ε$. These bounds match the corresponding upper bound up to logarithmic factors. To our knowledge, this is the first near-optimal lower bound for arbitrary adaptive randomized algorithms throughout both accuracy regimes of exact value Lipschitz convex optimization. The proof uses a random support function hard family and develops a posterior mean energy method for adaptive exact max observations, in place of first-order zero chain constructions and noise based transcript inequalities.

math.OC↗

arXiv2026

What Makes a Strong Model? A Unified Spectral Analysis of Knowledge Transfer over High-dimensional Linear Regression

Teacher-Student Knowledge Transfer (KT) is ubiquitous in modern machine learning, ranging from classical model compression via Knowledge Distillation (KD) to the emergent phenomenon of Weak-to-Strong (W2S) generalization. While existing studies offer isolated insights, a unified theoretical framework explaining the efficacy of KT across these disparate regimes remains lacking. In this work, we establish a unified spectral analysis of SGD dynamics in high-dimensional linear regression, elucidating the efficiency of KT across seemingly disparate regimes. We characterize KT efficiency through two distinct mechanisms: \emph{Spectral Horizon Expansion} in KD, which enables the capture of statistically inaccessible high-frequency signals, and \emph{Spectral Denoising} in W2S, where the student acts as a filter for optimization noise. Our framework unifies these phenomena, revealing that the efficacy of transfer is governed by the interplay between implicit regularization and heterogeneous spectral learning speeds over the spectrum.

cs.LG↗

arXiv2026

Reliable Fine-Grained Evaluation of Natural Language Math Proofs

Recent advances in large language models (LLMs) for mathematical reasoning have largely focused on tasks with easily verifiable final answers while generating and verifying natural language math proofs remains an open challenge. We identify the absence of a reliable, fine-grained evaluator for LLM-generated math proofs as a critical gap. To address this, we propose a systematic methodology for developing and validating evaluators that assign fine-grained scores on a 0-7 scale to model-generated math proofs. To enable this study, we introduce ProofBench, the first expert-annotated dataset of fine-grained proof ratings, spanning 145 problems from six major math competitions (USAMO, IMO, Putnam, etc) and 435 LLM-generated solutions from Gemini-2.5-Pro, o3, and DeepSeek-R1. Using ProofBench as a testbed, we systematically explore the evaluator design space across key axes: the backbone model, input context, instructions and evaluation workflow. Our analysis delivers ProofGrader, an evaluator that combines a strong reasoning backbone LM, rich context from reference solutions and marking schemes, and a simple ensembling method; it achieves a low Mean Absolute Error (MAE) of 0.926 against expert scores, significantly outperforming naive baselines. Finally, we demonstrate its practical utility in a best-of-$n$ selection task: at $n=16$, ProofGrader achieves an average score of 4.14/7, closing 78\% of the gap between a naive binary evaluator (2.48) and the human oracle (4.62), highlighting its potential to advance downstream proof generation.

cs.CL↗

arXiv2025

Learning Curves of Stochastic Gradient Descent in Kernel Regression

This paper considers a canonical problem in kernel regression: how good are the model performances when it is trained by the popular online first-order algorithms, compared to the offline ones, such as ridge and ridgeless regression? In this paper, we analyze the foundational single-pass Stochastic Gradient Descent (SGD) in kernel regression under source condition where the optimal predictor can even not belong to the RKHS, i.e. the model is misspecified. Specifically, we focus on the inner product kernel over the sphere and characterize the exact orders of the excess risk curves under different scales of sample sizes $n$ concerning the input dimension $d$. Surprisingly, we show that SGD achieves min-max optimal rates up to constants among all the scales, without suffering the saturation, a prevalent phenomenon observed in (ridge) regression, except when the model is highly misspecified and the learning is in a final stage where $n\gg d^γ$ with any constant $γ>0$. The main reason for SGD to overcome the curse of saturation is the exponentially decaying step size schedule, a common practice in deep neural network training. As a byproduct, we provide the \emph{first} provable advantage of the scheme over the iterative averaging method in the common setting.

stat.ML↗

arXiv2025

Optimal Algorithms in Linear Regression under Covariate Shift: On the Importance of Precondition

A common pursuit in modern statistical learning is to attain satisfactory generalization out of the source data distribution (OOD). In theory, the challenge remains unsolved even under the canonical setting of covariate shift for the linear model. This paper studies the foundational (high-dimensional) linear regression where the ground truth variables are confined to an ellipse-shape constraint and addresses two fundamental questions in this regime: (i) given the target covariate matrix, what is the min-max \emph{optimal} algorithm under covariate shift? (ii) for what kinds of target classes, the commonly-used SGD-type algorithms achieve optimality? Our analysis starts with establishing a tight lower generalization bound via a Bayesian Cramer-Rao inequality. For (i), we prove that the optimal estimator can be simply a certain linear transformation of the best estimator for the source distribution. Given the source and target matrices, we show that the transformation can be efficiently computed via a convex program. The min-max optimal analysis for SGD leverages the idea that we recognize both the accumulated updates of the applied algorithms and the ideal transformation as preconditions on the learning variables. We provide sufficient conditions when SGD with its acceleration variants attain optimality.

stat.ML↗

arXiv2025

Scaling Law for Stochastic Gradient Descent in Quadratically Parameterized Linear Regression

In machine learning, the scaling law describes how the model performance improves with the model and data size scaling up. From a learning theory perspective, this class of results establishes upper and lower generalization bounds for a specific learning algorithm. Here, the exact algorithm running using a specific model parameterization often offers a crucial implicit regularization effect, leading to good generalization. To characterize the scaling law, previous theoretical studies mainly focus on linear models, whereas, feature learning, a notable process that contributes to the remarkable empirical success of neural networks, is regretfully vacant. This paper studies the scaling law over a linear regression with the model being quadratically parameterized. We consider infinitely dimensional data and slope ground truth, both signals exhibiting certain power-law decay rates. We study convergence rates for Stochastic Gradient Descent and demonstrate the learning rates for variables will automatically adapt to the ground truth. As a result, in the canonical linear regression, we provide explicit separations for generalization curves between SGD with and without feature learning, and the information-theoretical lower bound that is agnostic to parametrization method and the algorithm. Our analysis for decaying ground truth provides a new characterization for the learning dynamic of the model.

cs.LG↗

arXiv2024

The Optimality of (Accelerated) SGD for High-Dimensional Quadratic Optimization

Stochastic gradient descent (SGD) is a widely used algorithm in machine learning, particularly for neural network training. Recent studies on SGD for canonical quadratic optimization or linear regression show it attains well generalization under suitable high-dimensional settings. However, a fundamental question -- for what kinds of high-dimensional learning problems SGD and its accelerated variants can achieve optimality has yet to be well studied. This paper investigates SGD with two essential components in practice: exponentially decaying step size schedule and momentum. We establish the convergence upper bound for momentum accelerated SGD (ASGD) and propose concrete classes of learning problems under which SGD or ASGD achieves min-max optimal convergence rates. The characterization of the target function is based on standard power-law decays in (functional) linear regression. Our results unveil new insights for understanding the learning bias of SGD: (i) SGD is efficient in learning ``dense'' features where the corresponding weights are subject to an infinity norm constraint; (ii) SGD is efficient for easy problem without suffering from the saturation effect; (iii) momentum can accelerate the convergence rate by order when the learning problem is relatively hard. To our knowledge, this is the first work to clearly identify the optimal boundary of SGD versus ASGD for the problem under mild settings.

cs.LG↗

arXiv2022

Bregman Divergence-Based Data Integration with Application to Polygenic Risk Score (PRS) Heterogeneity Adjustment

Polygenic risk scores (PRS) have recently received much attention for genetics risk prediction. While successful for the Caucasian population, the PRS based on the minority population suffer from small sample sizes, high dimensionality and low signal-to-noise ratios, exacerbating already severe health disparities. Due to population heterogeneity, direct trans-ethnic prediction by utilizing the Caucasian model for the minority population also has limited performance. In addition, due to data privacy, the individual genotype data is not accessible for either the Caucasian population or the minority population. To address these challenges, we propose a Bregman divergence-based estimation procedure to measure and optimally balance the information from different populations. The proposed method only requires the use of encrypted summary statistics and improves the PRS performance for ethnic minority groups by incorporating additional information. We provide the asymptotic consistency and weak oracle property for the proposed method. Simulations and real data analyses also show its advantages in prediction and variable selection.

stat.ME↗