arXiv · 2611.11277
Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning
Abstract
Stochastic first-order methods use sampled gradients to reduce per-iteration costs, but estimation errors can slow convergence and, in some settings, cause divergence. To improve gradient accuracy while retaining acceleration, we develop scalar and diagonal versions of VR-Adam-SHANG for smooth stochastic convex optimization. The methods combine STORM-type recursive estimation, Adam-style preconditioning, and SHANG's accelerated interpolation and correction, using at most two stochastic gradient evaluations per iteration without full-gradient refreshes. Under unbiased sampling, bounded variance, and mean-square smoothness of gradient noise, polynomial steps with prescribed initialization yield $O(1/K)$ mean-square gradient-estimation error and expected objective-gap bounds combining $O(1/K^2)$ initial-energy decay with an $O(1/\sqrt K)$ stochastic term after $K$ iterations. These guarantees require neither monotonicity of the preconditioner nor a uniform positive lower bound on its eigenvalues. The analysis also establishes polynomial-step convergence of projected nonrecursive Adam-SHANG-s, filling a gap in its original theory. Experiments compare the methods at matched sample-gradient budgets. In every tested setting, at least one of the two versions achieves the lowest mean terminal objective gap among all compared methods.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Yaxin Yu, Long Chen, Minfu Feng. 2026-11-08. Variance-Reduced Stochastic Hessian-Driven Nesterov Acceleration with Adam-Style Preconditioning. https://searcharxiv.com/abs/2611.11277
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.