arXiv subjects

Jinhui Bai

Publications and source records attributed to Jinhui Bai.

3 recordsLinked to original sources

arXiv2026

Adaptive Schauder Stochastic Mirror Descent in Banach Spaces

In this paper, we extend stochastic mirror descent (SMD) to infinite-dimensional Banach spaces for solving a class of risk functional minimization problems, where stochastic gradient information is only available through sampling. We first choose the Bregman distance according to the uniform convexity properties of the Banach space. For the non-uniformly convex $\mathcal{L}^1$ space, we instead construct a Bregman distance induced by the entropy function. Based on a Schauder basis of the Banach space, we introduce a family of finite-dimensional subspaces that adapt to the sample size $n$. At each SMD iteration, we restrict the subproblem to the corresponding finite-dimensional subspace and project the stochastic gradient onto the associated finite-dimensional dual space, thereby introducing an adaptive regularization in Banach spaces. This regularization strategy allows the SMD subproblem to be solved efficiently. By developing a new analytical framework, we prove that the proposed algorithm achieves a convergence rate of $\mathcal{O}\left(n^{-1/p_1}\right)$ up to logarithmic factors, where $p_1\geq 2$ is determined by the convexity properties of the underlying space. In the misspecified setting, where the minimizer satisfies only weaker regularity conditions, we further show that the risk functional still converges to its minimum value. Moreover, processing $n$ samples requires only $\mathcal{O}(n^{1+\theta})$ computational time and $\mathcal{O}(n^\theta)$ memory, where $\theta>0$ can be chosen arbitrarily small when the minimizer has sufficient regularity. We further apply the algorithm to solve statistical inverse problems and validate its effectiveness in numerical experiments.

math.OC↗

arXiv2025

Truncated Kernel Stochastic Gradient Descent with General Losses and Spherical Radial Basis Functions

In this paper, we propose a novel kernel stochastic gradient descent (SGD) algorithm for large-scale supervised learning with general losses. Compared to traditional kernel SGD, our algorithm improves efficiency and scalability through an adaptive regularization strategy. By leveraging the infinite series expansion of spherical radial basis functions, this strategy projects the stochastic gradient onto a finite-dimensional hypothesis space, which is adaptively scaled according to the bias-variance trade-off, thereby enhancing generalization performance. To handle the gradient nonlinearity arising from general losses, we develop a new generalization framework combining an inequality-based characterization of the kernel-induced covariance operator with optimization techniques. We prove that both the last iterate and the suffix average converge at minimax-optimal rates, and we further establish optimal strong convergence in the reproducing kernel Hilbert space. Our framework accommodates a broad class of classical loss functions, including least-squares, Huber, and logistic losses. Moreover, the proposed algorithm significantly reduces computational complexity and achieves optimal storage complexity by incorporating coordinate-wise updates from linear SGD, thereby avoiding the costly pairwise operations typical of kernel SGD and enabling efficient processing of streaming data. Finally, extensive numerical experiments provide empirical support for the theoretical results and the computational advantages of our algorithm.

cs.LG↗

arXiv2025

Truncated Kernel Stochastic Gradient Descent on Spheres

Inspired by the structure of spherical harmonics, we propose the truncated kernel stochastic gradient descent (T-kernel SGD) algorithm with a least-square loss function for spherical data fitting. T-kernel SGD introduces a novel regularization strategy by implementing stochastic gradient descent through a closed-form solution of the projection of the stochastic gradient in a low-dimensional subspace. In contrast to traditional kernel SGD, the regularization strategy implemented by T-kernel SGD is more effective in balancing bias and variance by dynamically adjusting the hypothesis space during iterations. The most significant advantage of the proposed algorithm is that it can achieve theoretically optimal convergence rates using a constant step size (independent of the sample size) while overcoming the inherent saturation problem of kernel SGD. Additionally, we leverage the structure of spherical polynomials to derive an equivalent T-kernel SGD, significantly reducing storage and computational costs compared to kernel SGD. Typically, T-kernel SGD requires only $\mathcal{O}(n^{1+\frac{d}{d-1}ε})$ computational complexity and $\mathcal{O}(n^{\frac{d}{d-1}ε})$ storage to achieve optimal rates for the d-dimensional sphere, where $0<ε<\frac{1}{2}$ can be arbitrarily small if the optimal fitting or the underlying space possesses sufficient regularity. This regularity is determined by the smoothness parameter of the objective function and the decaying rate of the eigenvalues of the integral operator associated with the kernel function, both of which reflect the difficulty of the estimation problem. Our main results quantitatively characterize how this prior information influences the convergence of T-kernel SGD. The numerical experiments further validate the theoretical findings presented in this paper.

cs.LG↗