arXiv subjects

Xiaoming Yuan

Publications and source records attributed to Xiaoming Yuan.

At least 19 recordsLinked to original sources

arXiv2026

From Pólya's Conditions to a Complete Characterization of the $L^2$ Convergence of Hyperinterpolation

It has remained open to identify the necessary and sufficient conditions for the $L^2$ convergence of hyperinterpolation since it was introduced by Sloan in 1995. We show that the $L^1$-$L^2$ Marcinkiewicz-Zygmund (MZ) condition, together with the asymptotic functional approximation property for polynomials, is the answer. We further prove that the optimal $L^1$-$L^2$ MZ constant coincides with the operator norm of the hyperinterpolation operator, and it admits a natural Banach space duality interpretation. With an explicit construction, we also show that Pólya's classical conditions for quadrature convergence are not sufficient for the $L^2$ convergence of hyperinterpolation. This reveals a fundamental distinction between the convergence of linear functionals (quadrature formulas) and that of linear operators (hyperinterpolation operators). We establish a strict logical hierarchy for the stability and accuracy conditions governing the convergence of quadrature and hyperinterpolation.

math.NA↗

arXiv2026

An Optimization Approach to Weight Collocation for Scattered Spherical Data

We introduce an optimization approach for constructing spherical quadrature rules on arbitrarily scattered data. Rather than designing node placements, the new approach focuses on optimally computing the weights for fixed configurations. Motivated by Pólya's necessary and sufficient conditions for quadrature convergence in 1933, we argue that pursuing weight positivity and high algebraic exactness for scattered data approximation is not necessary. To align the quadrature design with the underlying theory of approximation, we construct convex optimization models with suitable objective functionals by examining the accuracy of numerical integration with reproducing kernels of Sobolev spaces and the performance of hyperinterpolation with Marcinkiewicz-Zygmund (MZ) inequalities. The resulting optimization models encode the spatial distribution of the scattered sites and the analytic properties of the target function spaces. The proposed approach enables the derivation of rigorous theoretical stability bounds, and the resulting quadrature weights are efficiently computable by modern convex optimization techniques. Numerical results are reported to demonstrate the performance of the optimization approach for fundamental approximation tasks such as numerical integration and hyperinterpolation for scattered spherical data.

math.NA↗

arXiv2026

Optimization on Affine-Transversal Hilbert Submanifolds: Part I -- Theoretical Foundations

In this paper, we establish the theoretical foundations for the generic optimization problem in a Hilbert space whose feasible set is an affine-transversal Hilbert submanifold given by the intersection of a nonlinear manifold and an affine subspace. We develop the geometric and analytical toolkit, such as the tangent-space characterization, projection operators, and implicit retraction operators, to essentially ensure the feasibility of iterates for algorithmic design. We also derive weak-form expressions for the derivatives of lifted objective functionals. In particular, we introduce a projection-induced Riemannian metric whose induced norm is uniformly equivalent to the ambient norm, under which the projection-induced gradient becomes an exact Riemannian gradient with an explicit formula. This construction replaces the implicit tangent-space Riesz representation underlying classical Riemannian optimization with directly computable operator evaluations, yielding a practical variable-metric framework for algorithmic design while preserving the geometric structure required for convergence analysis. With these theoretical foundations, it becomes possible to apply standard techniques in Euclidean spaces to design algorithms with strictly feasible iterates for the optimization problem on an affine-transversal Hilbert submanifold. We also propose the theoretical frameworks for algorithmic design on affine-transversal Hilbert submanifolds by showcasing the Riemannian line-search and trust-region algorithms with rigorous convergence analysis.

math.OC↗

arXiv2026

Learning Theory of the SVRG: Generalization and Convergence Analysis

Variance reduction (VR) methods employ stochastic gradients with decreasing variance, and they have been widely applied to solve large-scale optimization problems in machine learning because of their efficiency. Existing theoretical studies of VR methods are mainly focused on the convergence analysis, leaving the generalization behavior largely unexplored. In this paper, we bridge this gap by developing the first non-vacuous generalization analysis of the representative VR method: Stochastic Variance Reduced Gradient (SVRG), through the lens of algorithmic stability. In particular, we establish sharp stability bounds of the SVRG in both convex and strongly convex settings by exploiting its algorithmic structure. The obtained bounds are data-dependent, because the training errors are incorporated along the trajectory. Our analysis clarifies the interplay between optimization and generalization, leading to optimal excess population risk bounds in both settings. Our approach differs substantially from existing analyses of stochastic algorithms in the sense that we decompose the SVRG update as an SGD-like step plus a zero-mean correction term and then introduce novel Lyapunov functions to absorb the additional gradient terms induced by the reference points. Our analytical framework can be generalized to other VR methods, and we demonstrate the generalization by the well-known Stochastic Average Gradient Accelerated (SAGA) method.

cs.LG↗

arXiv2026

Stochastic Gradient Descent with Momentum is Algorithmically Stable

Stochastic gradient descent with momentum (SGDM) is one of the most widely used optimization algorithms in machine learning. While optimization properties of SGDM have been extensively studied in the literature, it remains insufficiently understood whether and when SGDM can generalize well to unseen data. In particular, it has been conjectured that while momentum accelerates training, it may degrade generalization. In this paper, we close this gap by developing a comprehensive generalization analysis of SGDM through the lens of algorithmic stability. More specifically, we introduce a generalized SGDM framework that encompasses both Polyak's and Nesterov's momentum schemes, and establish tight on-average model stability bounds for smooth and convex problems. Notably, the obtained bounds exploit small optimization error bounds along the trajectory, apply to any momentum parameter in the interval $[0, 1)$, and do not require the commonly assumed Lipschitzness of loss functions. We further derive optimization error bounds for the generalized SGDM, and combine them with our generalization analyses to obtain optimal excess population risk bounds for SGDM with both Polyak's and Nesterov's momentum.

cs.LG↗

arXiv2026

Learning to Control: The iUzawa-Net for Nonsmooth Optimal Control of Linear PDEs

We propose an optimization-informed deep neural network approach, named iUzawa-Net, aiming for the first solver that enables real-time solutions for a class of nonsmooth optimal control problems of linear partial differential equations (PDEs). The iUzawa-Net unrolls an inexact Uzawa method for saddle point problems, replacing classical preconditioners and PDE solvers with specifically designed learnable neural networks. We prove universal approximation properties and establish the asymptotic $\varepsilon$-optimality for the iUzawa-Net, and validate its promising numerical efficiency through nonsmooth elliptic and parabolic optimal control problems. Our techniques offer a versatile framework for designing and analyzing various optimization-informed deep learning approaches to optimal control and other PDE-constrained optimization problems. The proposed learning-to-control approach synergizes model-based optimization algorithms and data-driven deep learning techniques, inheriting the merits of both methodologies.

math.OC↗

arXiv2026

Stability and Generalization of Nonconvex Optimization with Heavy-Tailed Noise

The empirical evidence indicates that stochastic optimization with heavy-tailed gradient noise is more appropriate to characterize the training of machine learning models than that with standard bounded gradient variance noise. Most existing works on this phenomenon focus on the convergence of optimization errors, while the analysis for generalization bounds under the heavy-tailed gradient noise remains limited. In this paper, we develop a general framework for establishing generalization bounds under heavy-tailed noise. Specifically, we introduce a truncation argument to achieve the generalization error bound based on the algorithmic stability under the assumption of bounded $p$th centered moment with $p\in(1,2]$. Building on this framework, we further provide the stability and generalization analysis for several popular stochastic algorithms under heavy-tailed noise, including clipped and normalized stochastic gradient descent, as well as their mini-batch and momentum variants.

cs.LG↗

arXiv2025

Breaking quadrature exactness: A spectral method for the Allen--Cahn equation on spheres

We present a novel spectral method for the Allen-Cahn equation on spheres, eliminating the reliance on conventional quadrature exactness conditions. By replacing these conditions with a restricted isometry relation derived from Marcinkiewicz--Zygmund quadrature systems, our method achieves precise control over quadrature errors for polynomial integrands. This theoretical advancement enables the use of substantially more choices of quadrature points than classical spectral methods while maintaining rigorous error bounds. The proposed method requires only mild constraints on the polynomial degree of numerical solutions to establish both the maximum principle and energy stability, representing a considerable departure from existing techniques that depend on restrictive time stepping sizes, Lipschitz property of the nonlinear term, or $L^{\infty}$ boundedness of numerical solutions. Notably, our method permits time stepping sizes independent of the diffusion coefficient, making it suitable for long-time simulations. Inspired by the effective maximum principle proposed by Li (Ann. Appl. Math., 37(2): 131--290, 2021), we develop an almost sharp maximum principle that allows controllable deviation of numerical solutions from the sharp bound with large time stepping sizes. Furthermore, we prove that when the quadrature rule attains sufficient exactness, our method preserves energy stability and coincides mathematically with the Galerkin method. In addition, we propose an energy-stable mixed-quadrature scheme which works well even with randomly sampled initial condition data. Our numerical experiments on $\mathbb{S}^2$ validate the theoretical results about the energy stability and the almost sharp maximum principle.

math.NA↗

arXiv2025

GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models

Large Language Models (LLMs) face significant deployment challenges due to their substantial resource requirements. While low-bit quantized weights can reduce memory usage and improve inference efficiency, current hardware lacks native support for mixed-precision General Matrix Multiplication (mpGEMM), resulting in inefficient dequantization-based implementations. Moreover, uniform quantization methods often fail to capture weight distributions adequately, leading to performance degradation. We propose GANQ (GPU-Adaptive Non-Uniform Quantization), a layer-wise post-training non-uniform quantization framework optimized for hardware-efficient lookup table-based mpGEMM. GANQ achieves superior quantization performance by utilizing a training-free, GPU-adaptive optimization algorithm to efficiently reduce layer-wise quantization errors. Extensive experiments demonstrate GANQ's ability to reduce the perplexity gap from the FP16 baseline compared to state-of-the-art methods for both 3-bit and 4-bit quantization. Furthermore, when deployed on a single NVIDIA RTX 4090 GPU, GANQ's quantized models achieve up to 2.57$\times$ speedup over the baseline, advancing memory and inference efficiency in LLM deployment.

cs.LG↗

arXiv2025

BestServe: Serving Strategies with Optimal Goodput in Collocation and Disaggregation Architectures

Serving large language models (LLMs) to millions of users requires efficient resource allocation and parallelism strategies. It is a labor intensive trial-and-error process to find such a strategy. We present BestServe, a novel framework for ranking serving strategies by estimating goodput under various operating scenarios. Supporting both collocated and disaggregated architectures, BestServe leverages an inference simulator built on an adapted roofline model and CPU-GPU dispatch dynamics. Our framework determines the optimal strategy in minutes on a single standard CPU, eliminating the need for costly benchmarking, while achieving predictions within a $20\%$ error margin. It appeals to be practical for rapid deployment planning because of its lightweight design and strong extensibility.

cs.LG↗

arXiv2025

SPAP: Structured Pruning via Alternating Optimization and Penalty Methods

The deployment of large language models (LLMs) is often constrained by their substantial computational and memory demands. While structured pruning presents a viable approach by eliminating entire network components, existing methods suffer from performance degradation, reliance on heuristic metrics, or expensive finetuning. To address these challenges, we propose SPAP (Structured Pruning via Alternating Optimization and Penalty Methods), a novel and efficient structured pruning framework for LLMs grounded in optimization theory. SPAP formulates the pruning problem through a mixed-integer optimization model, employs a penalty method that effectively makes pruning decisions to minimize pruning errors, and introduces an alternating minimization algorithm tailored to the splittable problem structure for efficient weight updates and performance recovery. Extensive experiments on OPT, LLaMA-3/3.1/3.2, and Qwen2.5 models demonstrate SPAP's superiority over state-of-the-art methods, delivering linear inference speedups (1.29$\times$ at 30% sparsity) and proportional memory reductions. Our work offers a practical, optimization-driven solution for pruning LLMs while preserving model performance.

cs.LG↗

arXiv2025

An Operator Learning Approach to Nonsmooth Optimal Control of Nonlinear PDEs

Optimal control problems with nonsmooth objectives and nonlinear partial differential equation (PDE) constraints are challenging, mainly because of the underlying nonsmooth and nonconvex structures and the demanding computational cost for solving multiple high-dimensional and ill-conditioned systems after mesh-based discretization. To mitigate these challenges numerically, we propose an operator learning approach in combination with an effective primal-dual optimization idea which can decouple the treatment of the control and state variables so that each of the resulting iterations only requires solving two PDEs. Our main purpose is to construct neural surrogate models for the involved PDEs by operator learning, allowing the solution of a PDE to be obtained with only a forward pass of the neural network. The resulting algorithmic framework offers a hybrid approach that combines the efficiency and generalization of operator learning with the model-based nature and structure-friendly efficiency of primal-dual-based algorithms. The primal-dual-based operator learning approach offers numerical methods that are mesh-free, easy to implement, and adaptable to various optimal control problems with nonlinear PDEs. It is notable that the neural surrogate models can be reused across iterations and parameter settings, hence retraining of neural networks can be avoided and computational cost can be substantially alleviated. We affirmatively validate the efficiency of the primal-dual-based operator learning approach across a range of typical optimal control problems with nonlinear PDEs.

math.OC↗

arXiv2025

FASP: Fast and Accurate Structured Pruning of Large Language Models

The rapid increase in the size of large language models (LLMs) has significantly escalated their computational and memory demands, posing challenges for efficient deployment, especially on resource-constrained devices. Structured pruning has emerged as an effective model compression method that can reduce these demands while preserving performance. In this paper, we introduce FASP (Fast and Accurate Structured Pruning), a novel structured pruning framework for LLMs that emphasizes both speed and accuracy. FASP employs a distinctive pruning structure that interlinks sequential layers, allowing for the removal of columns in one layer while simultaneously eliminating corresponding rows in the preceding layer without incurring additional performance loss. The pruning metric, inspired by Wanda, is computationally efficient and effectively selects components to prune. Additionally, we propose a restoration mechanism that enhances model fidelity by adjusting the remaining weights post-pruning. We evaluate FASP on the OPT and LLaMA model families, demonstrating superior performance in terms of perplexity and accuracy on downstream tasks compared to state-of-the-art methods. Our approach achieves significant speed-ups, pruning models such as OPT-125M in 17 seconds and LLaMA-30B in 15 minutes on a single NVIDIA RTX 4090 GPU, making it a highly practical solution for optimizing LLMs.

cs.LG↗

arXiv2024

A Single-Loop Stochastic Proximal Quasi-Newton Method for Large-Scale Nonsmooth Convex Optimization

We propose a new stochastic proximal quasi-Newton method for minimizing the sum of two convex functions in the particular context that one of the functions is the average of a large number of smooth functions and the other one is nonsmooth. The new method integrates a simple single-loop SVRG (L-SVRG) technique for sampling the gradient and a stochastic limited-memory BFGS (L-BFGS) scheme for approximating the Hessian of the smooth function components. The globally linear convergence rate of the new method is proved under mild assumptions. It is also shown that the new method covers a proximal variant of the L-SVRG as a special case, and it allows for various generalizations through the integration with other variance reduction methods. For example, the L-SVRG can be replaced with the SAGA or SEGA in the proposed new method and thus other new stochastic proximal quasi-Newton methods with rigorously guaranteed convergence can be proposed accordingly. Moreover, we meticulously analyze the resulting nonsmooth subproblem at each iteration and utilize a compact representation of the L-BFGS matrix with the storage of some auxiliary matrices. As a result, we propose a very efficient and easily implementable semismooth Newton solver for solving the involved subproblems, whose arithmetic operation per iteration is merely order of $O(d)$, where d denotes the dimensionality of the problem. With this efficient inner solver, the new method performs well and its numerical efficiency is validated through extensive experiments on a regularized logistic regression problem.

math.OC↗

arXiv2024

A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models

Pruning is a critical strategy for compressing trained large language models (LLMs), aiming at substantial memory conservation and computational acceleration without compromising performance. However, existing pruning methods often necessitate inefficient retraining for billion-scale LLMs or rely on heuristic methods such as the optimal brain surgeon framework, which degrade performance. In this paper, we introduce FISTAPruner, the first post-training pruner based on convex optimization models and algorithms. Specifically, we propose a convex optimization model incorporating $\ell_1$ norm to induce sparsity and utilize the FISTA solver for optimization. FISTAPruner incorporates an intra-layer cumulative error correction mechanism and supports parallel pruning. We comprehensively evaluate FISTAPruner on models such as OPT, LLaMA, LLaMA-2, and LLaMA-3 with 125M to 70B parameters under unstructured and 2:4 semi-structured sparsity, demonstrating superior performance over existing state-of-the-art methods across various language benchmarks.

cs.LG↗

arXiv2024

The Hard-Constraint PINNs for Interface Optimal Control Problems

We show that the physics-informed neural networks (PINNs), in combination with some recently developed discontinuity capturing neural networks, can be applied to solve optimal control problems subject to partial differential equations (PDEs) with interfaces and some control constraints. The resulting algorithm is mesh-free and scalable to different PDEs, and it ensures the control constraints rigorously. Since the boundary and interface conditions, as well as the PDEs, are all treated as soft constraints by lumping them into a weighted loss function, it is necessary to learn them simultaneously and there is no guarantee that the boundary and interface conditions can be satisfied exactly. This immediately causes difficulties in tuning the weights in the corresponding loss function and training the neural networks. To tackle these difficulties and guarantee the numerical accuracy, we propose to impose the boundary and interface conditions as hard constraints in PINNs by developing a novel neural network architecture. The resulting hard-constraint PINNs approach guarantees that both the boundary and interface conditions can be satisfied exactly or with a high degree of accuracy, and they are decoupled from the learning of the PDEs. Its efficiency is promisingly validated by some elliptic and parabolic interface optimal control problems.

math.OC↗

arXiv2024

The ADMM-PINNs Algorithmic Framework for Nonsmooth PDE-Constrained Optimization: A Deep Learning Approach

We study the combination of the alternating direction method of multipliers (ADMM) with physics-informed neural networks (PINNs) for a general class of nonsmooth partial differential equation (PDE)-constrained optimization problems, where additional regularization can be employed for constraints on the control or design variables. The resulting ADMM-PINNs algorithmic framework substantially enlarges the applicable range of PINNs to nonsmooth cases of PDE-constrained optimization problems. The application of the ADMM makes it possible to untie the PDE constraints and the nonsmooth regularization terms for iterations. Accordingly, at each iteration, one of the resulting subproblems is a smooth PDE-constrained optimization which can be efficiently solved by PINNs, and the other is a simple nonsmooth optimization problem which usually has a closed-form solution or can be efficiently solved by various standard optimization algorithms or pre-trained neural networks. The ADMM-PINNs algorithmic framework does not require to solve PDEs repeatedly, and it is mesh-free, easy to implement, and scalable to different PDE settings. We validate the efficiency of the ADMM-PINNs algorithmic framework by different prototype applications, including inverse potential problems, source identification in elliptic equations, control constrained optimal control of the Burgers equation, and sparse optimal control of parabolic equations.

math.OC↗

arXiv2024

Coverage Axis++: Efficient Inner Point Selection for 3D Shape Skeletonization

We introduce Coverage Axis++, a novel and efficient approach to 3D shape skeletonization. The current state-of-the-art approaches for this task often rely on the watertightness of the input or suffer from substantial computational costs, thereby limiting their practicality. To address this challenge, Coverage Axis++ proposes a heuristic algorithm to select skeletal points, offering a high-accuracy approximation of the Medial Axis Transform (MAT) while significantly mitigating computational intensity for various shape representations. We introduce a simple yet effective strategy that considers shape coverage, uniformity, and centrality to derive skeletal points. The selection procedure enforces consistency with the shape structure while favoring the dominant medial balls, which thus introduces a compact underlying shape representation in terms of MAT. As a result, Coverage Axis++ allows for skeletonization for various shape representations (e.g., water-tight meshes, triangle soups, point clouds), specification of the number of skeletal points, few hyperparameters, and highly efficient computation with improved reconstruction accuracy. Extensive experiments across a wide range of 3D shapes validate the efficiency and effectiveness of Coverage Axis++. Our codes are available at https://github.com/Frank-ZY-Dou/Coverage_Axis.

cs.CV↗