arXiv subjects

Ya-xiang Yuan

Publications and source records attributed to Ya-xiang Yuan.

At least 19 recordsLinked to original sources

arXiv2026

Quotient geometry of tensor ring decomposition

Differential geometries derived from tensor decompositions have been extensively studied and provided the foundations for a variety of efficient numerical methods. Despite the practical success of the tensor ring (TR) decomposition, its intrinsic geometry remains less understood, primarily due to the underlying ring structure and the resulting nontrivial gauge invariance. We establish the quotient geometry and immersed-submanifold structure of TR decomposition by imposing full-rank conditions on all unfolding matrices of the core tensors and capturing the gauge invariance. The intrinsic ring structure of TR leads to an analysis that is substantially different from other tensor formats. Additionally, for the uniform TR decomposition, where all core tensors are identical and the manifold structure is known, we derive explicit parameterizations for the vertical and horizontal spaces, which enable Riemannian optimization. Numerical experiments validate the developed geometries via tensor ring completion tasks.

math.NA↗

arXiv2026

Structured Spectral Step-Sizes and Rebound-Aware Ordering for Gradient Methods

The performance of gradient methods depends critically on step-size selection. Spectral step-size design involves constructing candidates from finite history and ordering them during iteration. We address both through a finite-moment spectral framework. For candidate construction, we relate Huang--Dai--Liu determinant pencils, limited-memory steepest-descent Krylov--Ritz extraction, pseudo-memory moment realization, and Gu--Du recovery of spectral nodes and component weights. The framework distinguishes generalized-eigenvalue recoverability, finite-window implementability, and component-weight interpretability. Under explicit positivity, rank-one, regularity, and exact-recovery assumptions, these properties become compatible in a common finite-moment regime, with the long, short, and mean LMSD constructions as representatives. For ordering, we identify spectral rebound: a step targeting a lower spectral scale may reactivate previously reduced high-frequency components. Idealized one-target, block, and recomputed memory-$m$ models yield phase-count recurrences motivating a high-frequency-first strategy with spectral refreshes. We propose a rebound-aware spectral gradient method (RASG) that ranks Ritz candidates by frequency and estimated strength, and sets phase duration via a Ritz-uncertainty estimator. Two variants, RASG-KT and RASG-LMSD, use a Kato--Temple-inspired estimator and a long/short LMSD discrepancy. For strictly convex quadratics, global $R$-linear convergence holds under an admissibility safeguard. Experiments on ill-conditioned quadratic problems and two large-scale convex problems illustrate the behavior.

math.OC↗

arXiv2026

Lyapunov Functions and $R$-Linear Convergence for Quadratic Barzilai-Borwein Dynamics

The Barzilai--Borwein (BB) method is $R$-linearly convergent on strongly convex quadratics, although neither the objective value nor the gradient norm is generally monotone. We construct an explicit current-point Lyapunov function for the quadratic dynamics after the warm-up step, assuming that both endpoints of the initial active spectrum survive this step. If $a<b$ are these endpoints and $P_a,P_b$ are the corresponding spectral projectors, then \[ \overline{f}(x) :=\lVert P_a\nabla f(x)\rVert^{\frac{2b}{a+b}} \lVert P_b\nabla f(x)\rVert^{\frac{2a}{a+b}} \] satisfies, for BB1, BB2, and every fixed positive weighted delayed Rayleigh rule, \[ \overline{f}(x_{k+1}) =\left(\frac{b-a}{b+a}\right)^2 e^{-2D_{a,b}(τ_k)}\overline{f}(x_k), \qquad D_{a,b}(τ_k)\ge0, \] where $τ_k=1/α_k$ is the reciprocal step. The function is obtained by denormalizing the endpoint coboundary certificate in the sharp-rate analysis of Yang and Yuan. We prove that the endpoint exponents are the unique minimax choice among endpoint monomials and illustrate the Lyapunov law on a nonmonotone BB1 trajectory. Using this law, we also give a proof of $R$-linear convergence of the method in finite dimensions.

math.OC↗

arXiv2026

$R$-Linear Convergence of Barzilai-Borwein Methods on Strictly Convex Quadratics via Closed-Cone Homogeneous Dynamics

We give an $R$-linear convergence proof for BB1, BB2, and fixed positive spectral-weight variants on finite-dimensional strictly convex quadratics. The delayed recurrence is written as a first-order system on the closed cone of compatible consecutive-gradient pairs. Its transition is continuous and positively homogeneous of degree one, including at finite termination. A compactness theorem shows that pointwise convergence of every cone orbit implies a uniform finite-step contraction and hence a global $R$-linear estimate. By a realization lemma, every compatible state initiates a BB1 trajectory, so Raydan's theorem yields pointwise stability on the entire cone. Finally, the transformation $g\mapsto W^{1/2}g$ conjugates every fixed positive spectral-weight rule to BB1. Consequently, BB1, BB2, and all such weighted rules have the same homogeneous growth radius.

math.OC↗

arXiv2026

Global and Unconditional $R$-Linear Convergence of Rank-Compressed Weighted LMSD Sweeps for Strictly Convex Quadratics

We study limited memory steepest descent (LMSD) with exact algebraic rank compression for strictly convex quadratic optimization. At each cycle, the method restricts the gradient history to its column space and applies the reciprocals of all Ritz values of the compressed projection in the next sweep. If the block-start gradient has at most $p$ active distinct eigenvalues, the delayed sweep terminates finitely. For nonterminating trajectories, a determinant and Cauchy--Binet formula for the complete-sweep polynomial yields global convergence without a full-rank history or a run-wise normalized-conditioning bound. Consecutive sweep endpoints define a continuous positively homogeneous map on a closed cone of compatible states. Compactness then gives an $R$-linear endpoint estimate, uniform over compatible initial states, with constants depending only on $H$ and $p$; the estimate extends to inner gradients, iterate errors, and objective gaps. Every fixed positive spectral weight $W=ω(H)$ is linearly conjugate to the standard method. The weighted methods therefore share its decay factor, while the Euclidean-norm prefactor is increased by at most $\sqrt{κ_2(W)}$. This includes harmonic-Ritz LMSD, fixed power weights, and the delayed BB1 and BB2 recurrences.

math.OC↗

arXiv2026

A Decomposed Bilevel Search for Variable-Metric Proximal Gradient Methods

Variable-metric proximal methods accelerate composite convex optimization, but the scaled proximal map induced by a quasi-Newton metric rarely has a closed form. We develop \emph{Decomposed Bilevel Search} (DBS), based on a diagonal-plus-rank-one factor $X=D+uv^\top$ whose diagonal scaling satisfies a weak secant equation. The induced inverse metric $B^{-1}=XX^\top$ recovers zero-memory DFP/BFGS-type Broyden members, while the factor form reduces each scaled proximal step to a two-dimensional monotone residual system. Each residual evaluation requires one diagonal-metric proximal map, and a certified bilevel solve reaches target accuracy $ε$ in $\mathcal O((d+T_p)\log^2(1/ε))$ work, where $T_p$ is the cost of that proximal map. Under strong convexity, the outer method converges linearly with exact and inexact inner solves. In the scalar specialization $D=αI$, the oracle uses only ordinary proximal evaluations of the regularizer and no generalized Jacobian or active-set information. Experiments on ordered-weighted $\ell_1$ (SLOPE/OWL) and group-lasso logistic regression evaluate both the inner oracle and the full outer method. The oracle solves high-dimensional SLOPE scaled proximal subproblems of condition number up to $4\times10^6$ using a few hundred ordinary proximal evaluations. In SLOPE outer benchmarks, the warm-started oracle keeps the scaled-proximal overhead controlled and DBS reaches stringent targets with far fewer gradient evaluations than Lipschitz-normalized FISTA; on \texttt{real-sim} this becomes a clear target-time advantage. On group-lasso logistic regression, DBS is reliable on synthetic correlated instances and fastest on a real-data grouped-copy instance.

math.OC↗

arXiv2026

Optimization over the intersection of manifolds

Optimization over the intersection of two manifolds arises in a broad range of applications, but is hindered by the coupled geometry of the feasible region. In this paper, we prove that the regularities -- clean intersection and intrinsic transversality -- are equivalent, which yields a tractable projection onto the tangent space of the intersection. Therefore, we propose a geometric method that employs a retraction on only one manifold and updates the iterate along two orthogonal directions. Specifically, the iterates stay on one manifold, and the two directions are responsible for asymptotically approaching the other manifold and decreasing the objective function, respectively. Under intrinsic transversality, we derive the convergence rate for both the feasibility and optimality measures, and show that every accumulation point is first-order stationary. Numerical experiments on problems stemming from sparse and low-rank optimization, including fitting spherical data, approximating hyperbolic embeddings on real data, and computing compressed modes, demonstrate the effectiveness of the proposed method.

math.OC↗

arXiv2026

The Error in Multivariate Linear Extrapolation with Applications to Derivative-Free Optimization

We study in this paper the function approximation error of multivariate linear extrapolation. The sharp error bound of linear interpolation already exists in the literature. However, linear extrapolation is used far more often in applications such as derivative-free optimization, while its error is not well-studied. We introduce in this paper a method to numerically compute the sharp bound on the error, and then present several analytical bounds along with the conditions under which they are sharp. We also provide a complexity analysis of a basic simplicial search method to illustrate an application of these error bounds in derivative-free optimization. All results are under the assumptions that the function being interpolated has Lipschitz continuous gradient and is interpolated on an affinely independent sample set.

math.OC↗

arXiv2026

First-order methods on bounded-rank tensors converging to stationary points

Provably finding stationary points on bounded-rank tensors turns out to be an open problem [E. Levin, J. Kileel, and N. Boumal, Math. Program., 199 (2023), pp. 831--864] due to the inherent non-smoothness of the set of bounded-rank tensors. In contrast with bounded-rank matrices, tensors where some but not all modes are of full rank render essential difficulties in developing provable first-order methods. We resolve this problem by proposing two first-order methods with guaranteed convergence to stationary points. Specifically, we revisit the variational geometry of bounded-rank tensors and explicitly characterize its normal cones. Moreover, we propose gradient-related approximate projection methods that are provable to find stationary points, where the decisive ingredients are gradient-related vectors from tangent cones, line search along approximate projections, and rank-decreasing mechanisms near rank-deficient points. Numerical experiments on tensor completion validate that the proposed methods converge to stationary points across various rank parameters.

math.OC↗

arXiv2025

Variational analysis of determinantal varieties

Determinantal varieties -- the sets of bounded-rank matrices or tensors -- have attracted growing interest in low-rank optimization. The tangent cone to low-rank sets is widely studied and underpins a range of geometric methods. The second-order geometry, which encodes curvature information, is more intricate. In this work, we develop a unified framework to derive explicit formulas for both first- and second-order tangent sets to various low-rank sets, including low-rank matrices, tensors, symmetric matrices, and positive semidefinite matrices. The framework also accommodates the intersection of a low-rank set and another set satisfying mild assumptions, thereby yielding a tangent intersection rule. Through the lens of tangent sets, we establish a necessary and sufficient condition under which a nonsmooth problem and its smooth parameterization share equivalent second-order stationary points. Moreover, we exploit tangent sets to characterize optimality conditions for low-rank optimization and prove that verifying second-order optimality is NP-hard. In a separate line of analysis, we investigate variational geometry of the graph of the normal cone to matrix varieties, deriving the explicit Bouligand tangent cone, Fréchet and Mordukhovich normal cones to the graph. These results are further applied to develop optimality conditions for low-rank bilevel programs.

math.OC↗

arXiv2025

Normalized tensor train decomposition

Tensors with unit Frobenius norm are fundamental objects in many fields, including scientific computing and quantum physics, which are able to represent normalized eigenvectors and pure quantum states. While the tensor train decomposition provides a powerful low-rank format for tackling high-dimensional problems, it does not intrinsically enforce the unit-norm constraint. To address this, we introduce the normalized tensor train (NTT) decomposition, which aims to approximate a tensor by unit-norm tensors in tensor train format. The low-rank structure of NTT decomposition not only saves storage and computational cost but also preserves the underlying unit-norm structure. We prove that the set of fixed-rank NTT tensors forms a smooth manifold, and the corresponding Riemannian geometry is derived, paving the way for geometric methods. We propose NTT-based methods for low-rank tensor recovery, high-dimensional eigenvalue problem, estimation of stabilizer rank, and calculation of the minimum output Rényi 2-entropy of quantum channels. Numerical experiments demonstrate the superior efficiency and scalability of the proposed NTT-based methods.

math.NA↗

arXiv2025

A space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints

Imposing additional constraints on low-rank optimization has garnered growing interest. However, the geometry of coupled constraints hampers the well-developed low-rank structure and makes the problem intricate. To this end, we propose a space-decoupling framework for optimization on bounded-rank matrices with orthogonally invariant constraints. The "space-decoupling" is reflected in several ways. We show that the tangent cone of coupled constraints is the intersection of tangent cones of each constraint. Moreover, we decouple the intertwined bounded-rank and orthogonally invariant constraints into two spaces, leading to optimization on a smooth manifold. Implementing Riemannian algorithms on this manifold is painless as long as the geometry of additional constraints is known. In addition, we unveil the equivalence between the reformulated problem and the original problem. Numerical experiments on real-world applications -- spherical data fitting, graph similarity measuring, low-rank SDP, model reduction of Markov processes, reinforcement learning, and deep learning -- validate the superiority of the proposed framework.

math.OC↗

arXiv2025

Convergent Proximal Multiblock ADMM for Nonconvex Dynamics-Constrained Optimization

This paper proposes a provably convergent multiblock ADMM for nonconvex optimization with nonlinear dynamics constraints, overcoming the divergence issue in classical extensions. We consider a class of optimization problems that arise from discretization of dynamics-constrained variational problems that are optimization problems for a functional constrained by time-dependent ODEs or PDEs. This is a family of $n$-sum nonconvex optimization problems with nonlinear constraints. We study the convergence properties of the proximal alternating direction method of multipliers (proximal ADMM) applied to those problems. Taking the advantage of the special problem structure, we show that under local Lipschitz and local $L$-smooth conditions, the sequence generated by the proximal ADMM is bounded and all accumulation points are KKT points. Based on our analysis, we also design a procedure to determine the penalty parameters $ρ_i$ and the proximal parameters $η_i$. We further prove that among all the subsequences that converge, the fast one converges at the rate of $o(1/k)$. The numerical experiments are performed on 4D variational data assimilation problems and as the solver of implicit schemes for stiff problems. The proposed proximal ADMM has more stable performance than gradient-based methods. We discuss the implementation to solve the subproblems, a new way to solve the implicit schemes, and the advantages of the proposed algorithm.

math.OC↗

arXiv2025

Symplectic Extra-gradient Type Method for Solving General Non-monotone Inclusion Problem

In recent years, accelerated extra-gradient methods have attracted much attention by researchers, for solving monotone inclusion problems. A limitation of most current accelerated extra-gradient methods lies in their direct utilization of the initial point, which can potentially decelerate numerical convergence rate. In this work, we present a new accelerated extra-gradient method, by utilizing the symplectic acceleration technique. We establish the inverse of quadratic convergence rate by employing the Lyapunov function technique. Also, we demonstrate a faster inverse of quadratic convergence rate alongside its weak convergence property under stronger assumptions. To improve practical efficiency, we introduce a line search technique for our symplectic extra-gradient method. Theoretically, we prove the convergence of the symplectic extra-gradient method with line search. Numerical tests show that this adaptation exhibits faster convergence rates in practice compared to several existing extra-gradient type methods.

math.OC↗

arXiv2025

Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity

Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order information, thus circumventing the assumption of lower-level convexity. This, remarkably, distinguishes our development of hyper-gradient from the general AID-based bilevel frameworks since we take advantage of the specific structure of RL problems. Moreover, we design both model-based and model-free bilevel reinforcement learning algorithms, facilitated by access to the fully first-order hyper-gradient. Both algorithms enjoy the convergence rate $O(ε^{-1})$. To extend the applicability, a stochastic version of the model-free algorithm is proposed, along with results on its iteration and sample complexity. In addition, numerical experiments demonstrate that the hyper-gradient indeed serves as an integration of exploitation and exploration.

math.OC↗

arXiv2025

LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace

Bilevel optimization, with broad applications in machine learning, has an intricate hierarchical structure. Gradient-based methods have emerged as a common approach to large-scale bilevel problems. However, the computation of the hyper-gradient, which involves a Hessian inverse vector product, confines the efficiency and is regarded as a bottleneck. To circumvent the inverse, we construct a sequence of low-dimensional approximate Krylov subspaces with the aid of the Lanczos process. As a result, the constructed subspace is able to dynamically and incrementally approximate the Hessian inverse vector product with less effort and thus leads to a favorable estimate of the hyper-gradient. Moreover, we propose a provable subspace-based framework for bilevel problems where one central step is to solve a small-size tridiagonal linear system. To the best of our knowledge, this is the first time that subspace techniques are incorporated into bilevel optimization. This successful trial not only enjoys $\mathcal{O}(ε^{-1})$ convergence rate but also demonstrates efficiency in a synthetic problem and two deep learning tasks.

math.OC↗

arXiv2025

Optimization on product manifolds under a preconditioned metric

Since optimization on Riemannian manifolds relies on the chosen metric, it is appealing to know that how the performance of a Riemannian optimization method varies with different metrics and how to exquisitely construct a metric such that a method can be accelerated. To this end, we propose a general framework for optimization problems on product manifolds endowed with a preconditioned metric, and we develop Riemannian methods under this metric. Generally, the metric is constructed by an operator that aims to approximate the diagonal blocks of the Riemannian Hessian of the cost function. We propose three specific approaches to design the operator: exact block diagonal preconditioning, left and right preconditioning, and Gauss--Newton type preconditioning. Specifically, we tailor new preconditioned metrics and adapt the proposed Riemannian methods to the canonical correlation analysis and the truncated singular value decomposition problems, which provably accelerate the Riemannian methods. Additionally, we adopt the Gauss--Newton type preconditioning to solve the tensor ring completion problem. Numerical results among these applications verify that a delicate metric does accelerate the Riemannian optimization methods.

math.OC↗

arXiv2024

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

The proximal point algorithm plays a central role in non-smooth convex programming. The Augmented Lagrangian Method, one of the most famous optimization algorithms, has been found to be closely related to the proximal point algorithm. Due to its importance, accelerated variants of the proximal point algorithm have received considerable attention. In this paper, we first study an Ordinary Differential Equation (ODE) system, which provides valuable insights into proving the convergence rate of the desired algorithm. Using the Lyapunov function technique, we establish the convergence rate of the ODE system. Next, we apply the Symplectic Euler Method to discretize the ODE system to derive a new proximal point algorithm, called the Symplectic Proximal Point Algorithm (SPPA). By utilizing the proof techniques developed for the ODE system, we demonstrate the convergence rate of the SPPA. Additionally, it is shown that existing accelerated proximal point algorithm can be considered a special case of the SPPA in a specific manner. Furthermore, under several additional assumptions, we prove that the SPPA exhibits a finer convergence rate.

math.OC↗