arXiv subjects
Publications and source records attributed to Ya-xiang Yuan.
In this paper, we prove that the global version of the $Ł$ojasiewicz gradient inequality holds for quadratic sphere constrained optimization problem with exponent $θ=\frac{3}{4}$. An example from Ting Kei Pong shows that $θ=\frac{3}{4}$ is tight. This is the first $Ł$ojasiewicz gradient inequality established for the sphere constrained optimization problem with a linear term.
We investigate the stochastic gradient descent (SGD) method where the step size lies within a banded region instead of being given by a fixed formula. The optimal convergence rate under mild conditions and large initial step size is proved. Our analysis provides comparable theoretical error bounds for SGD associated with a variety of step sizes. In addition, the convergence rates for some existing step size strategies, e.g., triangular policy and cosine-wave, can be revealed by our analytical framework under the boundary constraints. The bandwidth-based step size provides efficient and flexible step size selection in optimization. We also propose a $1/t$ up-down policy and give several non-monotonic step sizes. Numerical experiments demonstrate the efficiency and significant potential of the bandwidth-based step-size in many applications.
We present a stochastic trust-region model-based framework in which its radius is related to the probabilistic models. Especially, we propose a specific algorithm, termed STRME, in which the trust-region radius depends linearly on the latest model gradient. The complexity of STRME method in non-convex, convex and strongly convex settings has all been analyzed, which matches the existing algorithms based on probabilistic properties. In addition, several numerical experiments are carried out to reveal the benefits of the proposed methods compared to the existing stochastic trust-region methods and other relevant stochastic gradient methods.
To construct a parallel approach for solving optimization problems with orthogonality constraints is usually regarded as an extremely difficult mission, due to the low scalability of the orthonormalization procedure. However, such demand is particularly huge in some application areas such as materials computation. In this paper, we propose a proximal linearized augmented Lagrangian algorithm (PLAM) for solving optimization problems with orthogonality constraints. Unlike the classical augmented Lagrangian methods, in our algorithm, the prime variables are updated by minimizing a proximal linearized approximation of the augmented Lagrangian function, meanwhile the dual variables are updated by a closed-form expression which holds at any first-order stationary point. The orthonormalization procedure is only invoked once at the last step of the above mentioned algorithm if high-precision feasibility is needed. Consequently, the main parts of the proposed algorithm can be parallelized naturally. We establish global subsequence convergence, worst-case complexity and local convergence rate for PLAM under some mild assumptions. To reduce the sensitivity of the penalty parameter, we put forward a modification of PLAM, which is called parallelizable column-wise block minimization of PLAM (PCAL). Numerical experiments in serial illustrate that the novel updating rule for the Lagrangian multipliers significantly accelerates the convergence of PLAM and makes it comparable with the existent feasible solvers for optimization problems with orthogonality constraints, and the performance of PCAL does not highly rely on the choice of the penalty parameter. Numerical experiments under parallel environment demonstrate that PCAL attains good performance and high scalability in solving discretized Kohn-Sham total energy minimization problems.
In this paper, we propose a class of penalty methods with stochastic approximation for solving stochastic nonlinear programming problems. We assume that only noisy gradients or function values of the objective function are available via calls to a stochastic first-order or zeroth-order oracle. In each iteration of the proposed methods, we minimize an exact penalty function which is nonsmooth and nonconvex with only stochastic first-order or zeroth-order information available. Stochastic approximation algorithms are presented for solving this particular subproblem. The worst-case complexity of calls to the stochastic first-order (or zeroth-order) oracle for the proposed penalty methods for obtaining an $ε$-stochastic critical point is analyzed.
The $p$-regularized subproblem (p-RS) is a regularisation technique in computing a Newton-like step for unconstrained optimization, which globally minimizes a local quadratic approximation of the objective function while incorporating with a weighted regularisation term $\fracσ{p} \|x\|^p$. The global solution of the $p$-regularized subproblem for $p=3$, also known as the cubic regularization, has been characterized in literature. In this paper, we resolve both the global and the local non-global minimizers of (p-RS) for $p>2$ with necessary and sufficient optimality conditions. Moreover, we prove a parallel result of Mart\'ınez \cite{Mar} that the (p-RS) for $p>2$, analogous to the trust region subproblem, can have at most one local non-global minimizer. When the (p-RS) is subject to a fixed number $m$ additional linear inequality constraints, we show that the uniqueness of the local solution of the (p-RS) (if exists at all), especially for $p=4$, can be applied to solve such an extension in polynomial time.