arXiv subjects

Yanli Liu

Publications and source records attributed to Yanli Liu.

24 records · Page 2Linked to original sources

arXiv2019

Acceleration of Primal-Dual Methods by Preconditioning and Simple Subproblem Procedures

Primal-Dual Hybrid Gradient (PDHG) and Alternating Direction Method of Multipliers (ADMM) are two widely-used first-order optimization methods. They reduce a difficult problem to simple subproblems, so they are easy to implement and have many applications. As first-order methods, however, they are sensitive to problem conditions and can struggle to reach the desired accuracy. To improve their performance, researchers have proposed techniques such as diagonal preconditioning and inexact subproblems. This paper realizes additional speedup about one order of magnitude. Specifically, we choose non-diagonal preconditioners that are much more effective than diagonal ones. Because of this, we lose closed-form solutions to some subproblems, but we found simple procedures to replace them such as a few proximal-gradient iterations or a few epochs of proximal block-coordinate descent, which are in closed forms. We show global convergence while fixing the number of those steps in every outer iteration. Therefore, our method is reliable and straightforward. Our method opens the choices of preconditioners and maintains both low per-iteration cost and global convergence. Consequently, on several typical applications of primal-dual first-order methods, we obtain 4-95$\times$ speedup over the existing state-of-the-art.

math.OC↗

arXiv2019

Acceleration of SVRG and Katyusha X by Inexact Preconditioning

Empirical risk minimization is an important class of optimization problems with many popular machine learning applications, and stochastic variance reduction methods are popular choices for solving them. Among these methods, SVRG and Katyusha X (a Nesterov accelerated SVRG) achieve fast convergence without substantial memory requirement. In this paper, we propose to accelerate these two algorithms by \textit{inexact preconditioning}, the proposed methods employ \textit{fixed} preconditioners, although the subproblem in each epoch becomes harder, it suffices to apply \textit{fixed} number of simple subroutines to solve it inexactly, without losing the overall convergence. As a result, this inexact preconditioning strategy gives provably better iteration complexity and gradient complexity over SVRG and Katyusha X. We also allow each function in the finite sum to be nonconvex while the sum is strongly convex. In our numerical experiments, we observe an on average $8\times$ speedup on the number of iterations and $7\times$ speedup on runtime.

math.OC↗

arXiv2018

Breaking the Span Assumption Yields Fast Finite-Sum Minimization

In this paper, we show that SVRG and SARAH can be modified to be fundamentally faster than all of the other standard algorithms that minimize the sum of $n$ smooth functions, such as SAGA, SAG, SDCA, and SDCA without duality. Most finite sum algorithms follow what we call the "span assumption": Their updates are in the span of a sequence of component gradients chosen in a random IID fashion. In the big data regime, where the condition number $κ=\mathcal{O}(n)$, the span assumption prevents algorithms from converging to an approximate solution of accuracy $ε$ in less than $n\ln(1/ε)$ iterations. SVRG and SARAH do not follow the span assumption since they are updated with a hybrid of full-gradient and component-gradient information. We show that because of this, they can be up to $Ω(1+(\ln(n/κ))_+)$ times faster. In particular, to obtain an accuracy $ε= 1/n^α$ for $κ=n^β$ and $α,β\in(0,1)$, modified SVRG requires $\mathcal{O}(n)$ iterations, whereas algorithms that follow the span assumption require $\mathcal{O}(n\ln(n))$ iterations. Moreover, we present lower bound results that show this speedup is optimal, and provide analysis to help explain why this speedup exists. With the understanding that the span assumption is a point of weakness of finite sum algorithms, future work may purposefully exploit this to yield even faster algorithms in the big data regime.

math.OC↗

arXiv2018

An Envelope for Davis-Yin Splitting and Strict Saddle Point Avoidance

It is known that operator splitting methods based on Forward Backward Splitting (FBS), Douglas-Rachford Splitting (DRS), and Davis-Yin Splitting (DYS) decompose a difficult optimization problems into simpler subproblems under proper convexity and smoothness assumptions. In this paper, we identify an envelope (an objective function) whose gradient descent iteration under a variable metric coincides with DYS iteration. This result generalizes the Moreau envelope for proximal-point iteration and the envelopes for FBS and DRS iterations identified by Patrinos, Stella, and Themelis. Based on the new envelope and the Stable-Center Manifold Theorem, we further show that, when FBS or DRS iterations start from random points, they avoid all strict saddle points with probability one. This results extends the similar results by Lee et al. from gradient descent to splitting methods.

math.OC↗

arXiv2017

A New Use of Douglas-Rachford Splitting and ADMM for Identifying Infeasible, Unbounded, and Pathological Conic Programs

In this paper, we present a method for identifying infeasible, unbounded, and pathological conic programs based on Douglas-Rachford splitting, or equivalently ADMM. When an optimization program is infeasible, unbounded, or pathological, the iterates of Douglas-Rachford splitting diverge. Somewhat surprisingly, such divergent iterates still provide useful information, which our method uses for identification. In addition, for strongly infeasible problems the method produces a separating hyperplane and informs the user on how to minimally modify the given problem to achieve strong feasibility. As a first-order method, the proposed algorithm relies on simple subroutines, and therefore is simple to implement and has low per-iteration cost.

math.OC↗

arXiv2017

High-resolution reconstruction of cellular traction-force distributions: the role of physically motivated constraints and compressive regularization

We develop a method to reconstruct, from measured displacements of an underlying elastic substrate, the spatially dependent forces that cells or tissues impart on it. Given newly available high-resolution images of substrate displacements, it is desirable to be able to reconstruct small scale, compactly supported focal adhesions which are often localized and exist only within the footprint of a cell. In addition to the standard quadratic data mismatch terms that define least-squares fitting, we motivate a regularization term in the objective function that penalizes vectorial invariants of the reconstructed surface stress while preserving boundaries. We solve this inverse problem by providing a numerical method for setting up a discretized inverse problem that is solvable by standard convex optimization techniques. By minimizing the objective function subject to a number of important physically motivated constraints, we are able to efficiently reconstruct stress fields with localized structure from simulated and experimental substrate displacements. Our method incorporates the exact solution for the stress tensor accurate to first-order finite-differences and motivates the use of distance-based cut-offs for data inclusion and problem sparsification.

q-bio.QM↗