arXiv subjects
Publications and source records attributed to C. H. Jeffrey Pang.
We show that the averaged consensus algorithm on directed graphs with unreliable communications by Bof-Carli-Schenato has a dual optimization interpretation, which could be extended to the case of distributed optimization. We report on our numerical simulations for the distributed optimization algorithm for smooth and nonsmooth functions.
We show the linear convergence of Dykstra's algorithm for sets intersecting in a manner slightly stronger than the usual constraint qualifications.
Consider the setting where each vertex of a graph has a function, and communications can only occur between vertices connected by an edge. We wish to minimize the sum of these functions. For the case when each function is the sum of a strongly convex quadratic and a convex function, we propose a distributed version of Dykstra's algorithm. The computations to optimize the dual objective function can run asynchronously without a global clock, and in a distributed manner without a central controller. Convergence to the primal minimizer is deterministic instead of being probabilistic, and is guaranteed as long as in each cycle, the edges where two-way communications occur connects all vertices. We also look at an accelerated algorithm, and an algorithm for the case when the functions on the nodes are not strongly convex.
This work builds on our recent work on a distributed optimization algorithm for graphs with directed unreliable communications. We show its linear convergence when we take either the proximal of each function or an affine minorant for when the function is smooth.
We investigate the convergence rate of the distributed Dykstra's algorithm when some of the sets are defined as the level sets of convex functions. We carry out numerical experiments to compare with the theoretical results obtained.
We described a decentralized distributed deterministic asynchronous Dykstra's algorithm that allows for time-varying graphs in an earlier paper. In this paper, we show how to incorporate subdifferentiable functions into the framework using a step similar to the bundle method. We point out that our algorithm also allows for partial data communications. We discuss a standard step for treating the composition of a convex and linear function.
In two earlier papers, we designed a distributed deterministic asynchronous algorithm for minimizing the sum of subdifferentiable and proximable functions and a regularizing quadratic on time-varying graphs based on Dykstra's algorithm, or block coordinate dual ascent. Each node in the distributed optimization problem is the sum of a known regularizing quadratic and a function to be minimized. In this paper, we prove sublinear convergence rates for the general algorithm, and a linear rate of convergence if the function on each node is smooth with Lipschitz gradient.
We show that Dykstra's splitting for projecting onto the intersection of convex sets can be extended to minimize the sum of convex functions and a regularizing quadratic. We give conditions for which convergence to the primal minimizer holds so that more than one convex function can be minimized at a time, the convex functions are not necessarily sampled in a cyclic manner, and the SHQP strategy for problems involving the intersection of more than one convex set can be applied. When the sum does not involve the regularizing quadratic, we discuss an approximate proximal point method combined with Dykstra's splitting to minimize this sum.
We show a general framework of parallelizing Dykstra splitting that includes the classical Dykstra's algorithm and the product space formulation as special cases, and prove their convergence. The key idea is to split up the function whose conjugate takes in the sum of all dual variables in the dual formulation.
We show that an almost cyclic (or generalized Gauss- Seidel) Dykstra's algorithm which incorporates the SHQP (supporting halfspace- quadratic programming) strategy can achieve nonasymptotic and asymptotic linear convergence for polyhedral problems.
We consider the best approximation problem (BAP) of projecting a point onto the intersection of a number of convex sets. It is known that Dykstra's algorithm is alternating minimization on the dual problem. We extend Dykstra's algorithm so that it can be enhanced by the SHQP strategy of using quadratic programming to project onto the intersection of supporting halfspaces generated by earlier projection operations. By looking at a structured alternating minimization problem, we show the convergence rate of Dykstra's algorithm when reasonable conditions are imposed to guarantee a dual minimizer. We also establish convergence of using a warmstart iterate for Dykstra's algorithm, show how all the results for the Dykstra's algorithm can be carried over to the simultaneous Dykstra's algorithm, and discuss a different way of incorporating the SHQP strategy. Lastly, we show that the dual of the best approximation problem can have an O(1/k^2) accelerated algorithm that also incorporates the SHQP strategy.
The problem of finding a point in the intersection of closed sets can be solved by the method of alternating projections and its variants. It was shown in earlier papers that for convex sets, the strategy of using quadratic programming (QP) to project onto the intersection of supporting halfspaces generated earlier by the projection process can lead to an algorithm that converges multiple-term superlinearly. The main contributions of this paper are to show that this strategy can be effective for super-regular sets, which are structured nonconvex sets introduced by Lewis, Luke and Malick. Manifolds should be approximated by hyperplanes rather than halfspaces. We prove the linear convergence of this strategy, followed by proving that superlinear and quadratic convergence can be obtained when the problem is similar to the setting of the Newton method. We also show an algorithm that converges at an arbitrarily fast linear rate if halfspaces from older iterations are used to construct the QP.
We propose first order algorithms for convex optimization problems where the feasible set is described by a large number of convex inequalities that is to be explored by subgradient projections. The first algorithm is an adaptation of a subgradient algorithm, and has convergence rate $1/\sqrt{k}$. The second algorithm has convergence rate 1/k when (1) one has linear metric inequality in the feasible set, (2) the objective function is strongly convex, differentiable and has Lipschitz gradient, and (3) it is easy to optimize the objective function on the intersection of two halfspaces. This second algorithm generalizes Haugazeau's algorithm. The third algorithm adapts the second algorithm when condition (3) is dropped. We give examples to show that the second algorithm performs poorly when the objective function is not strongly convex, or when the linear metric inequality is absent.
Abstract. The Set Intersection Problem (SIP) is the problem of finding a point in the intersection of convex sets. This problem is typically solved by the method of alternating projections. To accelerate the convergence, the idea of using Quadratic Programming (QP) to project a point onto the intersection of halfspaces generated by the projection process was discussed in earlier papers. This paper looks at how one can integrate projection algorithms together with an active set QP algorithm. As a byproduct of our analysis, we show how to accelerate an SIP algorithm involving box constraints, and how to extend a version of the Algebraic Reconstruction Technique (ART) while preserving finite convergence. Lastly, the warmstart property of active set QP algorithms is a valuable property for the problem of projecting onto the intersection of convex sets.
A known first order method to find a feasible solution to a conic problem is an adapted von Neumann algorithm. We improve the distance reduction step there by projecting onto the convex hull of previously generated points using a primal active set quadratic programming (QP) algorithm. The convergence theory is improved when the QPs are as large as possible. For problems in R^2, we analyze our algorithm by epigraphs and the monotonicity of subdifferentials. Logically, the larger the set to project onto, the better the performance per iteration, and this is indeed seen in our numerical experiments.
We extend Fukushima's result on the finite convergence of an algorithm for the global convex feasibility problem to the local nonconvex case.
The von Neumann-Halperin method of alternating projections converges strongly to the projection of a given point onto the intersection of finitely many closed affine subspaces. We propose acceleration schemes making use of two ideas: Firstly, each projection onto an affine subspace identifies a hyperplane of codimension 1 containing the intersection, and secondly, it is easy to project onto a finite intersection of such hyperplanes. We give conditions for which our accelerations converge strongly. Finally, we perform numerical experiments to show that these accelerations perform well for a matrix model updating problem.
This paper improves the algorithms based on supporting halfspaces and quadratic programming for convex set intersection problems in our earlier paper in several directions. First, we give conditions so that much smaller quadratic programs (QPs) and approximate projections arising from partially solving the QPs are sufficient for multiple-term superlinear convergence for nonsmooth problems. Second, we identify additional regularity, which we call the second order supporting hyperplane property (SOSH), that gives multiple-term quadratic convergence. Third, we show that these fast convergence results carry over for the convex inequality problem. Fourth, we show that infeasibility can be detected in finitely many operations. Lastly, we explain how we can use the dual active set QP algorithm of Goldfarb and Idnani to get useful iterates by solving the QPs partially, overcoming the problem of solving large QPs in our algorithms.