Convergence Rate Analysis for Monotone Accelerated Proximal Gradient Method

Abstract

We propose a monotone accelerated proximal gradient method for solving convex composite optimization problems, guaranteeing nonincreasing function values along the iterates -- a property that improves numerical stability and is not enjoyed by standard accelerated schemes. The method generalizes the Monotone FISTA algorithm of Beck and Teboulle. In the convex setting, we establish the optimal rate of $\mathcal{O}\left( \frac{1}{k^2} \right)$ and show that all weak subsequential limit points of the iterates are minimizers. In the strongly convex setting, we establish a linear rate $\mathcal{O}\left( \frac{1}{k^2}(1+\rho)^{-k} \right)$ with $\rho\sim \frac{\mu}{3L}$, without requiring knowledge of the strong convexity parameter -- roughly a five-fold improvement in the contraction constant over the best previously known rate for monotone methods, and more than 30% larger than that of the best known rate for non-monotone accelerated methods. Following a similar idea, we propose a variant of Nesterov's accelerated proximal method and establish a linear rate under strong convexity, which is the same as the one above, and is faster than the known results for non-monotone methods.