Accelerated Implicit GDA Schemes: Theoretical Guarantees and Application to Proximal Augmented Lagrangian Methods

Abstract

Convex optimization problems with linear equality constraints arise ubiquitously in scientific computing, machine learning, and control theory. Classical Krylov methods are effective but rely on problem-specific preconditioners and high memory. Conversely, gradient-based methods like the augmented Lagrangian method (ALM) avoid these issues yet suffer from slow outer iterations. Developing accelerated outer-iteration schemes, therefore, remains a critical research objective. In this study, we demonstrate that incorporating a proximal operation into the augmented Lagrangian framework yields the proximal ALM, where the outer iteration is equivalent to an implicit gradient descent-ascent (GDA) scheme. We further establish that this equivalence extends naturally to the setting of variable step sizes. Through Lyapunov analysis, we show that the underlying potential function must be shifted from the conventional objective gap to a variational inequality measure, signaling a shift in perspective from pure convex optimization to minimax optimization. Motivated by these observations, we first develop an implicit GDA scheme with variable step sizes based on a continuous-time ODE framework, which achieves an $o(1/k)$ last-iterate convergence rate for both the primal-dual objective gap and the gradient norm. Building upon a second-order ODE framework, we then propose a family of Nesterov-type implicit GDA schemes parameterized by $r \geq 0$, which achieves an $o(1/k^{r+1})$ last-iterate convergence rate for the primal-dual objective gap. Furthermore, specializing the second-order ODE formulation to the case $r=0$, we derive a corresponding explicit GDA scheme and prove an $o(1/k)$ last-iterate convergence rate for the primal-dual objective gap. Finally, we present several numerical experiments to validate these theoretical results and demonstrate the effectiveness of the proposed methods.