Efficient First Order Method for Saddle Point Problems with Higher Order Smoothness

Abstract

This paper studies the complexity of finding approximate stationary points for the smooth nonconvex-strongly-concave (NC-SC) saddle point problem: $\min_x\max_yf(x,y)$. Under the standard first-order smoothness conditions where $f$ is $\ell$-smooth in both arguments and $\mu_y$-strongly concave in $y$, existing literature shows that the optimal complexity for first-order methods to obtain an $\epsilon$-stationary point is $\tilde{O}\big(\sqrt{\kappa_y}\ell\epsilon^{-2}\big)$, where $\kappa_y=\ell/\mu_y$ is the condition number. However, when $\Phi(x):=\max_y f(x,y)$ has $L_2$-Lipschitz continuous Hessian in addition, we derive a first-order algorithm with an $\tilde{O}\big(\sqrt{\kappa_y}\ell^{1/2}L_2^{1/4}\epsilon^{-7/4}\big)$ complexity by designing an accelerated proximal point algorithm enhanced with the "Convex Until Proven Guilty" technique. Moreover, an improved $\Omega\big(\sqrt{\kappa_y}\ell^{3/7}L_2^{2/7}\epsilon^{-12/7}\big)$ lower bound for first-order method is also derived for sufficiently small $\epsilon$. As a result, given the second-order smoothness of the problem, the complexity of our method improves the state-of-the-art result by a factor of $\tilde{O}\big(\big(\frac{\ell^2}{L_2\epsilon}\big)^{1/4}\big)$, while almost matching the lower bound except for a small $\tilde{O}\big(\big(\frac{\ell^2}{L_2\epsilon}\big)^{1/28}\big)$ factor.