On the complexity of proximal gradient and proximal gradient-Newton-CG methods for $\ell_1$-regularized Optimization

Abstract

In this paper, we propose two second-order methods for solving the \(\ell_1\)-regularized composite optimization problem, which are developed based on two distinct definitions of approximate second-order stationary points. We introduce a hybrid proximal gradient and negative curvature method, as well as an adaptive hybrid proximal gradient-Newton-conjugate gradient (Newton-CG) method with negative curvature directions, to find a strong* approximate second-order stationary point and a weak approximate second-order stationary point for \(\ell_1\)-regularized optimization problems, respectively. We provide comprehensive analyses of the iteration complexity and operation complexity, measured in terms of gradient evaluations and Hessian-vector products. We demonstrate that the proximal gradient-Newton-CG method achieves the best-known iteration complexity for attaining the proposed weak approximate second-order stationary point, consistent with results for finding an approximate second-order stationary point in unconstrained optimization. Through a toy example, we show that our proposed methods can effectively escape from a first-order approximate stationary point. Numerical experiments on the \(\ell_1\)-regularized Student's \(t\)-regression problem validate the effectiveness of both methods.