OFF (Proximal) Newton-type Methods with Inexact Derivatives for Unconstrained Optimization

Abstract

In this paper, we propose objective-function-free (OFF) variants of the proximal Newton method for nonconvex composite optimization problems and the regularized Newton method for unconstrained optimization problems, respectively, using inexact gradients and Hessians. Theoretical analyses verify that the global and local convergence rates of the proposed algorithms remain consistent with those attained under exact evaluations of the objective function and derivatives. To validate the practical applicability of the theoretical assumptions, a lazy gradient strategy is adopted to provide a verifiable scheme for satisfying the accuracy criteria of approximate gradients. For finite-sum optimization problems, an adaptive sampling strategy is further developed to eliminate the circular dependency between sample size and gradient estimation. The proposed algorithm is proven to achieve locally superlinear and quadratic convergence in expectation. Furthermore, we present an OFF regularized Newton and negative curvature algorithm, which uses inexact derivatives to locate approximate second-order stationary points in unconstrained optimization. The worst-case iteration and (sample) operation complexity of the developed algorithm is consistent with the optimal results in the existing literature.