The method of ellipcenters with momentum and relaxation for convex quadratic minimization

Abstract

The method of ellipcenters (ME) is a recent technique developed for unconstrained minimization. Its iteration relies only on first order information and consists of building a suitable two-dimensional ellipse that tries to capture intrinsic ill-conditioning of the problem. The center of the ellipse is taken as the next iterate, which justifies the name of the method. ME was already shown to converge linearly when the objective function is smooth and strongly convex. The special case when the objective is quadratic was studied in the first paper on ME and is also the subject of our work here. In this paper, we propose two variants of ME: RelaxME which adds in ME an additional relaxation step at each iteration and MomME which embeds momentum in ME. We prove linear convergence of RelaxME and MomME and, in particular, we show convergence of RelaxME and MomME (and also of ME) in one iteration when the matrix of the quadratic form has only two distinct eigenvalues. Finally, we provide the results of numerical experiments which compare on five optimization problems (and different combinations of parameters defining these problems) ME, RelaxME, and MomME, with 3 other optimizers: the conjugate gradient method [10], Barzilai and Borwein gradient method with long step [2], and the gradient method with adaptive spectral step length [7]. On most instances, MomME provides the smallest number of iterations and conjugate gradient and MomME provide the smallest CPU times and similar CPU times. This opens optimistic possibilities for ME with momentum in broader settings.