arXiv subjects

Mihai I. Florea

Publications and source records attributed to Mihai I. Florea.

8 recordsLinked to original sources

arXiv2024

A template for gradient norm minimization

The gradient mapping norm is a strong and easily verifiable stopping criterion for first-order methods on composite problems. When the objective exhibits the quadratic growth property, the gradient mapping norm minimization problem can be solved by online parameter-free and adaptive first-order schemes with near-optimal worst-case rates. In this work we address problems where quadratic growth is absent, a class for which no methods with all the aforementioned properties are known to exist. We formulate a template whose instantiation recovers the existing Performance Estimation derived approaches. Our framework provides a simple human-readable interpretation along with runtime convergence rates for these algorithms. Moreover, our template can be used to construct a quasi-online parameter-free method applicable to the entire class of composite problems while retaining the optimal worst-case rates with the best known proportionality constant. The analysis also allows for adaptivity. Preliminary simulation results suggest that our scheme is highly competitive in practice with the existing approaches, either obtained via Performance Estimation or employing Accumulative Regularization.

math.OC↗

arXiv2024

Adaptive first-order methods with enhanced worst-case rates

The Optimized Gradient Method (OGM), its strongly convex extension, the Information Theoretical Exact Method (ITEM), as well as the related Triple Momentum Method (TMM) have superior convergence guarantees when compared to the Fast Gradient Method but lack adaptivity and their derivation is incompatible with composite problems. In this work we introduce a slightly modified version of the estimate sequence that can be used to simultaneously derive OGM, ITEM and TMM while adding memory along with the ability to dynamically adjust the convergence guarantees at runtime. Our framework can be extended to the composite setup and we use it to construct an Enhanced Accelerated Composite Gradient Method equipped with fully-adaptive line-search.

math.OC↗

arXiv2024

An optimal lower bound for smooth convex functions

First order methods endowed with global convergence guarantees operate using global lower bounds on the objective. The tightening of the bounds has been shown to increase both the theoretical guarantees and the practical performance. In this work, we define a global lower bound for smooth differentiable objectives that is optimal with respect to the collected oracle information. The bound can be readily employed by the Gradient Method with Memory to improve its performance. Further using the machinery underlying the optimal bounds, we introduce a modified version of the estimate sequence that we use to construct an Optimized Gradient Method with Memory possessing the best known convergence guarantees for its class of algorithms, even in terms of the proportionality constant. We additionally equip the method with an adaptive convergence guarantee adjustment procedure that is an effective replacement for line-search. Simulation results on synthetic but otherwise difficult smooth problems validate the theoretical properties of the bound and proposed methods.

math.OC↗

arXiv2024

Gradient Methods with Memory for Minimizing Composite Functions

The recently introduced Gradient Methods with Memory use a subset of the past oracle information to create an accurate model of the objective function that enables them to surpass the Gradient Method in practical performance. The model introduces an overhead that is substantial on all problems but the smooth unconstrained ones. In this work, we introduce several Gradient Methods with Memory that can solve composite problems efficiently, including unconstrained problems with non-smooth objectives. The auxiliary problem at each iteration still cannot be solved exactly but we show how to alter the model and how to initialize the auxiliary problem solver to ensure that this inexactness does not degrade the convergence guarantees. Moreover, we dynamically increase the convergence guarantees as to provably surpass those of their memory-less counterparts. These properties are preserved when applying acceleration and the containment of inexactness further prevents error accumulation. Our methods are able to estimate key geometry parameters to attain state-of-the-art worst-case rates on many important subclasses of composite problems, where the objective smooth part satisfies a strong convexity condition or a relaxation thereof. In particular, we formulate a near-optimal restart strategy applicable to optimization methods with sublinear convergence guarantees of any order. We support the theoretical results with simulations.

math.OC↗

arXiv2021

Gradient Methods with Memory

In this paper, we consider gradient methods for minimizing smooth convex functions, which employ the information obtained at the previous iterations in order to accelerate the convergence towards the optimal solution. This information is used in the form of a piece-wise linear model of the objective function, which provides us with much better prediction abilities as compared with the standard linear model. To the best of our knowledge, this approach was never really applied in Convex Minimization to differentiable functions in view of the high complexity of the corresponding auxiliary problems. However, we show that all necessary computations can be done very efficiently. Consequently, we get new optimization methods, which are better than the usual Gradient Methods both in the number of oracle calls and in the computational time. Our theoretical conclusions are confirmed by preliminary computational experiments.

math.OC↗

arXiv2019

A Generalized Accelerated Composite Gradient Method: Uniting Nesterov's Fast Gradient Method and FISTA

The most popular first-order accelerated black-box methods for solving large-scale convex optimization problems are the Fast Gradient Method (FGM) and the Fast Iterative Shrinkage Thresholding Algorithm (FISTA). FGM requires that the objective be finite and differentiable with known gradient Lipschitz constant. FISTA is applicable to the more broad class of composite objectives and is equipped with a line-search procedure for estimating the Lipschitz constant. Nonetheless, FISTA cannot increase the step size and is unable to take advantage of strong convexity. FGM and FISTA are very similar in form. Despite this, they appear to have vastly differing convergence analyses. In this work we generalize the previously introduced augmented estimate sequence framework as well as the related notion of the gap sequence. We showcase the flexibility of our tools by constructing a Generalized Accelerated Composite Gradient Method, that unites FGM and FISTA, along with their most popular variants. The Lyapunov property of the generalized gap sequence used in deriving our method implies that both FGM and FISTA are amenable to a Lyapunov analysis, common among optimization algorithms. We further showcase the flexibility of our tools by endowing our method with monotonically decreasing objective function values alongside a versatile line-search procedure. By simultaneously incorporating the strengths of FGM and FISTA, our method is able to surpass both in terms of robustness and usability. We support our findings with simulation results on an extensive benchmark of composite problems. Our experiments show that monotonicity has a stabilizing effect on convergence and challenge the notion present in the literature that for strongly convex objectives, accelerated proximal schemes can be reduced to fixed momentum methods.

math.OC↗

arXiv2018

An axially-variant kernel imaging model applied to ultrasound image reconstruction

Existing ultrasound deconvolution approaches unrealistically assume, primarily for computational reasons, that the convolution model relies on a spatially invariant kernel and circulant boundary conditions. We discard both restrictions and introduce an image formation model applicable to ultrasound imaging and deconvolution based on an axially varying kernel, that accounts for arbitrary boundary conditions. Our model has the same computational complexity as the one employing spatially invariant convolution and has negligible memory requirements. To accommodate state-of-the-art deconvolution approaches when applied to a variety of inverse problem formulations, we also provide an equally efficient adjoint expression of our model. Simulation results confirm the tractability of our model for the deconvolution of large images. Moreover, the quality of reconstruction using our model is superior to that obtained using spatially invariant convolution.

eess.SP↗

arXiv2017

An Accelerated Composite Gradient Method for Large-scale Composite Objective Problems

We introduce a framework, which we denote as the augmented estimate sequence, for deriving fast algorithms with provable convergence guarantees. We use this framework to construct a new first-order scheme, the Accelerated Composite Gradient Method (ACGM), for large-scale problems with composite objective structure. ACGM surpasses the state-of-the-art methods for this problem class in terms of provable convergence rate, both in the strongly and non-strongly convex cases, and is endowed with an efficient step size search procedure. We support the effectiveness of our new method with simulation results.

math.OC↗