arXiv subjects

Aude Rondepierre

Publications and source records attributed to Aude Rondepierre.

14 recordsLinked to original sources

arXiv2026

Continuized Nesterov Momentum Achieves the $O(\varepsilon^{-7/4})$ Complexity in Smooth Nonconvex Optimization

For first-order optimization of non-convex functions with Lipschitz-continuous gradient and Hessian, the best-known complexity for reaching an $\varepsilon$-approximation of a stationary point is $\mathcal{O}(\varepsilon^{-7/4})$. The existing algorithms achieving this bound are based on momentum, but are always complemented with safeguard mechanisms that erase the accumulated momentum if a certain condition is violated. Whether such momentum-control mechanisms are fundamentally necessary has remained an open question. We show that randomizing the parameters enables one to achieve this complexity in expectation when using momentum without any of such mechanisms, and we improve the numerical constant factor of the bound in the case of a large enough number of iterations. From an analysis perspective, we do so by leveraging the continuized method, which interprets the algorithm as a realization of a continuous-time stochastic differential equation (SDE) involving a Poisson process. We show that this SDE converges in probability to the Heavy Ball ordinary differential equation when the stepsize goes to zero, paralleling the behavior of more classical instances of Nesterov momentum.

math.OC↗

arXiv2025

Continuized Nesterov Acceleration for Non-Convex Optimization

In convex optimization, continuous-time counterparts have been a fruitful tool for analyzing momentum algorithms. Fewer such examples are available when the function to minimize is non-convex. In several cases, discrepancies arise between the existing discrete-time results, namely those obtained for momentum algorithms, and their continuous-time counterparts, with the latter typically yielding stronger guarantees. We argue that the continuized framework (Even et al., 2021), mixing continuous and discrete components, can tighten the gap between known continuous and discrete results. This framework relies on computations akin to standard Lyapunov analyses, from which are deduced convergence bounds for an algorithm that can be written as a Nesterov momentum algorithm with stochastic parameters. In this work, we extend the range of applicability of the continuized framework, e.g. by allowing it to handle non-smooth Lyapunov functions. We then strengthen its trajectory-wise guarantees for linear convergence rate, deriving finite time bounds with high probability and asymptotic almost sure bounds. We apply this framework to the non-convex class of strongly quasar convex functions. Adapting continuous-time results that have weaker discrete equivalents to the continuized method, we improve by a constant factor the known convergence rate, and relax the existing assumptions on the set of minimizers.

math.OC↗

arXiv2025

Heavy Ball Momentum for Non-Strongly Convex Optimization

When considering the minimization of a quadratic or strongly convex function, it is well known that first-order methods involving an inertial term weighted by a constant-in-time parameter are particularly efficient (see Polyak [32], Nesterov [28], and references therein). By setting the inertial parameter according to the condition number of the objective function, these methods guarantee a fast exponential decay of the error. We prove that this type of schemes (which are later called Heavy Ball schemes) is relevant in a relaxed setting, i.e. for composite functions satisfying a quadratic growth condition. In particular, we adapt V-FISTA, introduced by Beck in [10] for strongly convex functions, to this broader class of functions. To the authors' knowledge, the resulting worst-case convergence rates are faster than any other in the literature, including those of FISTA restart schemes. No assumption on the set of minimizers is required and guarantees are also given in the non-optimal case, i.e. when the condition number is not exactly known. This analysis follows the study of the corresponding continuous-time dynamical system (Heavy Ball with friction system), for which new convergence results of the trajectory are shown.

math.OC↗

arXiv2024

Gradient correlation is a key ingredient to accelerate SGD with momentum

Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results.

math.OC↗

arXiv2024

Strong Convergence of FISTA Iterates under H{ö}lderian and Quadratic Growth Conditions

Introduced by Beck and Teboulle, FISTA (for Fast Iterative Shrinkage-Thresholding Algorithm) is a first-order method widely used in convex optimization. Adapted from Nesterov's accelerated gradient method for convex functions, the generated sequence guarantees a decay of the function values of $\mathcal{O}\left(n^{-2}\right)$ in the convex setting. We show that for coercive functions satisfying some local growth condition (namely a H\''olderian or quadratic growth condition), this sequence strongly converges to a minimizer. This property, which has never been proved without assuming the uniqueness of the minimizer, is associated with improved convergence rates for the function values. The proposed analysis is based on a preliminary study of the Asymptotic Vanishing Damping system introduced by Su et al. in to modelNesterov's accelerated gradient method in a continuous setting. Novel improved convergence results are also shown for the solutions of this dynamical system, including the finite length of the trajectory under the aforementioned geometry conditions.

math.OC↗

arXiv2024

Study of the behaviour of Nesterov Accelerated Gradient in a non convex setting: the strongly quasar convex case

We study the convergence of Nesterov Accelerated Gradient (NAG) minimization algorithmapplied to a class of non convex functions called strongly quasar convex functions. We show thatNAG can achieve an accelerated convergence speed at the cost of a lower curvature assumption.We provide a continuous analysis through high resolution ODEs, where we show that despite thatnegative friction may appear, the solution of the system achieves accelerated rate of convergenceto the minimum. Finally, we identify the key geometrical property that, if dropped, theoreticallycancels the acceleration phenomenon.

math.OC↗

arXiv2023

Parameter-Free FISTA by Adaptive Restart and Backtracking

We consider a combined restarting and adaptive backtracking strategy for the popular Fast Iterative Shrinking-Thresholding Algorithm frequently employed for accelerating the convergence speed of large-scale structured convex optimization problems. Several variants of FISTA enjoy a provable linear convergence rate for the function values $F(x_n)$ of the form $\mathcal{O}( e^{-K\sqrt{μ/L}~n})$ under the prior knowledge of problem conditioning, i.e. of the ratio between the (Łojasiewicz) parameter $μ$ determining the growth of the objective function and the Lipschitz constant $L$ of its smooth component. These parameters are nonetheless hard to estimate in many practical cases. Recent works address the problem by estimating either parameter via suitable adaptive strategies. In our work both parameters can be estimated at the same time by means of an algorithmic restarting scheme where, at each restart, a non-monotone estimation of $L$ is performed. For this scheme, theoretical convergence results are proved, showing that a $\mathcal{O}( e^{-K\sqrt{μ/L}n})$ convergence speed can still be achieved along with quantitative estimates of the conditioning. The resulting Free-FISTA algorithm is therefore parameter-free. Several numerical results are reported to confirm the practical interest of its use in many exemplar problems.

math.OC↗

arXiv2022

Fast convergence of inertial dynamics with Hessian-driven damping under geometry assumptions

First-order optimization algorithms can be considered as a discretization of ordinary differential equations (ODEs) \cite{su2014differential}. In this perspective, studying the properties of the corresponding trajectories may lead to convergence results which can be transfered to the numerical scheme. In this paper we analyse the following ODE introduced by Attouch et al. in \cite{attouch2016fast}: \begin{equation*} \forall t\geqslant t_0,~\ddot{x}(t)+\fracα{t}\dot{x}(t)+βH_F(x(t))\dot{x}(t)+\nabla F(x(t))=0,\end{equation*} where $α>0$, $β>0$ and $H_F$ denotes the Hessian of $F$. This ODE can be derived to build numerical schemes which do not require $F$ to be twice differentiable as shown in \cite{attouch2020first,attouch2021convergence}. We provide strong convergence results on the error $F(x(t))-F^*$ and integrability properties on $\|\nabla F(x(t))\|$ under some geometry assumptions on $F$ such as quadratic growth around the set of minimizers. In particular, we show that the decay rate of the error for a strongly convex function is $O(t^{-α-\varepsilon})$ for any $\varepsilon>0$. These results are briefly illustrated at the end of the paper.

math.OC↗

arXiv2019

Optimal convergence rates for Nesterov acceleration

In this paper, we study the behavior of solutions of the ODE associated to Nesterov acceleration. It is well-known since the pioneering work of Nesterov that the rate of convergence $O(1/t^2)$ is optimal for the class of convex functions with Lipschitz gradient. In this work, we show that better convergence rates can be obtained with some additional geometrical conditions, such as Łojasiewicz property. More precisely, we prove the optimal convergence rates that can be obtained depending on the geometry of the function $F$ to minimize. The convergence rates are new, and they shed new light on the behavior of Nesterov acceleration schemes. We prove in particular that the classical Nesterov scheme may provide convergence rates that are worse than the classical gradient descent scheme on sharp functions: for instance, the convergence rate for strongly convex functions is not geometric for the classical Nesterov scheme (while it is the case for the gradient descent algorithm). This shows that applying the classical Nesterov acceleration on convex functions without looking more at the geometrical properties of the objective functions may lead to sub-optimal algorithms.

math.OC↗

arXiv2019

Nesterov's acceleration and Polyak's heavy ball method in continuous time: convergence rate analysis under geometric conditions and perturbations

In this article a family of second order ODEs associated to inertial gradient descend is studied. These ODEs are widely used to build trajectories converging to a minimizer $x^*$ of a function $F$, possibly convex. This family includes the continuous version of the Nesterov inertial scheme and the continuous heavy ball method. Several damping parameters, not necessarily vanishing, and a perturbation term $g$ are thus considered. The damping parameter is linked to the inertia of the associated inertial scheme and the perturbation term $g$ is linked to the error that can be done on the gradient of the function $F$. This article presents new asymptotic bounds on $F(x(t))-F(x^*)$ where $x$ is a solution of the ODE, when $F$ is convex and satisfies local geometrical properties such as Łojasiewicz properties and under integrability conditions on $g$. Even if geometrical properties and perturbations were already studied for most ODEs of these families, it is the first time they are jointly studied. All these results give an insight on the behavior of these inertial and perturbed algorithms if $F$ satisfies some Łojasiewicz properties especially in the setting of stochastic algorithms.

math.OC↗

arXiv2014

On local convergence of the method of alternating projections

The method of alternating projections is a classical tool to solve feasibility problems. Here we prove local convergence of alternating projections between subanalytic sets $A,B$ under a mild regularity hypothesis on one of the sets. We show that the speed of convergence is O$(k^{-ρ})$ for some $ρ\in (0,\infty)$.

math.OC↗

arXiv2009

A proximal approach to the inversion of ill-conditioned matrices

We propose a general proximal algorithm for the inversion of ill-conditioned matrices. This algorithm is based on a variational characterization of pseudo-inverses. We show that a particular instance of it (with constant regularization parameter) belongs to the class of {\sl fixed point} methods. Convergence of the algorithm is also discussed.

math.NA↗

arXiv2008

An hybrid system approach to nonlinear optimal control problems

We consider a nonlinear ordinary differential equation and want to control its behavior so that it reaches a target by minimizing a cost function. Our approach is to use hybrid systems to solve this problem: the complex dynamic is replaced by piecewise affine approximations which allow an analytical resolution. The sequence of affine models then forms a sequence of states of a hybrid automaton. Given a sequence of states, we introduce an hybrid approximation of the nonlinear controllable domain and propose a new algorithm computing a controllable, piecewise convex approximation. The same way the nonlinear optimal control problem is replaced by an hybrid piecewise affine one. Stating a hybrid maximum principle suitable to our hybrid model, we deduce the global structure of the hybrid optimal control steering the system to the target.

math.OC↗

arXiv2006

Computing the Kalman form

We present two algorithms for the computation of the Kalman form of a linear control system. The first one is based on the technique developed by Keller-Gehrig for the computation of the characteristic polynomial. The cost is a logarithmic number of matrix multiplications. To our knowledge, this improves the best previously known algebraic complexity by an order of magnitude. Then we also present a cubic algorithm proven to more efficient in practice.

cs.SC↗