arXiv subjects

Julien Hermant

Publications and source records attributed to Julien Hermant.

8 recordsLinked to original sources

arXiv2026

Accelerated Stochastic Zeroth-Order Quasar-Convex Optimization

We consider unconstrained minimization of smooth quasar-convex functions when only noisy function evaluations are accessible through a stochastic zeroth-order oracle. For these non-convex functions, the standard acceleration method relies on subspace-search mechanisms that require first-order information, being therefore unavailable in zeroth-order regimes. In contrast, the alternative and less conventional continuized method enable to avoid such mechanisms. In this work, we design a zeroth-order continuized algorithm, leading to accelerated convergence guarantees that parallel those of smooth convex optimization up to a quasar-convexity parameter. Our method incorporates a mirror step, improving the dimension dependence when there exists sparse solution.

math.OC↗

arXiv2026

Acceleration for Polyak-{\L}ojasiewicz Functions with a Gradient Aiming Condition

It is known that when minimizing smooth Polyak-{\L}ojasiewicz (PL) functions, momentum algorithms cannot significantly improve the convergence bound of gradient descent, contrasting with the acceleration phenomenon occurring in the strongly convex case. To bridge this gap, the literature has proposed strongly quasar-convex functions as an intermediate non-convex class, for which accelerated bounds have been suggested to persist. We show that this is not true in general: the additional structure of strong quasar-convexity does not suffice to guaranty better worst-case bounds for momentum compared to gradient descent. As an alternative, we study PL functions under an aiming condition that measures how well the descent direction points toward a minimizer. This perspective clarifies the geometric ingredient enabling provable acceleration by momentum when minimizing PL functions.

math.OC↗

arXiv2026

Continuized Nesterov Momentum Achieves the $O(\varepsilon^{-7/4})$ Complexity in Smooth Nonconvex Optimization

For first-order optimization of non-convex functions with Lipschitz-continuous gradient and Hessian, the best-known complexity for reaching an $\varepsilon$-approximation of a stationary point is $\mathcal{O}(\varepsilon^{-7/4})$. The existing algorithms achieving this bound are based on momentum, but are always complemented with safeguard mechanisms that erase the accumulated momentum if a certain condition is violated. Whether such momentum-control mechanisms are fundamentally necessary has remained an open question. We show that randomizing the parameters enables one to achieve this complexity in expectation when using momentum without any of such mechanisms, and we improve the numerical constant factor of the bound in the case of a large enough number of iterations. From an analysis perspective, we do so by leveraging the continuized method, which interprets the algorithm as a realization of a continuous-time stochastic differential equation (SDE) involving a Poisson process. We show that this SDE converges in probability to the Heavy Ball ordinary differential equation when the stepsize goes to zero, paralleling the behavior of more classical instances of Nesterov momentum.

math.OC↗

arXiv2025

Continuized Nesterov Acceleration for Non-Convex Optimization

In convex optimization, continuous-time counterparts have been a fruitful tool for analyzing momentum algorithms. Fewer such examples are available when the function to minimize is non-convex. In several cases, discrepancies arise between the existing discrete-time results, namely those obtained for momentum algorithms, and their continuous-time counterparts, with the latter typically yielding stronger guarantees. We argue that the continuized framework (Even et al., 2021), mixing continuous and discrete components, can tighten the gap between known continuous and discrete results. This framework relies on computations akin to standard Lyapunov analyses, from which are deduced convergence bounds for an algorithm that can be written as a Nesterov momentum algorithm with stochastic parameters. In this work, we extend the range of applicability of the continuized framework, e.g. by allowing it to handle non-smooth Lyapunov functions. We then strengthen its trajectory-wise guarantees for linear convergence rate, deriving finite time bounds with high probability and asymptotic almost sure bounds. We apply this framework to the non-convex class of strongly quasar convex functions. Adapting continuous-time results that have weaker discrete equivalents to the continuized method, we improve by a constant factor the known convergence rate, and relax the existing assumptions on the set of minimizers.

math.OC↗

arXiv2025

Provably Accelerated Imaging with Restarted Inertia and Score-based Image Priors

Fast convergence and high-quality image recovery are two essential features of algorithms for solving ill-posed imaging inverse problems. Existing methods, such as regularization by denoising (RED), often focus on designing sophisticated image priors to improve reconstruction quality, while leaving convergence acceleration to heuristics. To bridge the gap, we propose Restarted Inertia with Score-based Priors (RISP) as a principled extension of RED. RISP incorporates a restarting inertia for fast convergence, while still allowing score-based image priors for high-quality reconstruction. We prove that RISP attains a faster stationary-point convergence rate than RED, without requiring the convexity of the image prior. We further derive and analyze the associated continuous-time dynamical system, offering insight into the connection between RISP and the heavy-ball ordinary differential equation (ODE). Experiments across a range of imaging inverse problems demonstrate that RISP enables fast convergence while achieving high-quality reconstructions.

cs.CV↗

arXiv2024

Convergence Analysis of a Proximal Stochastic Denoising Regularization Algorithm

Plug-and-Play methods for image restoration are iterative algorithms that solve a variational problem to recover a clean image from a degraded observation. These algorithms are known to be flexible to changes of degradation and to perform state-of-the-art restoration. Recently, significant efforts have been made to explore new stochastic algorithms based on the Plug-and-Play or REgularization by Denoising (RED) frameworks, such as SNORE, which is a convergent stochastic gradient descent algorithm. A variant of this algorithm, named SNORE Prox, reaches state-of-the-art performances, especially for inpainting tasks. However, the convergence of SNORE Prox, that can be seen as a stochastic proximal gradient descent, has not been analyzed so far. In this paper, we prove the convergence of SNORE Prox under non convex assumptions.

eess.IV↗

arXiv2024

Gradient correlation is a key ingredient to accelerate SGD with momentum

Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results.

math.OC↗

arXiv2024

Study of the behaviour of Nesterov Accelerated Gradient in a non convex setting: the strongly quasar convex case

We study the convergence of Nesterov Accelerated Gradient (NAG) minimization algorithmapplied to a class of non convex functions called strongly quasar convex functions. We show thatNAG can achieve an accelerated convergence speed at the cost of a lower curvature assumption.We provide a continuous analysis through high resolution ODEs, where we show that despite thatnegative friction may appear, the solution of the system achieves accelerated rate of convergenceto the minimum. Finally, we identify the key geometrical property that, if dropped, theoreticallycancels the acceleration phenomenon.

math.OC↗