Searcharxiv⌕ Search

arXiv subjects

Hedy Attouch

Publications and source records attributed to Hedy Attouch.

36 records · Page 2Linked to original sources
arXiv2020

First-order optimization algorithms via inertial systems with Hessian driven damping

In a Hilbert space setting, for convex optimization, we analyze the convergence rate of a class of first-order algorithms involving inertial features. They can be interpreted as discrete time versions of inertial dynamics involving both viscous and Hessian-driven dampings. The geometrical damping driven by the Hessian intervenes in the dynamics in the form $\nabla^2 f (x(t)) \dot{x} (t)$. By treating this term as the time derivative of $ \nabla f (x (t)) $, this gives, in discretized form, first-order algorithms in time and space. In addition to the convergence properties attached to Nesterov-type accelerated gradient methods, the algorithms thus obtained are new and show a rapid convergence towards zero of the gradients. On the basis of a regularization technique using the Moreau envelope, we extend these methods to non-smooth convex functions with extended real values. The introduction of time scale factors makes it possible to further accelerate these algorithms. We also report numerical results on structured problems to support our theoretical findings.

math.OC↗
arXiv2020

Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling

In a Hilbert setting, we develop fast methods for convex unconstrained optimization. We rely on the asymptotic behavior of an inertial system combining geometric damping with temporal scaling. The convex function to minimize enters the dynamic via its gradient. The dynamic includes three coefficients varying with time, one is a viscous damping coefficient, the second is attached to the Hessian-driven damping, the third is a time scaling coefficient. We study the convergence rate of the values under general conditions involving the damping and the time scale coefficients. The obtained results are based on a new Lyapunov analysis and they encompass known results on the subject. We pay particular attention to the case of an asymptotically vanishing viscous damping, which is directly related to the accelerated gradient method of Nesterov. The Hessian-driven damping significantly reduces the oscillatory aspects. As a main result, we obtain an exponential rate of convergence of values without assuming the strong convexity of the objective function. The temporal discretization of these dynamics opens the gate to a large class of inertial optimization algorithms.

math.OC↗
arXiv2020

Fast convex optimization via a third-order in time evolution equation: TOGES-V an improved version of TOGES

In a Hilbert space setting H, for convex optimization, we analyze the fast convergence properties as t tends to infinity of the trajectories generated by a third-order in time evolution system. The function f to minimize is supposed to be convex, continuously differentiable, with a nonempty set of minimizers. It enters into the dynamic through its gradient. Based on this new dynamical system, we improve the results obtained by [Attouch, Chbani, Riahi: Fast convex optimization via a third-order in time evolution equation, Optimization 2020]. As a main result, when the damping parameter $α$ satisfies $α> 3$, we show that the convergence of the values at the order 1/t3 as t goes to infinity, as well as the convergence of the trajectories. We complement these results by introducing into the dynamic an Hessian driven damping term, which reduces the oscillations. In the case of a strongly convex function f, we show an autonomous evolution system of the third order in time with an exponential rate of convergence. All these results have natural extensions to the case of a convex lower semicontinuous function with extended real values. Just replace f with its Moreau envelope.

math.OC↗
arXiv2017

Convergence of inertial dynamics and proximal algorithms governed by maximally monotone operators

We study the behavior of the trajectories of a second-order differential equation with vanishing damping, governed by the Yosida regularization of a maximally monotone operator with time-varying index, along with a new {\em Regularized Inertial Proximal Algorithm} obtained by means of a convenient finite-difference discretization. These systems are the counterpart to accelerated forward-backward algorithms in the context of maximally monotone operators. A proper tuning of the parameters allows us to prove the weak convergence of the trajectories to zeroes of the operator. Moreover, it is possible to estimate the rate at which the speed and acceleration vanish. We also study the effect of perturbations or computational errors that leave the convergence properties unchanged. We also analyze a growth condition under which strong convergence can be guaranteed. A simple example shows the criticality of the assumptions on the Yosida approximation parameter, and allows us to illustrate the behavior of these systems compared with some of their close relatives.

math.OC↗
arXiv2017

Rate of convergence of the Nesterov accelerated gradient method in the subcritical case $α\leq 3$

In a Hilbert space setting $\mathcal H$, given $Φ: \mathcal H \to \mathbb R$ a convex continuously differentiable function, and $α$ a positive parameter, we consider the inertial system with Asymptotic Vanishing Damping \begin{equation*} \mbox{(AVD)}_α \quad \quad \ddot{x}(t) + \fracα{t} \dot{x}(t) + \nabla Φ(x(t)) =0. \end{equation*} Depending on the value of $ α$ with respect to 3, we give a complete picture of the convergence properties as $t \to + \infty$ of the trajectories generated by $\mbox{(AVD)}_α$, as well as iterations of the corresponding algorithms. Our main result concerns the subcritical case $α\leq 3$, where we show that $Φ(x(t))-\min Φ= \mathcal O (t^{-\frac{2}{3}α})$. Then we examine the convergence of trajectories to optimal solutions. As a new result, in the one-dimensional framework, for the critical value $α= 3 $, we prove the convergence of the trajectories without any restrictive hypothesis on the convex function $Φ$. In the second part of this paper, we study the convergence properties of the associated forward-backward inertial algorithms. They aim to solve structured convex minimization problems of the form $\min \left\lbrace Θ:= Φ+ Ψ\right\rbrace$, with $Φ$ smooth and $Ψ$ nonsmooth. The continuous dynamics serves as a guideline for this study. We obtain a similar rate of convergence for the sequence of iterates $(x_k)$: for $α\leq 3$ we have $Θ(x_k)-\min Θ= \mathcal O (k^{-p})$ for all $p <\frac{2α}{3}$ , and for $α> 3$ \ $Θ(x_k)-\min Θ= o (k^{-2})$ . We conclude this study by showing that the results are robust with respect to external perturbations.

math.OC↗
arXiv2017

The rate of convergence of Nesterov's accelerated forward-backward method is actually faster than $1/k^{2}$

The {\it forward-backward algorithm} is a powerful tool for solving optimization problems with a {\it additively separable} and {\it smooth} + {\it nonsmooth} structure. In the convex setting, a simple but ingenious acceleration scheme developed by Nesterov has been proved useful to improve the theoretical rate of convergence for the function values from the standard $\mathcal O(k^{-1})$ down to $\mathcal O(k^{-2})$. In this short paper, we prove that the rate of convergence of a slight variant of Nesterov's accelerated forward-backward method, which produces {\it convergent} sequences, is actually $o(k^{-2})$, rather than $\mathcal O(k^{-2})$. Our arguments rely on the connection between this algorithm and a second-order differential inclusion with vanishing damping.

math.OC↗
arXiv2016

Combining fast inertial dynamics for convex optimization with Tikhonov regularization

In a Hilbert space setting $\mathcal H$, we study the convergence properties as $t \to + \infty$ of the trajectories of the second-order differential equation \begin{equation*} \mbox{(AVD)}_{α, ε} \quad \quad \ddot{x}(t) + \fracα{t} \dot{x}(t) + \nabla Φ(x(t)) + ε(t) x(t) =0, \end{equation*} where $\nablaΦ$ is the gradient of a convex continuously differentiable function $Φ: \mathcal H \to \mathbb R$, $α$ is a positive parameter, and $ε(t) x(t)$ is a Tikhonov regularization term, with $\lim_{t \to \infty}ε(t) =0$. In this damped inertial system, the damping coefficient $\fracα{t}$ vanishes asymptotically, but not too quickly, a key property to obtain rapid convergence of the values. In the case $ε(\cdot) \equiv 0$, this dynamic has been highlighted recently by Su, Boyd, and Candès as a continuous version of the Nesterov accelerated method. Depending on the speed of convergence of $ε(t)$ to zero, we analyze the convergence properties of the trajectories of $\mbox{(AVD)}_{α, ε}$. We obtain results ranging from the rapid convergence of $Φ(x(t))$ to $\min Φ$ when $ε(t)$ decreases rapidly to zero, up to the strong ergodic convergence of the trajectories to the element of minimal norm of the set of minimizers of $Φ$, when $ε(t)$ tends slowly to zero.

math.OC↗
arXiv2016

Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects

In a Hilbert space $\mathcal H$, we study the asymptotic behaviour, as time variable $t$ goes to $+\infty$, of nonautonomous gradient-like dynamical systems involving inertia and multiscale features. Given $\mathcal H$ a general Hilbert space, $Φ: \mathcal H \rightarrow \mathbb R$ and $Ψ: \mathcal H \rightarrow \mathbb R$ two convex differentiable functions, $γ$ a positive damping parameter, and $ε(t)$ a function of $t$ which tends to zero as $t$ goes to $+\infty$, we consider the second-order differential equation $$\ddot{x}(t) + γ\dot{x}(t) + \nabla Φ(x(t)) + ε(t) \nabla Ψ(x(t)) = 0. $$ This system models the emergence of various collective behaviors in game theory, as well as the asymptotic control of coupled nonlinear oscillators. Assuming that $ε(t)$ tends to zero moderately slowly as $t$ goes to infinity, we show that the trajectories converge weakly in $\mathcal H$. The limiting equilibria are solutions of the hierarchical minimization problem which consists in minimizing $Ψ$ over the set $C$ of minimizers of $Φ$. As key assumptions, we suppose that $ \int_{0}^{+\infty}ε(t) dt = + \infty $ and that, for every $p$ belonging to a convex cone $\mathcal C$ depending on the data $Φ$ and $Ψ$ $$ \int_{0}^{+\infty} \left[Φ^* \left(ε(t)p\right) -σ_C \left(ε(t)p\right)\right]dt < + \infty $$ where $Φ^*$ is the Fenchel conjugate of $Φ$, and $σ_C $ is the support function of $C$. An application is given to coupled oscillators.

math.OC↗
arXiv2016

Fast convex optimization via inertial dynamics with Hessian driven damping

We first study the fast minimization properties of the trajectories of the second-order evolution equation $$\ddot{x}(t) + \fracα{t} \dot{x}(t) + β\nabla^2 Φ(x(t))\dot{x} (t) + \nabla Φ(x(t)) = 0,$$ where $Φ:\mathcal H\to\mathbb R$ is a smooth convex function acting on a real Hilbert space $\mathcal H$, and $α$, $β$ are positive parameters. This inertial system combines an isotropic viscous damping which vanishes asymptotically, and a geometrical Hessian driven damping, which makes it naturally related to Newton's and Levenberg-Marquardt methods. For $α\geq 3$, $β>0$, along any trajectory, fast convergence of the values $$Φ(x(t))- \min_{\mathcal H}Φ=\mathcal O\left(t^{-2}\right)$$ is obtained, together with rapid convergence of the gradients $\nablaΦ(x(t))$ to zero. For $α>3$, just assuming that $Φ$ has minimizers, we show that any trajectory converges weakly to a minimizer of $Φ$, and $ Φ(x(t))-\min_{\mathcal H}Φ= o(t^{-2})$. Strong convergence is established in various practical situations. For the strongly convex case, convergence can be arbitrarily fast depending on the choice of $α$. More precisely, we have $Φ(x(t))- \min_{\mathcal H}Φ= \mathcal O(t^{-\frac{2}{3}α})$. We extend the results to the case of a general proper lower-semicontinuous convex function $Φ: \mathcal H \rightarrow \mathbb R \cup \{+\infty \}$. This is based on the fact that the inertial dynamic with Hessian driven damping can be written as a first-order system in time and space. By explicit-implicit time discretization, this opens a gate to new $-$ possibly more rapid $-$ inertial algorithms, expanding the field of FISTA methods for convex structured optimization problems.

math.OC↗
arXiv2016

Asymptotic behavior of nonautonomous monotone and subgradient evolution equations

In a Hilbert setting $H$, we study the asymptotic behavior of the trajectories of nonautonomous evolution equations $\dot x(t)+A_t(x(t))\ni 0$, where for each $t\geq 0$, $A_t:H\tto H$ denotes a maximal monotone operator. We provide general conditions guaranteeing the weak ergodic convergence of each trajectory $x(\cdot)$ to a zero of a limit maximal monotone operator $ A_\infty$, as the time variable $t$ tends to $+\infty$. The crucial point is to use the Brézis-Haraux function, or equivalently the Fitzpatrick function, to express at which rate the excess of $\gph A_\infty$ over $\gph A_t$ tends to zero. This approach gives a sharp and unifying view on this subject. In the case of operators $A_t= \partial ϕ_t$ which are subdifferentials of closed convex functions $ϕ_t$, we show convergence results for the trajectories. Then, we specialize our results to multiscale evolution equations, and obtain asymptotic properties of hierarchical minimization, and selection of viscosity solutions. Illustrations are given in the field of coupled systems, and partial differential equations.

math.OC↗
arXiv2015

Fast Convergence of an Inertial Gradient-like System with Vanishing Viscosity

In a real Hilbert space $\mathcal H$, we study the fast convergence properties as $t \to + \infty$ of the trajectories of the second-order evolution equation $$ \ddot{x}(t) + \fracα{t} \dot{x}(t) + \nabla Φ(x(t)) = 0, $$ where $\nabla Φ$ is the gradient of a convex continuously differentiable function $Φ: \mathcal H \rightarrow \mathbb R$, and $α$ is a positive parameter. In this inertial system, the viscous damping coefficient $\fracα{t}$ vanishes asymptotically in a moderate way. For $α> 3$, we show that any trajectory converges weakly to a minimizer of $Φ$, just assuming that the set of minimizers is nonempty. The strong convergence is established in various practical situations. These results complement the $\mathcal O(t^{-2})$ rate of convergence for the values obtained by Su, Boyd and Candès. Time discretization of this system, and some of its variants, provides new fast converging algorithms, expanding the field of rapid methods for structured convex minimization introduced by Nesterov, and further developed by Beck and Teboulle. This study also complements recent advances due to Chambolle and Dossal.

math.OC↗
arXiv2015

A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert spaces, with complexity O(1/n^2)

In a Hilbert setting, we introduce a new dynamical system and associated algorithms for solving monotone inclusions by rapid methods. Given a maximal monotone operator $A$, the evolution is governed by the time dependent operator $I -(I + λ(t) {A})^{-1}$, where the positive control parameter $λ(t)$ tends to infinity as $t \to + \infty$. The tuning of $ λ(\cdot) $ is done in a closed-loop way, by resolution of the algebraic equation $λ\norm{(I + λ{A})^{-1}x -x}=θ$, where $θ$ is a positive given constant. The existence and uniqueness of a strong global solution for the Cauchy problem follows from Cauchy-Lipschitz theorem. We prove the weak convergence of the trajectories to equilibria, and superlinear convergence under an error bound condition. When $A =\partial f$ is the subdifferential of a closed convex function $f$, we show a $\bigo(1/t^2)$ convergence property of $f(x(t))$ to the infimal value of the problem. Then, we introduce proximal-like algorithms which can be obtained by time discretization of the continuous dynamic, and which share the same fast convergence properties. As distinctive features, we allow a relative error tolerance for the solution of the proximal subproblem similar to the ones proposed in ~\cite{So-Sv1, So-Sv2}, and a large step condition, as proposed in~\cite{MS1,MS2}. For general convex minimization problems, the complexity is $\bigo(1/n^2)$. In the regular case, we show the global quadratic convergence of an associated proximal-Newton method.

math.OC↗
arXiv2014

A dynamic gradient approach to Pareto optimization with nonsmooth convex objective functions

In a general Hilbert framework, we consider continuous gradient-like dynamical systems for constrained multiobjective optimization involving non-smooth convex objective functions. Our approach is in the line of a previous work where was considered the case of convex di erentiable objective functions. Based on the Yosida regularization of the subdi erential operators involved in the system, we obtain the existence of strong global trajectories. We prove a descent property for each objective function, and the convergence of trajectories to weak Pareto minima. This approach provides a dynamical endogenous weighting of the objective functions. Applications are given to cooperative games, inverse problems, and numerical multiobjective optimization.

math.OC↗
arXiv2014

Dynamical systems and forward-backward algorithms associated with the sum of a convex subdifferential and a monotone cocoercive operator

In a Hilbert framework, we introduce continuous and discrete dynamical systems which aim at solving inclusions governed by structured monotone operators $A=\partialΦ+B$, where $\partialΦ$ is the subdifferential of a convex lower semicontinuous function $Φ$, and $B$ is a monotone cocoercive operator. We first consider the extension to this setting of the regularized Newton dynamic with two potentials. Then, we revisit some related dynamical systems, namely the semigroup of contractions generated by $A$, and the continuous gradient projection dynamic. By a Lyapunov analysis, we show the convergence properties of the orbits of these systems. The time discretization of these dynamics gives various forward-backward splitting methods (some new) for solving structured monotone inclusions involving non-potential terms. The convergence of these algorithms is obtained under classical step size limitation. Perspectives are given in the field of numerical splitting methods for optimization, and multi-criteria decision processes.

math.OC↗
arXiv2013

Proximal alternating minimization and projection methods for nonconvex problems. An approach based on the Kurdyka-Lojasiewicz inequality

We study the convergence properties of an alternating proximal minimization algorithm for nonconvex structured functions of the type: $L(x,y)=f(x)+Q(x,y)+g(y)$, where $f:\R^n\rightarrow\R\cup{+\infty}$ and $g:\R^m\rightarrow\R\cup{+\infty}$ are proper lower semicontinuous functions, and $Q:\R^n\times\R^m\rightarrow \R$ is a smooth $C^1$ function which couples the variables $x$ and $y$. The algorithm can be viewed as a proximal regularization of the usual Gauss-Seidel method to minimize $L$. We work in a nonconvex setting, just assuming that the function $L$ satisfies the Kurdyka-Łojasiewicz inequality. An entire section illustrates the relevancy of such an assumption by giving examples ranging from semialgebraic geometry to "metrically regular" problems. Our main result can be stated as follows: If L has the Kurdyka-Łojasiewicz property, then each bounded sequence generated by the algorithm converges to a critical point of $L$. This result is completed by the study of the convergence rate of the algorithm, which depends on the geometrical properties of the function $L$ around its critical points. When specialized to $Q(x,y)=|x-y|^2$ and to $f$, $g$ indicator functions, the algorithm is an alternating projection mehod (a variant of Von Neumann's) that converges for a wide class of sets including semialgebraic and tame sets, transverse smooth manifolds or sets with "regular" intersection. In order to illustrate our results with concrete problems, we provide a convergent proximal reweighted $\ell^1$ algorithm for compressive sensing and an application to rank reduction problems.

math.OC↗
arXiv2009

Worthwhile-to-move behaviors as temporary satisficing without too many sacrificing processes

The worthwhile-to-move incremental principle is a mechanism where, at each step, the agent, before moving and after exploration around the current state, compares intermediate advantages and costs to change to advantages and costs to stay. These advantages and costs to change include goal-setting, psychological, cognitive, learning and inertia aspects. Acceptables moves are such that advantages to move than to stay are higher than some fraction of costs to move than to stay, with, as a result, a limitation of the intermediate sacrifices to reach the goal. When the agent is more goal-oriented and improves enough at each step, the process ends in a permanent routine, a rest point where the agent prefers to stay than to change, in spite of some possible residual frustration to have missed his goal. In case of high local costs to move this approach leads to a cognitive proof of Ekeland epsilon-variational principle. The as if hypothesis (as if agents would optimize) is revisited, leading to the introduction of a new class of optimization algorithm with inertia, the local search and proximal algorithms.

math.OC↗
arXiv2009

Asymptotic behavior of second-order dissipative evolution equations combining potential with non-potential effects

We study the asymptotic convergence properties, as the time variable goes to infinity, of trajectories of second-order dissipative evolution equations combining potential with non-potential effects. We exhibit a sharp condition, involving the damping parameter and the cocoercive coefficient of the non-potential operator, which guarantees convergence to equilibria of the trajectories. Applications are given to constrained optimization, fixed point problems, dynamical approach to Nash equilibria, and asymptotic stabilization in the case of a continuum of equilibria.

math.OC↗