arXiv subjects
Publications and source records attributed to Patrick L. Combettes.
Olivier's 1827 theorem provides a rate of convergence to zero of the general term of a decreasing summable sequence of positive reals. We derive stochastic extensions of this result in the context of almost supermartingales. The results are applied to the analysis of stochastic iterative processes.
We propose an abstract stochastic scheme for solving a broad range of monotone operator inclusion problems in Hilbert spaces. This framework allows for the introduction of stochasticity at several levels in monotone operator splitting methods: approximation of operators, selection of coordinates and operators in block-iterative implementations, and relaxation parameters. The analysis involves an abstract reduced inclusion model with two operators. At each iteration of the proposed scheme, stochastic approximations to points in the graphs of these two operators are used to form the update. The results are applied to derive the almost sure and $L^2$ convergence of stochastic versions of the proximal point algorithm, as well as of randomized block-iterative projective splitting methods for solving systems of coupled inclusions involving a mix of set-valued, cocoercive, and Lipschitzian monotone operators combined via various monotonicity-preserving operations.
The Haraux function is an important tool in monotone operator theory and its applications. One of its salient properties for a maximally monotone operator is to be valued in $[0,+\infty]$ and to vanish only on the graph of the operator. Sharper lower bounds for this function have been proposed in specific cases. We derive lower bounds in the general context of set-valued operators in reflexive real Banach spaces. These bounds are new, even for maximally monotone operators acting on Euclidean spaces, a scenario in which we show that they can be better than existing ones. As a by-product, we obtain lower bounds on the Fenchel--Young function in variational analysis. Several examples are given and applications to composite monotone inclusions are discussed.
This paper concerns models and convergence principles for dealing with stochasticity in a wide range of algorithms arising in nonlinear analysis and optimization in Hilbert spaces. It proposes a flexible geometric framework within which existing solution methods can be recast and improved, and new ones can be designed. Almost sure weak, strong, and linear convergence results are established in particular for stochastic fixed point iterations, the stochastic gradient descent method, and stochastic extrapolated parallel algorithms for feasibility problems. In these areas, the proposed algorithms exceed the features and convergence guarantees of the state of the art. Numerical applications to signal and image recovery are provided.
The proximal gradient method is a splitting algorithm for the minimization of the sum of two convex functions, one of which is smooth. It has applications in areas such as mechanics, inverse problems, machine learning, image reconstruction, variational inequalities, statistics, operations research, and optimal transportation. Its formalism encompasses a wide variety of numerical methods in optimization such as gradient descent, projected gradient, iterative thresholding, alternating projections, the constrained Landweber method, as well as various algorithms in statistics and sparse data analysis. This paper aims at providing an account of the main properties of the proximal gradient method and to discuss some of its applications. --- La m\'ethode du gradient proxim\'e est un algorithme d'\'eclatement pour la minimisation de la somme de deux fonctions convexes, dont l'une est lisse. Elle trouve des applications dans des domaines tels que la m\'ecanique, le traitement du signal, les probl\`emes inverses, l'apprentissage automatique, la reconstruction d'images, les in\'equations variationnelles, les statistiques, la recherche op\'erationnelle et le transport optimal. Son formalisme englobe une grande vari\'et\'e de m\'ethodes num\'eriques en optimisation, telles que la descente de gradient, le gradient projet\'e, la m\'ethode de seuillage it\'eratif, la m\'ethode des projections altern\'ees, la m\'ethode de Landweber contrainte, ainsi que divers algorithmes en statistique et en analyse parcimonieuse de donn\'ees. Cette synth\`ese vise \`a donner un aper\c{c}u des principales propri\'et\'es de la m\'ethode du gradient proxim\'e et d'aborder certaines de ses applications.
This paper establishes various variational properties of parametrized versions of two convexity-preserving constructs that were recently introduced in the literature: the proximal composition of a function and a linear operator, and the integral proximal mixture of arbitrary families of functions and linear operators. We study in particular convexity, Legendre conjugacy, differentiability, Moreau envelopes, coercivity, minimizers, recession functions, and perspective functions of these constructs, as well as their asymptotic behavior as the parameter varies. The special case of the proximal expectation of a family of functions is also discussed.
We propose stochastic splitting algorithms for solving large-scale composite inclusion problems involving monotone and linear operators. They activate at each iteration blocks of randomly selected resolvents of monotone operators and, unlike existing methods, achieve almost sure convergence of the iterates to a solution without any regularity assumptions or knowledge of the norms of the linear operators. Applications to image recovery and machine learning are provided.
In minimization models for image recovery and data analysis problems, loss functions and linear operators are typically aggregated as an average of composite terms. Each term in the aggregate models a desired property of the ideal solution arising from the \emph{a priori} knowledge and the observed data. We propose an alternative minimization model based on proximal comixtures, an operation which combines functions and linear operators in such a way that the proximity operator of the resulting function is computable explicitly in terms of the individual proximity and linear operators. The mathematical properties of this operation are analyzed and comparisons between proximal comixtures and standard composite averages are made. Numerical illustrations of the benefits of minimization models based on proximal comixtures are provided in the context of image recovery and machine learning applications.
Using the theory of Hilbert direct integrals, we introduce and study a monotonicity-preserving operation, termed the integral resolvent mixture. It combines arbitrary families of monotone operators acting on different spaces and linear operators. As a special case, we investigate the resolvent expectation, an operation which combines monotone operators in such a way that the resulting resolvent is the Lebesgue expectation of the individual resolvents. Along the same lines, we introduce an operation that mixes arbitrary families of convex functions defined on different spaces and linear operators to create a composite convex function. Such constructs have so far been limited to finite families of operators and functions. The subdifferential of the integral proximal mixture is shown to be the integral resolvent mixture of the individual subdifferentials. Applications to the relaxation of systems of composite monotone inclusions are presented.
Finite Cartesian products of operators play a central role in monotone operator theory and its applications. Extending such products to arbitrary families of operators acting on different Hilbert spaces is an open problem, which we address by introducing the Hilbert direct integral of a family of monotone operators. The properties of this construct are studied and conditions under which the direct integral inherits the properties of the factor operators are provided. The question of determining whether the Hilbert direct integral of a family of subdifferentials of convex functions is itself a subdifferential leads us to introducing the Hilbert direct integral of a family of functions. We establish explicit expressions for evaluating the Legendre conjugate, subdifferential, recession function, Moreau envelope, and proximity operator of such integrals. Next, we propose a duality framework for monotone inclusion problems involving integrals of linearly composed monotone operators and show its pertinence towards the development of numerical solution methods. Applications to inclusion and variational problems are discussed.
We propose a geometric framework to describe and analyze a wide array of operator splitting methods for solving monotone inclusion problems. The initial inclusion problem, which typically involves several operators combined through monotonicity-preserving operations, is seldom solvable in its original form. We embed it in an auxiliary space, where it is associated with a surrogate monotone inclusion problem with a more tractable structure and which allows for easy recovery of solutions to the initial problem. The surrogate problem is solved by successive projections onto half-spaces containing its solution set. The outer approximation half-spaces are constructed by using the individual operators present in the model separately. This geometric framework is shown to encompass traditional methods as well as state-of-the-art asynchronous block-iterative algorithms, and its flexible structure provides a pattern to design new ones.
We introduce the resolvent composition, a monotonicity-preserving operation between a linear operator and a set-valued operator, as well as the proximal composition, a convexity-preserving operation between a linear operator and a function. The two operations are linked by the fact that, under mild assumptions, the subdifferential of the proximal composition of a convex function is the resolvent composition of its subdifferential. The resolvent and proximal compositions are shown to encapsulate known concepts, such as the resolvent and proximal averages, as well as new operations pertinent to the analysis of equilibrium problems. A large core of properties of these compositions is established and several instantiations are discussed. Applications to the relaxation of monotone inclusion and convex optimization problems are presented.
We introduce a framework based on Rockafellar's perturbation theory to analyze and solve general nonsmooth convex minimization and monotone inclusion problems involving nonlinearly composed functions as well as linear compositions. Such problems have been investigated only from a primal perspective and only for nonlinear compositions of smooth functions in finite-dimensional spaces in the absence of linear compositions. In the context of Banach spaces, the proposed perturbation analysis serves as a foundation for the construction of a dual problem and of a maximally monotone Kuhn--Tucker operator which is decomposable as the sum of simpler monotone operators. In the Hilbertian setting, this decomposition leads to a block-iterative primal-dual algorithm that fully splits all the components of the problem and appears to be the first proximal splitting algorithm for handling nonlinear composite problems. Various applications are discussed.
We first present an abstract principle for the interchange of infimization and integration over spaces of mappings taking values in topological spaces. New conditions on the underlying space and the integrand are then introduced to convert this principle into concrete scenarios that are shown to capture those of various existing interchange rules. These results are leveraged to improve state-of-the-art interchange rules for evaluating Legendre conjugates, subdifferentials, recessions, Moreau envelopes, and proximity operators of integral functions by bringing the corresponding operations under the integral sign.
A perspective function is a construction which combines a base function defined on a given space with a nonlinear scaling function defined on another space and which yields a lower semicontinuous convex function on the product space. Since perspective functions are typically nonsmooth, their use in first-order algorithms necessitates the computation of their proximity operator. This paper establishes closed-form expressions for the proximity operator of a perspective function defined on a Hilbert space in terms of a proximity operator involving its base function and one involving its scaling function.
The classical perspective of a function is a construction which transforms a convex function into one that is jointly convex with respect to an auxiliary scaling variable. Motivated by applications in several areas of applied analysis, we investigate an extension of this construct in which the scaling variable is replaced by a nonlinear term. Our construction is placed in the general context of locally convex spaces and it generates a lower semicontinuous convex function under broad assumptions on the underlying functions. Various convex-analytical properties are established and closed-form expressions are derived. Several applications are presented.
Neural networks have become ubiquitous tools for solving signal and image processing problems, and they often outperform standard approaches. Nevertheless, training neural networks is a challenging task in many applications. The prevalent training procedure consists of minimizing highly non-convex objectives based on data sets of huge dimension. In this context, current methodologies are not guaranteed to produce global solutions. We present an alternative approach which foregoes the optimization framework and adopts a variational inequality formalism. The associated algorithm guarantees convergence of the iterates to a true solution of the variational inequality and it possesses an efficient block-iterative structure. A numerical application is presented.
We investigate a modular convex Nash equilibrium problem involving nonsmooth functions acting on linear mixtures of strategies, as well as smooth coupling functions. An asynchronous block-iterative decomposition method is proposed to solve it.