Searcharxiv⌕ Search

arXiv subjects

Julien M. Hendrickx

Publications and source records attributed to Julien M. Hendrickx.

At least 37 records · Page 2Linked to original sources

A constraint-based approach to function interpolation, with application to performance estimation for weakly convex optimisation

We consider the problem of obtaining interpolation constraints for function classes, i.e., necessary and sufficient constraints that a set of points, function values and (sub)gradients must satisfy to ensure the existence of a global function of the class considered, consistent with this set. The derivation of such constraints is crucial, e.g., in the performance analysis of optimization methods, since obtaining a priori tight performance guarantees requires using a tight description of function classes of interest. We propose an approach that allows setting aside all analytic properties of the function class to work only at an algebraic level, and to obtain counterexamples when a condition characterizing a function class cannot serve as an interpolation constraint. As an illustration, we provide interpolation constraints for the class of weakly convex functions with bounded subgradients, and rely on these constraints to outperform state-of-the-art bounds on the performance of the subgradient method on this class.

math.OC↗

Several Performance Bounds on Decentralized Online Optimization are Highly Conservative and Potentially Misleading

We analyze Decentralized Online Optimization algorithms using the Performance Estimation Problem approach which allows, to automatically compute exact worst-case performance of optimization algorithms. Our analysis shows that several available performance guarantees are very conservative, sometimes by multiple orders of magnitude, and can lead to misguided choices of algorithm. Moreover, at least in terms of worst-case performance, some algorithms appear not to benefit from inter-agent communications for a significant period of time. We show how to improve classical methods by tuning their step-sizes, and find that we can save up to 20% on their actual worst-case performance regret.

math.OC↗

Performance Estimation of second-order optimization methods on classes of univariate functions

We develop a principled approach to obtain exact computer-aided worst-case guarantees on the performance of second-order optimization methods on classes of univariate functions. We first present a generic technique to derive interpolation conditions for a wide range of univariate functions, and use it to obtain such conditions for generalized self-concordant functions (including self-concordant and quasi-self-concordant functions) and functions with Lipschitz Hessian (both convex and non-convex). We then exploit these conditions within the Performance Estimation framework to tightly analyze the convergence of second-order methods on univariate functions, including (Cubic Regularized) Newton's method and several of its variants. Thereby, we improve on existing convergence rates, exhibit univariate lower bounds (that thus hold in the multivariate case), and analyze the performance of these methods with respect to the same criteria.

math.OC↗

Consensus on Open Multi-Agent Systems Over Graphs Sampled from Graphons

We show how graphons can be used to model and analyze open multi-agent systems, which are multi-agent systems subject to arrivals and departures, in the specific case of linear consensus. First, we analyze the case of replacements, where under the assumption of a deterministic interval between two replacements, we derive an upper bound for the disagreement in expectation. Then, we study the case of arrivals and departures, where we define a process for the evolution of the number of agents that guarantees a minimum and a maximum number of agents. Next, we derive an upper bound for the disagreement in expectation, and we establish a link with the spectrum of the expected graph used to generate the graph topologies. Finally, for stochastic block model (SBM) graphons, we prove that the computation of the spectrum of the expected graph can be performed based on a matrix whose dimension depends only on the graphon and it is independent of the number of agents.

eess.SY↗

Nonlinear identifiability of directed acyclic graphs with partial excitation and measurement

We analyze the identifiability of directed acyclic graphs in the case of partial excitation and measurement. We consider an additive model where the nonlinear functions located in the edges depend only on a past input, and we analyze the identifiability problem in the class of pure nonlinear functions satisfying $f(0)=0$. We show that any identification pattern (set of measured nodes and set of excited nodes) requires the excitation of sources, measurement of sinks and the excitation or measurement of the other nodes. Then, we show that a directed acyclic graph (DAG) is identifiable with a given identification pattern if and only if it is identifiable with the measurement of all the nodes. Next, we analyze the case of trees where we prove that any identification pattern guarantees the identifiability of the network. Finally, by introducing the notion of a generic nonlinear network matrix, we provide sufficient conditions for the identifiability of DAGs based on the notion of vertex-disjoint paths.

math.OC↗

Interpolation Conditions for Linear Operators and Applications to Performance Estimation Problems

The Performance Estimation Problem methodology makes it possible to determine the exact worst-case performance of an optimization method. In this work, we generalize this framework to first-order methods involving linear operators. This extension requires an explicit formulation of interpolation conditions for those linear operators. We consider the class of linear operators $\mathcal{M}:x \mapsto Mx$ where matrix $M$ has bounded singular values, and the class of linear operators where $M$ is symmetric and has bounded eigenvalues. We describe interpolation conditions for these classes, i.e. necessary and sufficient conditions that, given a list of pairs $\{(x_i,y_i)\}$, characterize the existence of a linear operator mapping $x_i$ to $y_i$ for all $i$. Using these conditions, we first identify the exact worst-case behavior of the gradient method applied to the composed objective $h\circ \mathcal{M}$, and observe that it always corresponds to $\mathcal{M}$ being a scaling operator. We then investigate the Chambolle-Pock method applied to $f+g\circ \mathcal{M}$, and improve the existing analysis to obtain a proof of the exact convergence rate of the primal-dual gap. In addition, we study how this method behaves on Lipschitz convex functions, and obtain a numerical convergence rate for the primal accuracy of the last iterate. We also show numerically that averaging iterates is beneficial in this setting.

math.OC↗

Comparison of Proximal First-Order Primal and Primal-Dual algorithms via Performance Estimation

Selecting the fastest algorithm for a specific signal/image processing task is a challenging question. We propose an approach based on the Performance Estimation Problem framework that numerically and automatically computes the worst-case performance of a given optimization method on a class of functions. We first propose a computer-assisted analysis and comparison of several first-order primal optimization methods, namely, the gradient method, the forward-backward, Peaceman-Rachford, and Douglas-Rachford splittings. We tighten the existing convergence results of these algorithms and extend them to new classes of functions. Our analysis is then extended and evaluated in the context of the primal-dual Chambolle-Pock and Condat-Vũ methods.

math.OC↗

On the Set of Possible Minimizers of a Sum of Convex Functions

Consider a sum of convex functions, where the only information known about each individual summand is the location of a minimizer. In this work, we give an exact characterization of the set of possible minimizers of the sum. Our results cover several types of assumptions on the summands, such as smoothness or strong convexity. Our main tool is the use of necessary and sufficient conditions for interpolating the considered function classes, which leads to shorter and more direct proofs in comparison with previous work. We also address the setting where each summand minimizer is assumed to lie in a unit ball, and prove a tight bound on the norm of any minimizer of the sum.

math.OC↗

On the Optimal Communication Weights in Distributed Optimization Algorithms

We establish that in distributed optimization, the prevalent strategy of minimizing the second-largest eigenvalue modulus (SLEM) of the averaging matrix for selecting communication weights, while optimal for existing theoretical performance bounds, is generally not optimal regarding the exact worst-case performance of the algorithms. This exact performance can be computed using the Performance Estimation Problem (PEP) approach. We thus rely on PEP to formulate an optimization problem that determines the optimal communication weights for a distributed optimization algorithm deployed on a specified undirected graph. Our results show that the optimal weights can outperform the weights minimizing the second-largest eigenvalue modulus (SLEM) of the averaging matrix. This suggests that the SLEM is not the best characterization of weighted network performance for decentralized optimization. Additionally, we explore and compare alternative heuristics for weight selection in distributed optimization.

math.OC↗

Interpolation Constraints for Computing Worst-Case Bounds in Performance Estimation Problems

The Performance Estimation Problem (PEP) approach consists in computing worst-case performance bounds on optimization algorithms by solving an optimization problem: one maximizes an error criterion over all initial conditions allowed and all functions in a given class of interest. The maximal value is then a worst-case bound, and the maximizer provides an example reaching that worst case. This approach was introduced for optimization algorithms but could in principle be applied to many other contexts involving worst-case bounds. The key challenge is the representation of infinite-dimensional objects involved in these optimization problems such as functions, and complex or non-convex objects as linear operators and their powers, networks in decentralized optimization etc. This challenge can be resolved by interpolation constraints, which allow representing the effect of these objects on vectors of interest, rather than the whole object, leading to tractable finite dimensional problems. We review several recent interpolation results and their implications in obtaining of worst-case bounds via PEP.

math.OC↗

Combinatorial Characterization for Global Identifiability of Separable Networks with Partial Excitation and Measurement

This work focuses on the generic identifiability of dynamical networks with partial excitation and measurement: a set of nodes are interconnected by transfer functions according to a known topology, some nodes are excited, some are measured, and only a part of the transfer functions are known. Our goal is to determine whether the unknown transfer functions can be generically recovered based on the input-output data collected from the excited and measured nodes. We introduce the notion of separable networks, for which global and so-called local identifiability are equivalent. A novel approach yields a necessary and sufficient combinatorial characterization for local identifiability for such graphs, in terms of existence of paths and conditions on their parity. Furthermore, this yields a necessary condition not only for separable networks, but for networks of any topology.

math.OC↗

Random Coordinate Descent for Resource Allocation in Open Multi-Agent Systems

We propose a method for analyzing the distributed random coordinate descent algorithm for solving separable resource allocation problems in the context of an open multiagent system, where agents can be replaced during the process. In particular, we characterize the evolution of the distance to the minimizer in expectation by following a time-varying optimization approach which builds on two components. First, we establish the linear convergence of the algorithm in closed systems, in terms of the estimate towards the minimizer, for general graphs and appropriate step-size. Second, we estimate the change of the optimal solution after a replacement, in order to evaluate its effect on the distance between the current estimate and the minimizer. From these two elements, we derive stability conditions in open systems and establish the linear convergence of the algorithm towards a steady-state expected error. Our results enable to characterize the trade-off between speed of convergence and robustness to agent replacements, under the assumptions that local functions are smooth, strongly convex, and have their minimizers located in a given ball. The approach proposed in this paper can moreover be extended to other algorithms guaranteeing linear convergence in closed system.

cs.MA↗

Nonlinear network identifiability: The static case

We analyze the problem of network identifiability with nonlinear functions associated with the edges. We consider a static model for the output of each node and by assuming a perfect identification of the function associated with the measurement of a node, we provide conditions for the identifiability of the edges in a specific class of functions. First, we analyze the identifiability conditions in the class of all nonlinear functions and show that even for a path graph, it is necessary to measure all the nodes except by the source. Then, we consider analytic functions satisfying $f(0)=0$ and we provide conditions for the identifiability of paths and trees. Finally, by restricting the problem to a smaller class of functions where none of the functions is linear, we derive conditions for the identifiability of directed acyclic graphs. Some examples are presented to illustrate the results.

math.OC↗

Trajectory Convergence from Coordinate-wise Decrease of General Energy Functions

We consider arbitrary trajectories subject to a coordinate-wise energy decrease: the sign of the derivative of each entry is never the same as that of the corresponding entry of the gradient of some energy function. We show that this simple condition guarantees convergence to a point, to the minimum of the energy functions, or to a set where its Hessian has very specific properties. This extends and strengthens recent results that were restricted to convex quadratic energy functions. We demonstrate the application of our result by using it to prove the convergence of a class of multi-agent systems subject to multiple uncertainties.

math.OC↗

Improved Mixing Rates of Directed Cycles with Additional Sparse Interconnections

We analyze the absolute spectral gap of Markov chains on graphs obtained from a cycle of $n$ vertices and perturbed only at approximately $n^{1/ρ}$ random locations with an appropriate, possibly sparse, interconnection structure. Together with a strong asymmetry along the cycle, the gap of the resulting chain can be bounded inversely proportionally by the longest arc length (up to logarithmic factors) with high probability, providing a significant mixing speedup compared to the reversible version.

math.PR↗

Automatic Performance Estimation for Decentralized Optimization

We present a methodology to automatically compute worst-case performance bounds for a large class of first-order decentralized optimization algorithms. These algorithms aim at minimizing the average of local functions that are distributed across a network of agents. They typically combine local computations and consensus steps. Our methodology is based on the approach of Performance Estimation Problem (PEP), which allows computing the worst-case performance and a worst-case instance of first-order optimization algorithms by solving an SDP. We propose two ways of representing consensus steps in PEPs, which allow writing and solving PEPs for decentralized optimization. The first formulation is exact but specific to a given averaging matrix. The second formulation is a relaxation but provides guarantees valid over an entire class of averaging matrices, characterized by their spectral range. This formulation often allows recovering a posteriori the worst possible averaging matrix for the given algorithm. We apply our methodology to three different decentralized methods. For each of them, we obtain numerically tight worst-case performance bounds that significantly improve on the existing ones, as well as insights about the parameters tuning and the worst communication networks.

math.OC↗

Convergence, Consensus and Dissensus in the Weighted-Median Opinion Dynamics

Mechanistic and tractable mathematical models play a key role in understanding how social influence shapes public opinions. Recently, a weighted-median mechanism has been proposed as a new micro-foundation of opinion dynamics and validated via experimental data. Numerical studies also indicate that this new mechanism recreates some non-trivial real-world features of opinion evolution. In this paper, we conduct a thorough theoretical analysis of the weighted-median opinion dynamics. We fully characterize the set of all equilibria, and we establish the almost-sure finite-time convergence for any initial condition. Moreover, we prove a necessary and sufficient graph-theoretic condition for the almost-sure convergence to consensus, as well as a sufficient graph-theoretic condition for almost-sure persistent dissensus. It turns out that the weighted-median opinion dynamics, despite its simplicity in form, exhibit rich dynamical behavior that depends on some delicate network structures. To complement our sufficient conditions for almost-sure dissensus, we further prove that, given the influence network, determining whether the system almost surely achieves persistent dissensus is NP-hard, which reflects the complexity the network topology contributes to opinion evolution.

eess.SY↗

On the Worst-Case Analysis of Cyclic Coordinate-Wise Algorithms on Smooth Convex Functions

We propose a unifying framework for the automated computer-assisted worst-case analysis of cyclic block coordinate algorithms in the unconstrained smooth convex optimization setup. We compute exact worst-case bounds for the cyclic coordinate descent and the alternating minimization algorithms over the class of smooth convex functions, and provide sublinear upper and lower bounds on the worst-case rate for the standard class of functions with coordinate-wise Lipschitz gradients. We obtain in particular a new upper bound for cyclic coordinate descent that outperforms the best available ones by an order of magnitude. We also demonstrate the flexibility of our approach by providing new numerical bounds using simpler and more natural assumptions than those normally made for the analysis of block coordinate algorithms. Finally, we provide numerical evidence for the fact that a standard scheme that provably accelerates random coordinate descent to a $O(1/k^2)$ complexity is actually inefficient when used in a (deterministic) cyclic algorithm.

math.OC↗