arXiv subjects

Victor Magron

Publications and source records attributed to Victor Magron.

At least 73 records · Page 4Linked to original sources

arXiv2022

Stability Analysis of Recurrent Neural Networks by IQC with Copositive Mutipliers

This paper is concerned with the stability analysis of the recurrent neural networks (RNNs) by means of the integral quadratic constraint (IQC) framework. The rectified linear unit (ReLU) is typically employed as the activation function of the RNN, and the ReLU has specific nonnegativity properties regarding its input and output signals. Therefore, it is effective if we can derive IQC-based stability conditions with multipliers taking care of such nonnegativity properties. However, such nonnegativity (linear) properties are hardly captured by the existing multipliers defined on the positive semidefinite cone. To get around this difficulty, we loosen the standard positive semidefinite cone to the copositive cone, and employ copositive multipliers to capture the nonnegativity properties. We show that, within the framework of the IQC, we can employ copositive multipliers (or their inner approximation) together with existing multipliers such as Zames-Falb multipliers and polytopic bounding multipliers, and this directly enables us to ensure that the introduction of the copositive multipliers leads to better (no more conservative) results. We finally illustrate the effectiveness of the IQC-based stability conditions with the copositive multipliers by numerical examples.

math.OC↗

arXiv2022

Noncommutative Christoffel-Darboux Kernels

We introduce from an analytic perspective Christoffel-Darboux kernels associated to bounded, tracial noncommutative distributions. We show that properly normalized traces, respectively norms, of evaluations of such kernels on finite dimensional matrices yield classical plurisubharmonic functions as the degree tends to infinity, and show that they are comparable to certain noncommutative versions of the Siciak extremal function. We prove estimates for Siciak functions associated to free products of distributions, and use the classical theory of plurisubharmonic functions in order to propose a notion of support for noncommutative distributions. We conclude with some conjectures and numerical experiments.

math.OA↗

arXiv2021

Noncommutative Polynomial Optimization

In this chapter we present the sums of Hermitian squares approach to noncommutative polynomial optimization problems. This is an extension of the sums of squares approach for polynomial optimization arising from real algebraic geometry. We provide a gentle introduction to the underlying theory of this methodology and highlight its importance.

math.OC↗

arXiv2021

Exact Optimization via Sums of Nonnegative Circuits and Sums of AM/GM Exponentials

We provide two hybrid numeric-symbolic optimization algorithms, computing exact sums of nonnegative circuits (SONC) and sums of arithmetic-geometric-exponentials (SAGE) decompositions. Moreover, we provide a hybrid numeric-symbolic decision algorithm for polynomials lying in the interior of the SAGE cone. Each framework, inspired by previous contributions of Parrilo and Peyrl, is a rounding-projection procedure. For a polynomial lying in the interior of the SAGE cone, we prove that the decision algorithm terminates within a number of arithmetic operations, which is polynomial in the number of terms of the input, and linear in the distance to the boundary of the cone. We also provide experimental comparisons regarding the implementation of the two optimization algorithms.

cs.SC↗

arXiv2021

Sum of Squares Decompositions of Polynomials over their Gradient Ideals with Rational Coefficients

Assessing non-negativity of multivariate polynomials over the reals, through the computation of {\em certificates of non-negativity}, is a topical issue in polynomial optimization. This is usually tackled through the computation of {\em sums-of-squares decompositions} which rely on efficient numerical solvers for semi-definite programming. This method faces two difficulties. The first one is that the certificates obtained this way are {\em approximate} and then non-exact. The second one is due to the fact that not all non-negative polynomials are sums-of-squares. In this paper, we build on previous works by Parrilo, Nie, Demmel and Sturmfels who introduced certificates of non-negativity modulo {\em gradient ideals}. We prove that, actually, such certificates can be obtained {\em exactly}, over the rationals if the polynomial under consideration has rational coefficients and we provide {\em exact} algorithms to compute them. We analyze the bit complexity of these algorithms and deduce bit size bounds of such certificates.

cs.SC↗

arXiv2021

CS-TSSOS: Correlative and term sparsity for large-scale polynomial optimization

This work proposes a new moment-SOS hierarchy, called CS-TSSOS, for solving large-scale sparse polynomial optimization problems. Its novelty is to exploit simultaneously correlative sparsity and term sparsity by combining advantages of two existing frameworks for sparse polynomial optimization. The former is due to Waki et al. while the latter was initially proposed by Wang et al. and later exploited in the TSSOS hierarchy. In doing so we obtain CS-TSSOS -- a two-level hierarchy of semidefinite programming relaxations with (i), the crucial property to involve blocks of SDP matrices and (ii), the guarantee of convergence to the global optimum under certain conditions. We demonstrate its efficiency and scalability on several large-scale instances of the celebrated Max-Cut problem and the important industrial optimal power flow problem, involving up to six thousand variables and tens of thousands of constraints.

math.OC↗

arXiv2021

Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification

Deep equilibrium models are based on implicitly defined functional relations and have shown competitive performance compared with the traditional deep networks. Monotone operator equilibrium networks (monDEQ) retain interesting performance with additional theoretical guaranties. Existing certification tools for classical deep networks cannot directly be applied to monDEQs for which much fewer tools exist. We introduce a semialgebraic representation for ReLU based monDEQs which allows to approximate the corresponding input output relation by semidefinite programming (SDP). We present several applications to network certification and obtain SDP models for the following problems : robustness certification, Lipschitz constant estimation, ellipsoidal uncertainty propagation. We use these models to certify robustness of monDEQs w.r.t. a general $L_q$ norm. Experimental results show that the proposed models outperform existing approaches for monDEQ certification. Furthermore, our investigations suggest that monDEQs are much more robust to $L_2$ perturbations than $L_{\infty}$ perturbations.

math.OC↗

arXiv2021

On the complexity of Putinar-Vasilescu's Positivstellensatz

We provide a new degree bound on the weighted sum-of-squares (SOS) polynomials for Putinar-Vasilescu's Positivstellensatz. This leads to another Positivstellensatz saying that if $f$ is a polynomial of degree at most $2 d_f$ nonnegative on a semialgebraic set having nonempty interior defined by finitely many polynomial inequalities $g_j(x)\ge 0$, $j=1,\dots,m$ with $g_1:=L-\|x\|_2^2$ for some $L>0$, then there exist positive constants $\bar c$ and $c$ depending on $f,g_j$ such that for any $\varepsilon>0$, for all $k\ge \bar c \varepsilon^{-c}$, $f$ has the decomposition \begin{equation} \begin{array}{l} (1+\|x\|_2^2)^k(f+\varepsilon)=σ_0+\sum_{j=1}^m σ_jg_j \,, \end{array} \end{equation} for some SOS polynomials $σ_j$ being such that the degrees of $σ_0,σ_jg_j$ are at most $2(d_f+k)$. Here $\|\cdot\|_2$ denotes the $\ell_2$ vector norm. As a consequence, we obtain a converging hierarchy of semidefinite relaxations for lower bounds in polynomial optimization on basic compact semialgebraic sets. The complexity of this hierarchy is $\mathcal{O}(\varepsilon^{-c})$ for prescribed accuracy $\varepsilon>0$. In particular, if $m=L=1$ then $c=65$, yielding the complexity $\mathcal{O}(\varepsilon^{-65})$ for the minimization of a polynomial on the unit ball. Our result improves the complexity bound $\mathcal{O}(\exp(\varepsilon^{-c}))$ due to Nie and Schweighofer in [Journal of Complexity 23.1 (2007): 135-150].

math.OC↗

arXiv2021

Optimization over trace polynomials

Motivated by recent progress in quantum information theory, this article aims at optimizing trace polynomials, i.e., polynomials in noncommuting variables and traces of their products. A novel Positivstellensatz certifying positivity of trace polynomials subject to trace constraints is presented, and a hierarchy of semidefinite relaxations converging monotonically to the optimum of a trace polynomial subject to tracial constraints is provided. This hierarchy can be seen as a tracial analog of the Pironio, Navascués and Acín scheme [New J. Phys., 2008] for optimization of noncommutative polynomials. The Gelfand-Naimark-Segal (GNS) construction is applied to extract optimizers of the trace optimization problem if flatness and extremality conditions are satisfied. These conditions are sufficient to obtain finite convergence of our hierarchy. The results obtained are applied to violations of polynomial Bell inequalities in quantum information theory. The main techniques used in this paper are inspired by real algebraic geometry, operator theory, and noncommutative algebra.

math-ph↗

arXiv2021

SparseJSR: A Fast Algorithm to Compute Joint Spectral Radius via Sparse SOS Decompositions

This paper focuses on the computation of joint spectral radii (JSR), when the involved matrices are sparse. We provide a sparse variant of the procedure proposed by Parrilo and Jadbabaie, to compute upper bounds of the JSR by means of sum-of-squares (SOS) relaxations. Our resulting iterative algorithm, called SparseJSR, is based on the term sparsity SOS (TSSOS) framework, developed by Wang, Magron and Lasserre, yielding SOS decompositions of polynomials with arbitrary sparse support. SparseJSR exploits the sparsity of the input matrices to significantly reduce the computational burden associated with the JSR computation. Our algorithmic framework is then successfully applied to compute upper bounds for JSR, on randomly generated benchmarks as well as on problems arising from stability proofs of controllers, in relation with possible hardware and software faults.

math.OC↗

arXiv2021

TSSOS: a Julia library to exploit sparsity for large-scale polynomial optimization

The Julia library TSSOS aims at helping polynomial optimizers to solve large-scale problems with sparse input data. The underlying algorithmic framework is based on exploiting correlative and term sparsity to obtain a new moment-SOS hierarchy involving potentially much smaller positive semidefinite matrices. TSSOS can be applied to numerous problems ranging from power networks to eigenvalue and trace optimization of noncommutative polynomials, involving up to tens of thousands of variables and constraints.

math.OC↗

arXiv2021

The Constant Trace Property in Noncommutative Optimization

In this article, we show that each semidefinite relaxation of a ball-constrained noncommutative polynomial optimization problem can be cast as a semidefinite program with a constant trace matrix variable. We then demonstrate how this constant trace property can be exploited via first order numerical methods to solve efficiently the semidefinite relaxations of the noncommutative problem.

math.OC↗

arXiv2021

A Sublevel Moment-SOS Hierarchy for Polynomial Optimization

We introduce a sublevel Moment-SOS hierarchy where each SDP relaxation can be viewed as an intermediate (or interpolation) between the d-th and (d+1)-th order SDP relaxations of the Moment-SOS hierarchy (dense or sparse version). With the flexible choice of determining the size (level) and number (depth) of subsets in the SDP relaxation, one is able to obtain different improvements compared to the d-th order relaxation, based on the machine memory capacity. In particular, we provide numerical experiments for d=1 and various types of problems both in combinatorial optimization (Max-Cut, Mixed Integer Programming) and deep learning (robustness certification, Lipschitz constant of neural networks), where the standard Lasserre's relaxation (or its sparse variant) is computationally intractable. In our numerical results, the lower bounds from the sublevel relaxations improve the bound from Shor's relaxation (first order Lasserre's relaxation) and are significantly closer to the optimal value or to the best-known lower/upper bounds.

math.OC↗

arXiv2020

Exploiting constant trace property in large-scale polynomial optimization

We prove that every semidefinite moment relaxation of a polynomial optimization problem (POP) with a ball constraint can be reformulated as a semidefinite program involving a matrix with constant trace property (CTP). As a result such moment relaxations can be solved efficiently by first-order methods that exploit CTP, e.g., the conditional gradient-based augmented Lagrangian method. We also extend this CTP-exploiting framework to large-scale POPs with different sparsity structures. The efficiency and scalability of our framework are illustrated on second-order moment relaxations for various randomly generated quadratically constrained quadratic programs.

math.OC↗

arXiv2020

Minimizing rational functions: a hierarchy of approximations via pushforward measures

This paper is concerned with minimizing a sum of rational functions over a compact set of high-dimension. Our approach relies on the second Lasserre's hierarchy (also known as the upper bounds hierarchy) formulated on the pushforward measure in order to work in a space of smaller dimension. We show that in the general case the minimum can be approximated as closely as desired from above with a hierarchy of semidefinite programs problems or, in the particular case of a single fraction, with a hierarchy of generalized eigenvalue problems. We numerically illustrate the potential of using the pushforward measure rather than the standard upper bounds hierarchy. In our opinion, this potential should be a strong incentive to investigate a related challenging problem interesting in its own; namely integrating an arbitrary power of a given polynomial on a simple set (e.g., unit box or unit sphere) with respect to Lebesgue or Haar measure.

math.OC↗

arXiv2020

$l_2$ Induced Norm Analysis of Discrete-Time LTI Systems for Nonnegative Input Signals and Its Application to Stability Analysis of Recurrent Neural Networks

In this paper, we focus on the "positive" $l_2$ induced norm of discrete-time linear time-invariant systems where the input signals are restricted to be nonnegative. To cope with the nonnegativity of the input signals, we employ copositive programming as the mathematical tool for the analysis. Then, by applying an inner approximation to the copositive cone, we derive numerically tractable semidefinite programming problems for the upper and lower bound computation of the "positive" $l_2$ induced norm. This norm is typically useful for the stability analysis of feedback systems constructed from an LTI system and nonlinearities where the nonlinear elements provide only nonnegative signals. As a concrete example, we illustrate the usefulness of the "positive" $l_2$ induced norm for the stability analysis of recurrent neural networks with activation functions being rectified linear units.

math.OC↗

arXiv2020

Semialgebraic Optimization for Lipschitz Constants of ReLU Networks

The Lipschitz constant of a network plays an important role in many applications of deep learning, such as robustness certification and Wasserstein Generative Adversarial Network. We introduce a semidefinite programming hierarchy to estimate the global and local Lipschitz constant of a multiple layer deep neural network. The novelty is to combine a polynomial lifting for ReLU functions derivatives with a weak generalization of Putinar's positivity certificate. This idea could also apply to other, nearly sparse, polynomial optimization problems in machine learning. We empirically demonstrate that our method provides a trade-off with respect to state of the art linear programming approach, and in some cases we obtain better bounds in less time.

math.OC↗

arXiv2020

Exploiting term sparsity in Noncommutative Polynomial Optimization

We provide a new hierarchy of semidefinite programming relaxations, called NCTSSOS, to solve large-scale sparse noncommutative polynomial optimization problems. This hierarchy features the exploitation of term sparsity hidden in the input data for eigenvalue and trace optimization problems. NCTSSOS complements the recent work that exploits correlative sparsity for noncommutative optimization problems by Klep, Magron and Povh in arXiv:1909.00569, and is the noncommutative analogue of the TSSOS framework by Wang, Magron and Lasserre in arXiv:1912.08899. We also propose an extension exploiting simultaneously correlative and term sparsity, as done previously in the commutative case arXiv:2005.02828. Under certain conditions, we prove that the optimums of the NCTSSOS hierarchy converge to the optimum of the corresponding dense SDP relaxation. We illustrate the efficiency and scalability of NCTSSOS by solving eigenvalue/trace optimization problems from the literature as well as randomly generated examples involving up to several thousands of variables.

math.OC↗