arXiv subjects

Alexander Davydov

Publications and source records attributed to Alexander Davydov.

32 records · Page 2Linked to original sources

arXiv2023

Contractivity of Distributed Optimization and Nash Seeking Dynamics

In this letter, we study distributed optimization and Nash equilibrium-seeking dynamics from a contraction theoretic perspective. Our first result is a novel bound on the logarithmic norm of saddle matrices. Second, for distributed gradient flows based upon incidence and Laplacian constraints over arbitrary topologies, we establish strong contractivity over an appropriate invariant vector subspace. Third, we give sufficient conditions for strong contractivity in pseudogradient and best response games with complete information, show the equivalence of these conditions, and consider the special case of aggregative games.

math.OC↗

arXiv2023

Euclidean Contractivity of Neural Networks with Symmetric Weights

This paper investigates stability conditions of continuous-time Hopfield and firing-rate neural networks by leveraging contraction theory. First, we present a number of useful general algebraic results on matrix polytopes and products of symmetric matrices. Then, we give sufficient conditions for strong and weak Euclidean contractivity, i.e., contractivity with respect to the $\ell_2$ norm, of both models with symmetric weights and (possibly) non-smooth activation functions. Our contraction analysis leads to contraction rates which are log-optimal in almost all symmetric synaptic matrices. Finally, we use our results to propose a firing-rate neural network model to solve a quadratic optimization problem with box constraints.

math.OC↗

arXiv2023

The Yakubovich S-Lemma Revisited: Stability and Contractivity in Non-Euclidean Norms

The celebrated S-Lemma was originally proposed to ensure the existence of a quadratic Lyapunov function in the Lur'e problem of absolute stability. A quadratic Lyapunov function is, however, nothing else than a squared Euclidean norm on the state space (that is, a norm induced by an inner product). A natural question arises as to whether squared non-Euclidean norms $V(x)=\|x\|^2$ may serve as Lyapunov functions in stability problems. This paper presents a novel non-polynomial S-Lemma that leads to constructive criteria for the existence of such functions defined by weighted $\ell_p$ norms. Our generalized S-Lemma leads to new absolute stability and absolute contractivity criteria for Lur'e-type systems, including, for example, a new simple proof of the Aizerman and Kalman conjectures for positive Lur'e systems.

math.OC↗

arXiv2022

Non-Euclidean Contraction Theory for Monotone and Positive Systems

In this note we study contractivity of monotone systems and exponential convergence of positive systems using non-Euclidean norms. We first introduce the notion of conic matrix measure as a framework to study stability of monotone and positive systems. We study properties of the conic matrix measures and investigate their connection with weak pairings and standard matrix measures. Using conic matrix measures and weak pairings, we characterize contractivity and incremental stability of monotone systems with respect to non-Euclidean norms. Moreover, we use conic matrix measures to provide sufficient conditions for exponential convergence of positive systems to their equilibria. We show that our framework leads to novel results on (i) the contractivity of excitatory Hopfield neural networks, and (ii) the stability of interconnected systems using non-monotone positive comparison systems.

math.OC↗

arXiv2022

Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach

This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks based upon non-Euclidean contraction theory. The basic idea is to cast the robustness analysis of a neural network as a reachability problem and use (i) the $\ell_{\infty}$-norm input-output Lipschitz constant and (ii) the tight inclusion function of the network to over-approximate its reachable sets. First, for a given implicit neural network, we use $\ell_{\infty}$-matrix measures to propose sufficient conditions for its well-posedness, design an iterative algorithm to compute its fixed points, and provide upper bounds for its $\ell_\infty$-norm input-output Lipschitz constant. Second, we introduce a related embedded network and show that the embedded network can be used to provide an $\ell_\infty$-norm box over-approximation of the reachable sets of the original network. Moreover, we use the embedded network to design an iterative algorithm for computing the upper bounds of the original system's tight inclusion function. Third, we use the upper bounds of the Lipschitz constants and the upper bounds of the tight inclusion functions to design two algorithms for the training and robustness verification of implicit neural networks. Finally, we apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.

cs.LG↗

arXiv2022

Non-Euclidean Contraction Theory for Robust Nonlinear Stability

We study necessary and sufficient conditions for contraction and incremental stability of dynamical systems with respect to non-Euclidean norms. First, we introduce weak pairings as a framework to study contractivity with respect to arbitrary norms, and characterize their properties. We introduce and study the sign and max pairings for the $\ell_1$ and $\ell_\infty$ norms, respectively. Using weak pairings, we establish five equivalent characterizations for contraction, including the one-sided Lipschitz condition for the vector field as well as matrix measure and Demidovich conditions for the corresponding Jacobian. Third, we extend our contraction framework in two directions: we prove equivalences for contraction of continuous vector fields and we formalize the weaker notion of equilibrium contraction, which ensures exponential convergence to an equilibrium. Finally, as an application, we provide (i) incremental input-to-state stability and finite input-state gain properties for contracting systems, and (ii) a general theorem about the Lipschitz interconnection of contracting systems, whereby the Hurwitzness of a gain matrix implies the contractivity of the interconnected system.

math.OC↗

arXiv2022

Non-Euclidean Monotone Operator Theory with Applications to Recurrent Neural Networks

We provide a novel transcription of monotone operator theory to the non-Euclidean finite-dimensional spaces $\ell_1$ and $\ell_{\infty}$. We first establish properties of mappings which are monotone with respect to the non-Euclidean norms $\ell_1$ or $\ell_{\infty}$. In analogy with their Euclidean counterparts, mappings which are monotone with respect to a non-Euclidean norm are amenable to numerous algorithms for computing their zeros. We demonstrate that several classic iterative methods for computing zeros of monotone operators are directly applicable in the non-Euclidean framework. We present a case-study in the equilibrium computation of recurrent neural networks and demonstrate that casting the computation as a suitable operator splitting problem improves convergence rates.

math.OC↗

arXiv2022

Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks

We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs). INNs are a class of implicit learning models that use implicit equations as layers and have been shown to exhibit several notable benefits over traditional deep neural networks. We first establish that tight inclusion functions of neural networks, which provide the tightest rectangular over-approximation of an input-output map, lead to sharper robustness guarantees than the well-studied robustness measures of local Lipschitz constants. Like Lipschitz constants, tight inclusions functions are computationally challenging to obtain, and we thus propose using mixed monotonicity and contraction theory to obtain computationally efficient estimates of tight inclusion functions for INNs. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs. We design a novel optimization problem for training robust INNs and we provide empirical evidence that suitably-trained INNs can be more robust than comparably-trained feedforward networks.

cs.LG↗

arXiv2022

Robust Implicit Networks via Non-Euclidean Contractions

Implicit neural networks, a.k.a., deep equilibrium networks, are a class of implicit-depth learning models where function evaluation is performed by solving a fixed point equation. They generalize classic feedforward models and are equivalent to infinite-depth weight-tied feedforward networks. While implicit models show improved accuracy and significant reduction in memory consumption, they can suffer from ill-posedness and convergence instability. This paper provides a new framework, which we call Non-Euclidean Monotone Operator Network (NEMON), to design well-posed and robust implicit neural networks based upon contraction theory for the non-Euclidean norm $\ell_{\infty}$. Our framework includes (i) a novel condition for well-posedness based on one-sided Lipschitz constants, (ii) an average iteration for computing fixed-points, and (iii) explicit estimates on input-output Lipschitz constants. Additionally, we design a training problem with the well-posedness condition and the average iteration as constraints and, to achieve robust models, with the input-output Lipschitz constant as a regularizer. Our $\ell_{\infty}$ well-posedness condition leads to a larger polytopic training search space than existing conditions and our average iteration enjoys accelerated convergence. Finally, we evaluate our framework in image classification through the MNIST and the CIFAR-10 datasets. Our numerical results demonstrate improved accuracy and robustness of the implicit models with smaller input-output Lipschitz bounds. Code is available at https://github.com/davydovalexander/Non-Euclidean_Mon_Op_Net.

cs.LG↗

arXiv2021

Robustness Certificates for Implicit Neural Networks: A Mixed Monotone Contractive Approach

Implicit neural networks are a general class of learning models that replace the layers in traditional feedforward models with implicit algebraic equations. Compared to traditional learning models, implicit networks offer competitive performance and reduced memory consumption. However, they can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks; our framework blends together mixed monotone systems theory and contraction theory. First, given an implicit neural network, we introduce a related embedded network and show that, given an $\ell_\infty$-norm box constraint on the input, the embedded network provides an $\ell_\infty$-norm box overapproximation for the output of the given network. Second, using $\ell_{\infty}$-matrix measures, we propose sufficient conditions for well-posedness of both the original and embedded system and design an iterative algorithm to compute the $\ell_{\infty}$-norm box robustness margins for reachability and classification problems. Third, of independent value, we propose a novel relative classifier variable that leads to tighter bounds on the certified adversarial robustness in classification problems. Finally, we perform numerical simulations on a Non-Euclidean Monotone Operator Network (NEMON) trained on the MNIST dataset. In these simulations, we compare the accuracy and run time of our mixed monotone contractive approach with the existing robustness verification approaches in the literature for estimating the certified adversarial robustness.

cs.LG↗

arXiv2021

From Contraction Theory to Fixed Point Algorithms on Riemannian and Non-Euclidean Spaces

The design of fixed point algorithms is at the heart of monotone operator theory, convex analysis, and of many modern optimization problems arising in machine learning and control. This tutorial reviews recent advances in understanding the relationship between Demidovich conditions, one-sided Lipschitz conditions, and contractivity theorems. We review the standard contraction theory on Euclidean spaces as well as little-known results for Riemannian manifolds. Special emphasis is placed on the setting of non-Euclidean norms and the recently introduced weak pairings for the $\ell_1$ and $\ell_\infty$ norms. We highlight recent results on explicit and implicit fixed point schemes for non-Euclidean contracting systems.

math.OC↗

arXiv2017

Upper bounds on the smallest size of a saturating set in projective planes and spaces of even dimension

In a projective plane $Π_{q}$ (not necessarily Desarguesian) of order $q$, a point subset $\mathcal{S}$ is saturating (or dense) if any point of $Π_{q}\setminus \mathcal{S}$ is collinear with two points in $\mathcal{S}$. Modifying an approach of [31], we proved the following upper bound on the smallest size $s(2,q)$ of a saturating set in $Π_{q}$: \begin{equation*} s(2,q)\leq \sqrt{(q+1)\left(3\ln q+\ln\ln q +\ln\frac{3}{4}\right)}+\sqrt{\frac{q}{3\ln q}}+3. \end{equation*} The bound holds for all q, not necessarily large. By using inductive constructions, upper bounds on the smallest size of a saturating set in the projective space $\mathrm{PG}(N,q)$ with even dimension $N$ are obtained. All the results are also stated in terms of linear covering codes.

math.CO↗

arXiv2011

On Close Relationship between Classical Time-Dependent Harmonic Oscillator and Non-Relativistic Quantum Mechanics in One Dimension

In this paper, I present a mapping between representation of some quantum phenomena in one dimension and behavior of a classical time-dependent harmonic oscillator. For the first time, it is demonstrated that quantum tunneling can be described in terms of classical physics without invoking violations of the energy conservation law at any time instance. A formula is presented that generates a wide class of potential barrier shapes with the desirable reflection (transmission) coefficient and transmission phase shift along with the corresponding exact solutions of the time-independent Schrödinger's equation. These results, with support from numerical simulations, strongly suggest that two uncoupled classical harmonic oscillators, which initially have a 90\degree relative phase shift and then are simultaneously disturbed by the same parametric perturbation of a finite duration, manifest behavior which can be mapped to that of a single quantum particle, with classical 'range relations' analogous to the uncertainty relations of quantum physics.

quant-ph↗

arXiv2011

Inequalities for non-equilibrium fluctuations of work

Five previously unknown inequalities relating equilibrium free energy differences and non-equilibrium work fluctuations are derived, and lucid path to derivation of many similar inequalities is presented. These results are based upon combined exploitation of the Jarzynski equality and the generalization of the scheme for producing uncertainty-type inequalities due to H. Weyl. The inequalities may possibly lead to better understanding of behavior of the equilibrium free-energy estimators from non-equilibrium experimental data in many important applications concerning biological, chemical, and physical molecular processes.

cond-mat.stat-mech↗