arXiv subjects

Panagiotis Patrinos

Publications and source records attributed to Panagiotis Patrinos.

At least 73 records · Page 4Linked to original sources

arXiv2023

Unsupervised Neighborhood Propagation Kernel Layers for Semi-supervised Node Classification

We present a deep Graph Convolutional Kernel Machine (GCKM) for semi-supervised node classification in graphs. The method is built of two main types of blocks: (i) We introduce unsupervised kernel machine layers propagating the node features in a one-hop neighborhood, using implicit node feature mappings. (ii) We specify a semi-supervised classification kernel machine through the lens of the Fenchel-Young inequality. We derive an effective initialization scheme and efficient end-to-end training algorithm in the dual variables for the full architecture. The main idea underlying GCKM is that, because of the unsupervised core, the final model can achieve higher performance in semi-supervised node classification when few labels are available for training. Experimental results demonstrate the effectiveness of the proposed framework.

cs.LG↗

arXiv2023

Convex relaxations for large-scale graphically structured nonconvex problems with spherical constraints: An optimal transport approach

In this paper we derive a moment relaxation for large-scale nonsmooth optimization problems with graphical structure and spherical constraints. In contrast to classical moment relaxations for global polynomial optimization that suffer from the curse of dimensionality we exploit the partially separable structure of the optimization problem to reduce the dimensionality of the search space. Leveraging optimal transport and Kantorovich--Rubinstein duality we decouple the problem and derive a tractable dual subspace approximation of the infinite-dimensional problem using spherical harmonics. This allows us to tackle possibly nonpolynomial optimization problems with spherical constraints and geodesic coupling terms. We show that the duality gap vanishes in the limit by proving that a Lipschitz continuous dual multiplier on a unit sphere can be approximated as closely as desired in terms of a Lipschitz continuous polynomial. The formulation is applied to sphere-valued imaging problems with total variation regularization and graph-based simultaneous localization and mapping (SLAM). In imaging tasks our approach achieves small duality gaps for a moderate degree. In graph-based SLAM our approach often finds solutions which after refinement with a local method are near the ground truth solution.

math.OC↗

arXiv2023

Ordered Risk Minimization: Learning More from Less Data

We consider the worst-case expectation of a permutation invariant ambiguity set of discrete distributions as a proxy-cost for data-driven expected risk minimization. For this framework, we coin the term ordered risk minimization to highlight how results from order statistics inspired the proxy-cost. Specifically, we show how such costs serve as point-wise high-confidence upper bounds of the expected risk. The confidence level can be determined tightly for any sample size. Conversely we also illustrate how to calibrate the size of the ambiguity set such that the high-confidence upper bound has some user specified confidence. This calibration procedure notably supports $ϕ$-divergence based ambiguity sets. Numerical experiments then illustrate how the resulting scheme both generalizes better and is less sensitive to tuning parameters compared to the empirical risk minimization approach.

math.OC↗

arXiv2023

Combining Primal and Dual Representations in Deep Restricted Kernel Machines Classifiers

In the context of deep learning with kernel machines, the deep Restricted Kernel Machine (DRKM) framework allows multiple levels of kernel PCA (KPCA) and Least-Squares Support Vector Machines (LSSVM) to be combined into a deep architecture using visible and hidden units. We propose a new method for DRKM classification coupling the objectives of KPCA and classification levels, with the hidden feature matrix lying on the Stiefel manifold. The classification level can be formulated as an LSSVM or as an MLP feature map, combining depth in terms of levels and layers. The classification level is expressed in its primal formulation, as the deep KPCA levels, in their dual formulation, can embed the most informative components of the data in a much lower dimensional space. The dual setting is independent of the dimension of the inputs and the primal setting is parametric, which makes the proposed method computationally efficient for both high-dimensional inputs and large datasets. In the experiments, we show that our developed algorithm can effectively learn from small datasets, while using less memory than the convolutional neural network (CNN) with high-dimensional data. and that models with multiple KPCA levels can outperform models with a single level. On the tested larger-scale datasets, DRKM is more energy efficient than CNN while maintaining comparable performance.

cs.LG↗

arXiv2023

Safety Envelope for Orthogonal Collocation Methods in Embedded Optimal Control

Orthogonal collocation methods are direct approaches for solving optimal control problems (OCP). A high solution accuracy is achieved with few optimization variables, making it more favorable for embedded and real-time NMPC applications. However, collocation approaches lack a guarantee about the safety of the resulting trajectory as inequality constraints are only set on a finite number of collocation points. In this paper we propose a method to efficiently create a convex safety envelope containing the trajectory such that the solution fully satisfies the OCP constraints. We make use of Bernstein approximations of a polynomial's extrema and span the solution over an orthogonal basis using Legendre polynomials. The tightness of the safety envelope estimation, high accuracy in solving the underlying differential equations, fast rate of convergence and little conservatism are properties of the presented approach making it a suitable method for safe real-time NMPC deployment. We show that our method has comparable computational performance to pseudospectral approaches and can accurately approximate the original OCP up to 9 times more quickly than standard multiple-shooting method in autonomous driving applications, without adding complexity to the formulation.

math.OC↗

arXiv2023

PANTR: A proximal algorithm with trust-region updates for nonconvex constrained optimization

This work presents PANTR, an efficient solver for nonconvex constrained optimization problems, that is well-suited as an inner solver for an augmented Lagrangian method. The proposed scheme combines forward-backward iterations with solutions to trust-region subproblems: the former ensures global convergence, whereas the latter enables fast update directions. We discuss how the algorithm is able to exploit exact Hessian information of the smooth objective term through a linear Newton approximation, while benefiting from the structure of box-constraints or l1-regularization. An open-source C++ implementation of PANTR is made available as part of the NLP solver library ALPAQA. Finally, the effectiveness of the proposed method is demonstrated in nonlinear model predictive control applications.

math.OC↗

arXiv2023

Nonlinear SVD with Asymmetric Kernels: feature learning and asymmetric Nyström method

Asymmetric data naturally exist in real life, such as directed graphs. Different from the common kernel methods requiring Mercer kernels, this paper tackles the asymmetric kernel-based learning problem. We describe a nonlinear extension of the matrix Singular Value Decomposition through asymmetric kernels, namely KSVD. First, we construct two nonlinear feature mappings w.r.t. rows and columns of the given data matrix. The proposed optimization problem maximizes the variance of each mapping projected onto the subspace spanned by the other, subject to a mutual orthogonality constraint. Through Lagrangian duality, we show that it can be solved by the left and right singular vectors in the feature space induced by the asymmetric kernel. Moreover, we start from the integral equations with a pair of adjoint eigenfunctions corresponding to the singular vectors on an asymmetrical kernel, and extend the Nyström method to asymmetric cases through the finite sample approximation, which can be applied to speedup the training in KSVD. Experiments show that asymmetric KSVD learns features outperforming Mercer-kernel based methods that resort to symmetrization, and also verify the effectiveness of the asymmetric Nyström method.

cs.LG↗

arXiv2023

Extending Kernel PCA through Dualization: Sparsity, Robustness and Fast Algorithms

The goal of this paper is to revisit Kernel Principal Component Analysis (KPCA) through dualization of a difference of convex functions. This allows to naturally extend KPCA to multiple objective functions and leads to efficient gradient-based algorithms avoiding the expensive SVD of the Gram matrix. Particularly, we consider objective functions that can be written as Moreau envelopes, demonstrating how to promote robustness and sparsity within the same framework. The proposed method is evaluated on synthetic and real-world benchmarks, showing significant speedup in KPCA training time as well as highlighting the benefits in terms of robustness and sparsity.

cs.LG↗

arXiv2023

Distributionally Robust Optimization using Cost-Aware Ambiguity Sets

We present a novel framework for distributionally robust optimization (DRO), called cost-aware DRO (CADRO). The key idea of CADRO is to exploit the cost structure in the design of the ambiguity set to reduce conservatism. Particularly, the set specifically constrains the worst-case distribution along the direction in which the expected cost of an approximate solution increases most rapidly. We prove that CADRO provides both a high-confidence upper bound and a consistent estimator of the out-of-sample expected cost, and show empirically that it produces solutions that are substantially less conservative than existing DRO methods, while providing the same guarantees.

math.OC↗

arXiv2023

Data-Driven Output Matching of Output-Generalized Bilinear and Linear Parameter-Varying systems

There is a growing interest in data-driven control of nonlinear systems over the last years. In contrast to related works, this paper takes a step back and aims to solve the output matching problem, a problem closely related to the reference tracking control problem, for a broader class of nonlinear systems called output-generalized bilinear, thereby offering a new direction to explore for data-driven control of nonlinear systems. It is shown that discrete time linear parameter-varying systems are included in this model class, with affine systems easily shown to also be included. This paper proposes a method to solve the output matching problem and offers a way to parameterize the solution set with a minimal number of parameters. The proposed model class and method are illustrated using simulations of two real-life systems.

math.OC↗

arXiv2023

Deep Kernel Principal Component Analysis for Multi-level Feature Learning

Principal Component Analysis (PCA) and its nonlinear extension Kernel PCA (KPCA) are widely used across science and industry for data analysis and dimensionality reduction. Modern deep learning tools have achieved great empirical success, but a framework for deep principal component analysis is still lacking. Here we develop a deep kernel PCA methodology (DKPCA) to extract multiple levels of the most informative components of the data. Our scheme can effectively identify new hierarchical variables, called deep principal components, capturing the main characteristics of high-dimensional data through a simple and interpretable numerical optimization. We couple the principal components of multiple KPCA levels, theoretically showing that DKPCA creates both forward and backward dependency across levels, which has not been explored in kernel methods and yet is crucial to extract more informative features. Various experimental evaluations on multiple data types show that DKPCA finds more efficient and disentangled representations with higher explained variance in fewer principal components, compared to the shallow KPCA. We demonstrate that our method allows for effective hierarchical data exploration, with the ability to separate the key generative factors of the input data both for large datasets and when few training samples are available. Overall, DKPCA can facilitate the extraction of useful patterns from high-dimensional data by learning more informative features organized in different levels, giving diversified aspects to explore the variation factors in the data, while maintaining a simple mathematical formulation.

cs.LG↗

arXiv2023

Escaping limit cycles: Global convergence for constrained nonconvex-nonconcave minimax problems

This paper introduces a new extragradient-type algorithm for a class of nonconvex-nonconcave minimax problems. It is well-known that finding a local solution for general minimax problems is computationally intractable. This observation has recently motivated the study of structures sufficient for convergence of first order methods in the more general setting of variational inequalities when the so-called weak Minty variational inequality (MVI) holds. This problem class captures non-trivial structures as we demonstrate with examples, for which a large family of existing algorithms provably converge to limit cycles. Our results require a less restrictive parameter range in the weak MVI compared to what is previously known, thus extending the applicability of our scheme. The proposed algorithm is applicable to constrained and regularized problems, and involves an adaptive stepsize allowing for potentially larger stepsizes. Our scheme also converges globally even in settings where the underlying operator exhibits limit cycles.

math.OC↗

arXiv2023

Solving stochastic weak Minty variational inequalities without increasing batch size

This paper introduces a family of stochastic extragradient-type algorithms for a class of nonconvex-nonconcave problems characterized by the weak Minty variational inequality (MVI). Unlike existing results on extragradient methods in the monotone setting, employing diminishing stepsizes is no longer possible in the weak MVI setting. This has led to approaches such as increasing batch sizes per iteration which can however be prohibitively expensive. In contrast, our proposed methods involves two stepsizes and only requires one additional oracle evaluation per iteration. We show that it is possible to keep one fixed stepsize while it is only the second stepsize that is taken to be diminishing, making it interesting even in the monotone setting. Almost sure convergence is established and we provide a unified analysis for this family of schemes which contains a nonlinear generalization of the celebrated primal dual hybrid gradient algorithm.

math.OC↗

arXiv2023

Dualities for non-Euclidean smoothness and strong convexity under the light of generalized conjugacy

Relative smoothness and strong convexity have recently gained considerable attention in optimization. These notions are generalizations of the classical Euclidean notions of smoothness and strong convexity that are known to be dual to each other. However, conjugate dualities for non-Euclidean relative smoothness and strong convexity remain an open problem as noted earlier by Lu, Freund and Nesterov [SIAM J. Optim., 28 (2018), pp. 333-354]. In this paper we address this question by introducing the notions of anisotropic strong convexity and smoothness as the respective dual counterparts. The dualities are developed under the light of generalized conjugacy which leads us embed the anticipated dual notions within the superclasses of certain upper and lower envelopes. In contrast to the Euclidean case these inclusions are proper in general as showcased by means of counterexamples.

math.OC↗

arXiv2023

A General Framework for Learning-Based Distributionally Robust MPC of Markov Jump Systems

We present a learning model predictive control (MPC) scheme for chance-constrained Markov jump systems with unknown switching probabilities. Using samples of the underlying Markov chain, ambiguity sets of transition probabilities are estimated which include the true conditional probability distributions with high probability. These sets are updated online and used to formulate a time-varying, risk-averse optimal control problem. We prove recursive feasibility of the resulting MPC scheme and show that the original chance constraints remain satisfied at every time step. Furthermore, we show that under sufficient decrease of the confidence levels, the resulting MPC scheme renders the closed-loop system mean-square stable with respect to the true-but-unknown distributions, while remaining less conservative than a fully robust approach. Finally, we show that the value function of the learning MPC converges from above to its nominal counterpart as the sample size grows to infinity. We illustrate our approach on a numerical example.

math.OC↗

arXiv2022

Gauss-Newton meets PANOC: A fast and globally convergent algorithm for nonlinear optimal control

PANOC is an algorithm for nonconvex optimization that has recently gained popularity in real-time control applications due to its fast, global convergence. The present work proposes a variant of PANOC that makes use of Gauss-Newton directions to accelerate the method. Furthermore, we show that when applied to optimal control problems, the computation of this Gauss-Newton step can be cast as a linear quadratic regulator (LQR) problem, allowing for an efficient solution through the Riccati recursion. Finally, we demonstrate that the proposed algorithm is more than twice as fast as the traditional L-BFGS variant of PANOC when applied to an optimal control benchmark problem, and that the performance scales favorably with increasing horizon length.

math.OC↗

arXiv2022

Anderson Accelerated Feasible Sequential Linear Programming

This paper proposes an accelerated version of Feasible Sequential Linear Programming (FSLP): the AA($d$)-FSLP algorithm. FSLP preserves feasibility in all intermediate iterates by means of an iterative update strategy which is based on repeated evaluation of zero-order information. This technique was successfully applied to techniques such as Model Predictive Control and Moving Horizon Estimation, but it can exhibit slow convergence. Moreover, keeping all iterates feasible in FSLP entails a large number of additional constraint evaluations. In this paper, Anderson Acceleration (AA($d$)) is applied to the zero-order update strategy improving the convergence rate and therefore decreasing the number of constraint evaluations in the inner iterative procedure of the FSLP algorithm. AA($d$) achieves an improved contraction rate in the inner iterations, with proven local linear convergence. In addition, it is observed that due to the improved zero-order update strategy, AA($d$)-FSLP takes larger steps to find an optimal solution, yielding faster overall convergence. The performance of AA($d$)-FSLP is examined for a time-optimal point-to-point motion problem of a parallel SCARA robot. The reduction of the number of constraint evaluations and overall iterations compared to FSLP is successfully demonstrated.

math.OC↗

arXiv2022

SPOCK: A proximal method for multistage risk-averse optimal control problems

Risk-averse optimal control problems have gained a lot of attention in the last decade, mostly due to their attractive mathematical properties and practical importance. They can be seen as an interpolation between stochastic and robust optimal control approaches, allowing the designer to trade-off performance for robustness and vice-versa. Due to their stochastic nature, risk-averse problems are of a very large scale, involving millions of decision variables, which poses a challenge in terms of efficient computation. In this work, we propose a splitting for general risk-averse problems and show how to efficiently compute iterates on a GPU-enabled hardware. Moreover, we propose Spock - a new algorithm that utilizes the proposed splitting and takes advantage of the SuperMann scheme combined with fast directions from Anderson's acceleration method for enhanced convergence speed. We implement Spock in Julia as an open-source solver, which is amenable to warm-starting and massive parallelization.

math.OC↗