arXiv subjects

Stefan Klus

Publications and source records attributed to Stefan Klus.

At least 55 records · Page 3Linked to original sources

arXiv2020

Singular Value Decomposition of Operators on Reproducing Kernel Hilbert Spaces

Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributions in order to implement the kernel Bayes rule and build sequential data models. It was recently shown that transfer operators such as the Perron-Frobenius or Koopman operator can also be approximated in a similar fashion using covariance and cross-covariance operators and that eigenfunctions of these operators can be obtained by solving associated matrix eigenvalue problems. The goal of this paper is to provide a solid functional analytic foundation for the eigenvalue decomposition of RKHS operators and to extend the approach to the singular value decomposition. The results are illustrated with simple guiding examples.

math.FA↗

arXiv2020

Data-driven approximation of the Koopman generator: Model reduction, system identification, and control

We derive a data-driven method for the approximation of the Koopman generator called gEDMD, which can be regarded as a straightforward extension of EDMD (extended dynamic mode decomposition). This approach is applicable to deterministic and stochastic dynamical systems. It can be used for computing eigenvalues, eigenfunctions, and modes of the generator and for system identification. In addition to learning the governing equations of deterministic systems, which then reduces to SINDy (sparse identification of nonlinear dynamics), it is possible to identify the drift and diffusion terms of stochastic differential equations from data. Moreover, we apply gEDMD to derive coarse-grained models of high-dimensional systems, and also to determine efficient model predictive control strategies. We highlight relationships with other methods and demonstrate the efficacy of the proposed methods using several guiding examples and prototypical molecular dynamics problems.

math.DS↗

arXiv2020

Dimensionality Reduction of Complex Metastable Systems via Kernel Embeddings of Transition Manifolds

We present a novel kernel-based machine learning algorithm for identifying the low-dimensional geometry of the effective dynamics of high-dimensional multiscale stochastic systems. Recently, the authors developed a mathematical framework for the computation of optimal reaction coordinates of such systems that is based on learning a parametrization of a low-dimensional transition manifold in a certain function space. In this article, we enhance this approach by embedding and learning this transition manifold in a reproducing kernel Hilbert space, exploiting the favorable properties of kernel embeddings. Under mild assumptions on the kernel, the manifold structure is shown to be preserved under the embedding, and distortion bounds can be derived. This leads to a more robust and more efficient algorithm compared to previous parametrization approaches.

math.DS↗

arXiv2019

Tensor-based algorithms for image classification

The interest in machine learning with tensor networks has been growing rapidly in recent years. We show that tensor-based methods developed for learning the governing equations of dynamical systems from data can, in the same way, be used for supervised learning problems and propose two novel approaches for image classification. One is a kernel-based reformulation of the previously introduced MANDy (multidimensional approximation of nonlinear dynamics), the other an alternating ridge regression in the tensor-train format. We apply both methods to the MNIST and fashion MNIST data set and show that the approaches are competitive with state-of-the-art neural network-based classifiers.

cs.LG↗

arXiv2019

Kernel Conditional Density Operators

We introduce a novel conditional density estimation model termed the conditional density operator (CDO). It naturally captures multivariate, multimodal output densities and shows performance that is competitive with recent neural conditional density models and Gaussian processes. The proposed model is based on a novel approach to the reconstruction of probability densities from their kernel mean embeddings by drawing connections to estimation of Radon-Nikodym derivatives in the reproducing kernel Hilbert space (RKHS). We prove finite sample bounds for the estimation error in a standard density reconstruction scenario, independent of problem dimensionality. Interestingly, when a kernel is used that is also a probability density, the CDO allows us to both evaluate and sample the output density efficiently. We demonstrate the versatility and performance of the proposed model on both synthetic and real-world data.

cs.LG↗

arXiv2019

Kernel methods for detecting coherent structures in dynamical data

We illustrate relationships between classical kernel-based dimensionality reduction techniques and eigendecompositions of empirical estimates of reproducing kernel Hilbert space (RKHS) operators associated with dynamical systems. In particular, we show that kernel canonical correlation analysis (CCA) can be interpreted in terms of kernel transfer operators and that it can be obtained by optimizing the variational approach for Markov processes (VAMP) score. As a result, we show that coherent sets of particle trajectories can be computed by kernel CCA. We demonstrate the efficiency of this approach with several examples, namely the well-known Bickley jet, ocean drifter data, and a molecular dynamics problem with a time-dependent potential. Finally, we propose a straightforward generalization of dynamic mode decomposition (DMD) called coherent mode decomposition (CMD). Our results provide a generic machine learning approach to the computation of coherent sets with an objective score that can be used for cross-validation and the comparison of different methods.

math.DS↗

arXiv2019

Learning chemical reaction networks from trajectory data

We develop a data-driven method to learn chemical reaction networks from trajectory data. Modeling the reaction system as a continuous-time Markov chain and assuming the system is fully observed, our method learns the propensity functions of the system with predetermined basis functions by maximizing the likelihood function of the trajectory data under $l^1$ sparse regularization. We demonstrate our method with numerical examples using synthetic data and carry out an asymptotic analysis of the proposed learning procedure in the infinite-data limit.

math.OC↗

arXiv2019

Koopman Operator-Based Finite-Control-Set Model Predictive Control for Electrical Drives

Predictive control of power electronic systems always requires a suitable model of the plant. Using typical physics-based white box models, a trade-off between model complexity (i.e. accuracy) and computational burden has to be made. This is a challenging task with a lot of constraints, since the model order is directly linked to the number of system states. Even though white-box models show suitable performance in most cases, parasitic real-world effects often cannot be modeled satisfactorily with an expedient computational load. Hence, a Koopman operator-based model reduction technique is presented which directly links the control action to the system's outputs in a black-box fashion. The Koopman operator is a linear but infinite-dimensional operator describing the dynamics of observables of nonlinear autonomous dynamical systems which can be nicely applied to the switching principle of power electronic devices. Following this data-driven approach, the model order and the number of system states are decoupled which allows us to consider more complex systems. Extensive experimental tests with an automotive-type permanent magnet synchronous motor fed by an IGBT 2-level inverter prove the feasibility of the proposed modeling technique in a finite-set model predictive control application.

math.OC↗

arXiv2019

On the Equivariance Properties of Self-adjoint Matrices

We investigate self-adjoint matrices $A\in\mathbb{R}^{n,n}$ with respect to their equivariance properties. We show in particular that a matrix is self-adjoint if and only if it is equivariant with respect to the action of a group $Γ_2(A)\subset \mathbf{O}(n)$ which is isomorphic to $\otimes_{k=1}^n\mathbf{Z}_2$. If the self-adjoint matrix possesses multiple eigenvalues -- this may, for instance, be induced by symmetry properties of an underlying dynamical system -- then $A$ is even equivariant with respect to the action of a group $Γ(A) \simeq \prod_{i = 1}^k \mathbf{O}(m_i)$ where $m_1,\ldots,m_k$ are the multiplicities of the eigenvalues $λ_1,\ldots,λ_k$ of $A$. We discuss implications of this result for equivariant bifurcation problems, and we briefly address further applications for the Procrustes problem, graph symmetries and Taylor expansions.

math.DS↗

arXiv2019

Eigendecompositions of Transfer Operators in Reproducing Kernel Hilbert Spaces

Transfer operators such as the Perron--Frobenius or Koopman operator play an important role in the global analysis of complex dynamical systems. The eigenfunctions of these operators can be used to detect metastable sets, to project the dynamics onto the dominant slow processes, or to separate superimposed signals. We extend transfer operator theory to reproducing kernel Hilbert spaces and show that these operators are related to Hilbert space representations of conditional distributions, known as conditional mean embeddings in the machine learning community. Moreover, numerical methods to compute empirical estimates of these embeddings are akin to data-driven methods for the approximation of transfer operators such as extended dynamic mode decomposition and its variants. One main benefit of the presented kernel-based approaches is that these methods can be applied to any domain where a similarity measure given by a kernel is available. We illustrate the results with the aid of guiding examples and highlight potential applications in molecular dynamics as well as video and text data analysis.

math.DS↗

arXiv2019

Multidimensional approximation of nonlinear dynamical systems

A key task in the field of modeling and analyzing nonlinear dynamical systems is the recovery of unknown governing equations from measurement data only. There is a wide range of application areas for this important instance of system identification, ranging from industrial engineering and acoustic signal processing to stock market models. In order to find appropriate representations of underlying dynamical systems, various data-driven methods have been proposed by different communities. However, if the given data sets are high-dimensional, then these methods typically suffer from the curse of dimensionality. To significantly reduce the computational costs and storage consumption, we propose the method MANDy which combines data-driven methods with tensor network decompositions. The efficiency of the introduced approach will be illustrated with the aid of several high-dimensional nonlinear dynamical systems.

math.DS↗

arXiv2019

Continuous Relaxations for the Traveling Salesman Problem

In this work, we aim to explore connections between dynamical systems techniques and combinatorial optimization problems. In particular, we construct heuristic approaches for the traveling salesman problem (TSP) based on embedding the relaxed discrete optimization problem into appropriate manifolds. We explore multiple embedding techniques -- namely, the construction of new dynamical systems on the manifold of orthogonal matrices and associated Procrustes approximations of the TSP cost function. Using these dynamical systems, we analyze the local neighborhood around the optimal TSP solutions (which are equilibria) using computations to approximate the associated \emph{stable manifolds}. We find that these flows frequently converge to undesirable equilibria. However, the solutions of the dynamical systems and the associated Procrustes approximation provide an interesting biasing approach for the popular Lin--Kernighan heuristic which yields fast convergence. The Lin--Kernighan heuristic is typically based on the computation of edges that have a `high probability' of being in the shortest tour, thereby effectively pruning the search space. Our new approach, instead, relies on a natural relaxation of the combinatorial optimization problem to the manifold of orthogonal matrices and the subsequent use of this solution to bias the Lin--Kernighan heuristic. Although the initial cost of computing these edges using the Procrustes solution is higher than existing methods, we find that the Procrustes solution, when coupled with a homotopy computation, contains valuable information regarding the optimal edges. We explore the Procrustes based approach on several TSP instances and find that our approach often requires fewer $k$-opt moves than existing approaches. Broadly, we hope that this work initiates more work in the intersection of dynamical systems theory and combinatorial optimization.

cs.DM↗

arXiv2019

Koopman operator-based model reduction for switched-system control of PDEs

We present a new framework for optimal and feedback control of PDEs using Koopman operator-based reduced order models (K-ROMs). The Koopman operator is a linear but infinite-dimensional operator which describes the dynamics of observables. A numerical approximation of the Koopman operator therefore yields a linear system for the observation of an autonomous dynamical system. In our approach, by introducing a finite number of constant controls, the dynamic control system is transformed into a set of autonomous systems and the corresponding optimal control problem into a switching time optimization problem. This allows us to replace each of these systems by a K-ROM which can be solved orders of magnitude faster. By this approach, a nonlinear infinite-dimensional control problem is transformed into a low-dimensional linear problem. In situations where the Koopman operator can be computed exactly using Extended Dynamic Mode Decomposition (EDMD), the proposed approach yields optimal control inputs. Furthermore, a recent convergence result for EDMD suggests that the approach can be applied to more complex dynamics as well. To illustrate the results, we consider the 1D Burgers equation and the 2D Navier--Stokes equations. The numerical experiments show remarkable performance concerning both solution times and accuracy.

math.DS↗

arXiv2018

A kernel-based approach to molecular conformation analysis

We present a novel machine learning approach to understanding conformation dynamics of biomolecules. The approach combines kernel-based techniques that are popular in the machine learning community with transfer operator theory for analyzing dynamical systems in order to identify conformation dynamics based on molecular dynamics simulation data. We show that many of the prominent methods like Markov State Models, EDMD, and TICA can be regarded as special cases of this approach and that new efficient algorithms can be constructed based on this derivation. The results of these new powerful methods will be illustrated with several examples, in particular the alanine dipeptide and the protein NTL9.

physics.comp-ph↗

arXiv2018

Feedback control of nonlinear PDEs using data-efficient reduced order models based on the Koopman operator

In the development of model predictive controllers for PDE-constrained problems, the use of reduced order models is essential to enable real-time applicability. Besides local linearization approaches, Proper Orthogonal Decomposition (POD) has been most widely used in the past in order to derive such models. Due to the huge advances concerning both theory as well as the numerical approximation, a very promising alternative based on the Koopman operator has recently emerged. In this chapter, we present two control strategies for model predictive control of nonlinear PDEs using data-efficient approximations of the Koopman operator. In the first one, the dynamic control system is replaced by a small number of autonomous systems with different yet constant inputs. The control problem is consequently transformed into a switching problem. In the second approach, a bilinear surrogate model, is obtained via linear interpolation between two of these autonomous systems. Using a recent convergence result for Extended Dynamic Mode Decomposition (EDMD), convergence to the true optimum can be proved. We study the properties of these two strategies with respect to solution quality, data requirements, and complexity of the resulting optimization problem using the 1D Burgers Equation and the 2D Navier-Stokes Equations as examples. Finally, an extension for online adaptivity is presented.

math.OC↗

arXiv2018

Analyzing high-dimensional time-series data using kernel transfer operator eigenfunctions

Kernel transfer operators, which can be regarded as approximations of transfer operators such as the Perron-Frobenius or Koopman operator in reproducing kernel Hilbert spaces, are defined in terms of covariance and cross-covariance operators and have been shown to be closely related to the conditional mean embedding framework developed by the machine learning community. The goal of this paper is to show how the dominant eigenfunctions of these operators in combination with gradient-based optimization techniques can be used to detect long-lived coherent patterns in high-dimensional time-series data. The results will be illustrated using video data and a fluid flow example.

stat.ML↗

arXiv2017

A Spectral Assignment Approach for the Graph Isomorphism Problem

In this paper, we propose algorithms for the graph isomorphism (GI) problem that are based on the eigendecompositions of the adjacency matrices. The eigenvalues of isomorphic graphs are identical. However, two graphs $ G_A $ and $ G_B $ can be isospectral but non-isomorphic. We first construct a graph isomorphism testing algorithm for friendly graphs and then extend it to unambiguous graphs. We show that isomorphisms can be detected by solving a linear assignment problem. If the graphs possess repeated eigenvalues, which typically correspond to graph symmetries, finding isomorphisms is much harder. By repeatedly perturbing the adjacency matrices and by using properties of eigenpolytopes, it is possible to break symmetries of the graphs and iteratively assign vertices of $ G_A $ to vertices of $ G_B $, provided that an admissible assignment exists. This heuristic approach can be used to construct a permutation which transforms $ G_A $ into $ G_B $ if the graphs are isomorphic. The methods will be illustrated with several guiding examples.

cs.DM↗

arXiv2017

Diffusion maps tailored to arbitrary non-degenerate Ito processes

We present two generalizations of the popular diffusion maps algorithm. The first generalization replaces the drift term in diffusion maps, which is the gradient of the sampling density, with the gradient of an arbitrary density of interest which is known up to a normalization constant. The second generalization allows for a diffusion map type approximation of the forward and backward generators of general Ito diffusions with given drift and diffusion coefficients. We use the local kernels introduced by Berry and Sauer, but allow for arbitrary sampling densities. We provide numerical illustrations to demonstrate that this opens up many new applications for diffusion maps as a tool to organize point cloud data, including biased or corrupted samples, dimension reduction for dynamical systems, detection of almost invariant regions in flow fields, and importance sampling.

math.DS↗