arXiv subjects

Ryan Mohr

Publications and source records attributed to Ryan Mohr.

17 recordsLinked to original sources

arXiv2023

A Koopman Operator-Based Prediction Algorithm and its Application to COVID-19 Pandemic

The problem of prediction of behavior of dynamical systems has undergone a paradigm shift in the second half of the 20th century with the discovery of the possibility of chaotic dynamics in simple, physical, dynamical systems for which the laws of evolution do not change in time. The essence of the paradigm is the long term exponential divergence of trajectories. However, that paradigm does not account for another type of unpredictability: the ``Black Swan" event. It also does not account for the fact that short-term prediction is often possible even in systems with exponential divergence. In our framework, the Black Swan type dynamics occurs when an underlying dynamical system suddenly shifts between dynamics of different types. A learning and prediction system should be capable of recognizing the shift in behavior, exemplified by ``confidence loss". In this paradigm, the predictive power is assessed dynamically and confidence level is used to switch between long term prediction and local-in-time prediction. Here we explore the problem of prediction in systems that exhibit such behavior. The mathematical underpinnings of our theory and algorithms are based on an operator-theoretic approach in which the dynamics of the system are embedded into an infinite-dimensional space. We apply the algorithm to a number of case studies including prediction of influenza cases and the COVID-19 pandemic. The results show that the predictive algorithm is robust to perturbations of the available data, induced for example by delays in reporting or sudden increase in cases due to increase in testing capability. This is achieved in an entirely data-driven fashion, with no underlying mathematical model of the disease.

math.DS↗

arXiv2023

Identifying Equivalent Training Dynamics

Study of the nonlinear evolution deep neural network (DNN) parameters undergo during training has uncovered regimes of distinct dynamical behavior. While a detailed understanding of these phenomena has the potential to advance improvements in training efficiency and robustness, the lack of methods for identifying when DNN models have equivalent dynamics limits the insight that can be gained from prior work. Topological conjugacy, a notion from dynamical systems theory, provides a precise definition of dynamical equivalence, offering a possible route to address this need. However, topological conjugacies have historically been challenging to compute. By leveraging advances in Koopman operator theory, we develop a framework for identifying conjugate and non-conjugate training dynamics. To validate our approach, we demonstrate that comparing Koopman eigenvalues can correctly identify a known equivalence between online mirror descent and online gradient descent. We then utilize our approach to: (a) identify non-conjugate training dynamics between shallow and wide fully connected neural networks; (b) characterize the early phase of training dynamics in convolutional neural networks; (c) uncover non-conjugate training dynamics in Transformers that do and do not undergo grokking. Our results, across a range of DNN architectures, illustrate the flexibility of our framework and highlight its potential for shedding new light on training dynamics.

cs.LG↗

arXiv2022

Koopman Reduced Order Modeling with Confidence Bounds

This paper introduces a reduced order modeling technique based on Koopman operator theory that gives confidence bounds on the model's predictions. It is based on a data-driven spectral decomposition of the Koopman operator. The reduced order model is constructed using a finite number of Koopman eigenvalues and modes, while the rest of spectrum is treated as a noise process. This noise process is used to extract the confidence bounds. Additionally, we propose a heuristic algorithm to choose the number of deterministic modes to keep in the model.

math.DS↗

arXiv2022

Algorithmic (Semi-)Conjugacy via Koopman Operator Theory

Iterative algorithms are of utmost importance in decision and control. With an ever growing number of algorithms being developed, distributed, and proprietarized, there is a similarly growing need for methods that can provide classification and comparison. By viewing iterative algorithms as discrete-time dynamical systems, we leverage Koopman operator theory to identify (semi-)conjugacies between algorithms using their spectral properties. This provides a general framework with which to classify and compare algorithms.

cs.DS↗

arXiv2022

An Operator Theoretic View on Pruning Deep Neural Networks

The discovery of sparse subnetworks that are able to perform as well as full models has found broad applied and theoretical interest. While many pruning methods have been developed to this end, the naïve approach of removing parameters based on their magnitude has been found to be as robust as more complex, state-of-the-art algorithms. The lack of theory behind magnitude pruning's success, especially pre-convergence, and its relation to other pruning methods, such as gradient based pruning, are outstanding open questions in the field that are in need of being addressed. We make use of recent advances in dynamical systems theory, namely Koopman operator theory, to define a new class of theoretically motivated pruning algorithms. We show that these algorithms can be equivalent to magnitude and gradient based pruning, unifying these seemingly disparate methods, and find that they can be used to shed light on magnitude pruning's performance during the early part of training.

cs.LG↗

arXiv2020

Predicting the Critical Number of Layers for Hierarchical Support Vector Regression

Hierarchical support vector regression (HSVR) models a function from data as a linear combination of SVR models at a range of scales, starting at a coarse scale and moving to finer scales as the hierarchy continues. In the original formulation of HSVR, there were no rules for choosing the depth of the model. In this paper, we observe in a number of models a phase transition in the training error -- the error remains relatively constant as layers are added, until a critical scale is passed, at which point the training error drops close to zero and remains nearly constant for added layers. We introduce a method to predict this critical scale a priori with the prediction based on the support of either a Fourier transform of the data or the Dynamic Mode Decomposition (DMD) spectrum. This allows us to determine the required number of layers prior to training any models.

cs.LG↗

arXiv2020

Applications of Koopman Mode Analysis to Neural Networks

We consider the training process of a neural network as a dynamical system acting on the high-dimensional weight space. Each epoch is an application of the map induced by the optimization algorithm and the loss function. Using this induced map, we can apply observables on the weight space and measure their evolution. The evolution of the observables are given by the Koopman operator associated with the induced dynamical system. We use the spectrum and modes of the Koopman operator to realize the above objectives. Our methods can help to, a priori, determine the network depth; determine if we have a bad initialization of the network weights, allowing a restart before training too long; speeding up the training time. Additionally, our methods help enable noise rejection and improve robustness. We show how the Koopman spectrum can be used to determine the number of layers required for the architecture. Additionally, we show how we can elucidate the convergence versus non-convergence of the training process by monitoring the spectrum, in particular, how the existence of eigenvalues clustering around 1 determines when to terminate the learning process. We also show how using Koopman modes we can selectively prune the network to speed up the training procedure. Finally, we show that incorporating loss functions based on negative Sobolev norms can allow for the reconstruction of a multi-scale signal polluted by very large amounts of noise.

cs.LG↗

arXiv2020

Extended Dynamic Mode Decomposition with Learned Koopman Eigenfunctions for Prediction and Control

This paper presents a novel learning framework to construct Koopman eigenfunctions for unknown, nonlinear dynamics using data gathered from experiments. The learning framework can extract spectral information from the full nonlinear dynamics by learning the eigenvalues and eigenfunctions of the associated Koopman operator. We then exploit the learned Koopman eigenfunctions to learn a lifted linear state-space model. To the best of our knowledge, our method is the first to utilize Koopman eigenfunctions as lifting functions for EDMD-based methods. We demonstrate the performance of the framework in state prediction and closed loop trajectory tracking of a simulated cart pole system. Our method is able to significantly improve the controller performance while relying on linear control methods to do nonlinear control.

eess.SY↗

arXiv2018

On the approximation of Koopman spectra for measure preserving transformations

For the class of continuous, measure-preserving automorphisms on compact metric spaces, a procedure is proposed for constructing a sequence of finite-dimensional approximations to the associated Koopman operator on a Hilbert space. These finite-dimensional approximations are obtained from the so-called "periodic approximation" of the underlying automorphism and take the form of permutation operators. Results are established on how these discretizations approximate the Koopman operator spectrally. Specificaly, it is shown that both the spectral measure and the spectral projectors of these permutation operators converge weakly to their infinite dimensional counterparts. Based on this result, a numerical method is derived for computing the spectra of volume-preserving maps on the unit $m$-torus. The discretized Koopman operator can be constructed from solving a bipartite matching problem with $\mathcal{O}(\tilde{n}^{3m/2})$ time-complexity, where $\tilde{n}$ denotes the gridsize on each dimension. By exploiting the permutation structure of the discretized Koopman operator, it is further shown that the projections and density functions are computable in $\mathcal{O}(m \tilde{n}^{m} \log \tilde{n})$ operations using the FFT algorithm. Our method is illustrated on several classical examples of automorphisms on the torus that contain either a discrete, continuous, or a mixed spectra. In addition, the spectral properties of the Chirikov standard map are examined using our method.

math.DS↗

arXiv2018

On least squares problems with certain Vandermonde--Khatri--Rao structure with applications to DMD

This paper proposes a new computational method for solving structured least squares problems that arise in the process of identification of coherent structures in fluid flows. It is deployed in combination with dynamic mode decomposition (DMD) which provides a non-orthogonal set of modes --- corresponding to particular temporal frequencies --- a subset of which is used to represent time snapshots of the underlying dynamics. The coefficients of the representation are determined from a solution of a structured linear least squares problem with the matrix that involves the Khatri--Rao product of a triangular and a Vandermonde matrix. Such a structure allows a very efficient normal equation based least squares solution, which is used in state of the art CFD tools such as the sparsity promoting DMD (DMDSP). A new numerical analysis of the normal equations approach provides insights about its applicability and its limitations. Relevant condition numbers that determine numerical robustness are identified and discussed. Further, the paper offers a corrected semi-normal solution and QR factorization based algorithms. It is shown how to use the Vandermonde--Khatri--Rao structure to efficiently compute the QR factorization of the least squares coefficient matrix, thus providing a new computational tool for the ill-conditioned cases where the normal equations may fail to compute a sufficiently accurate solution. Altogether, the presented material provides a firm numerical linear algebra framework for a class of structured least squares problems arising in a variety of applications.

math.NA↗

arXiv2018

Data driven Koopman spectral analysis in Vandermonde-Cauchy form via the DFT: numerical method and theoretical insights

The goals and contributions of this paper are twofold. It provides a new computational tool for data driven Koopman spectral analysis by taking up the formidable challenge to develop a numerically robust algorithm by following the natural formulation via the Krylov decomposition with the Frobenius companion matrix, and by using its eigenvectors explicitly -- these are defined as the inverse of the notoriously ill-conditioned Vandermonde matrix. The key step to curb ill-conditioning is the discrete Fourier transform of the snapshots; in the new representation, the Vandermonde matrix is transformed into a generalized Cauchy matrix, which then allows accurate computation by specially tailored algorithms of numerical linear algebra. The second goal is to shed light on the connection between the formulas for optimal reconstruction weights when reconstructing snapshots using subsets of the computed Koopman modes. It is shown how using a certain weaker form of generalized inverses leads to explicit reconstruction formulas that match the abstract results from Koopman spectral theory, in particular the Generalized Laplace Analysis.

math.NA↗

arXiv2018

On the approximation of Koopman spectra of measure-preserving flows

The method of using periodic approximations to compute the spectral decomposition of the Koop- man operator is generalized to the class of measure-preserving flows on compact metric spaces. It is shown that the spectral decomposition of the continuous one-parameter unitary group can be approximated from an intermediate time discretization of the flow. A sufficient condition is established between the time-discretization of the flow and the spatial discretization of the periodic approximation, so that weak convergence of spectra will occur in the limit. This condition effectively translates to the requirement that the spatial refinements must occur at a faster pace than the temporal refinements. This result is contrasted with the well-known CLF condition of finite difference schemes for advection equations. Numerical results of spectral computations are shown for some benchmark examples of volume-preserving flows.

math.DS↗

arXiv2017

Data driven modal decompositions: analysis and enhancements

The Dynamic Mode Decomposition (DMD) is a tool of trade in computational data driven analysis of fluid flows. More generally, it is a computational device for Koopman spectral analysis of nonlinear dynamical systems, with a plethora of applications in applied sciences and engineering. Its exceptional performance triggered developments of several modifications that make the DMD an attractive method in data driven framework. This work offers further improvements of the DMD to make it more reliable, and to enhance its functionality. In particular, data driven formula for the residuals allows selection of the Ritz pairs, thus providing more precise spectral information of the underlying Koopman operator, and the well-known technique of refining the Ritz vectors is adapted to data driven scenarios. Further, the DMD is formulated in a more general setting of weighted inner product spaces, and the consequences for numerical computation are discussed in detail. Numerical experiments are used to illustrate the advantages of the proposed method, designated as DDMD_RRR (Refined Rayleigh Ritz Data Driven Modal Decomposition).

math.NA↗

arXiv2017

Koopman Spectrum for Cascaded Systems

This paper considers the evolution of Koopman principal eigenfunctions of cascaded dynamical systems. If each component subsystem is asymptotically stable, the matrix norms of the linear parts of the component subsystems are strictly increasing, and the component subsystems have disjoint spectrums, there exist perturbation functions for the initial conditions of each component subsystem such that the orbits of the cascaded system and the decoupled component subsystems have zero asymptotic relative error. This implies that the evolutions are asymptotically equivalent; cascaded compositions of stable systems are stable. These results hold for both cascaded systems with linear component subsystem dynamics and linear coupling terms and nonlinear cascades topologically conjugate to the linear case. We further show that the Koopman principal eigenvalues of each component subsystem are also Koopman eigenvalues of the cascaded system. The corresponding Koopman eigenfunctions of the cascaded system are formed by extending the domain of definition of the component systems' principal eigenfunctions and then composing them with the perturbation function.

math.DS↗

arXiv2016

Koopman principle eigenfunctions and linearization of diffeomorphisms

This paper considers a nonlinear dynamical system on a complex, finite dimensional Banach space which has an asymptotically stable, hyperbolic fixed point. We investigate the connection between the so-called principle eigenfunctions of the Koopman operator and the existence of a topological conjugacy between the nonlinear dynamics and its linearization in the neighborhood of the fixed point. The principle eigenfunctions generate an algebra of observables for the linear dynamics --- called the principle algebra --- which can be used to generate a sequence of approximate conjugacy maps in the same manner as it is done in normal form theory. Each element of the principle algebra has an expansion into eigenfunctions of the Koopman operator and composing an eigenfunction with one of the approximate topological conjugacies gives an approximate eigenfunction of the Koopman operator associated with the nonlinear dynamical system. When the limit of the approximate conjugacies exists and attention is restricted to real Banach spaces, a simple application of the Stone-Weierstrass theorem shows that both the principle algebra and the pull-back algebra --- defined by composing the principle algebra with the topological conjugacy --- are uniformly dense in either the space of continuous functions or the maximal ideal of continuous functions vanishing at the fixed point. The point is that, a priori, it is difficult to know which space of observables to use for dissipative nonlinear dynamical systems whose elements have spectral expansions into eigenfunctions. These results say that any continuous observable is arbitrarily close to one that has such an expansion.

math.DS↗

arXiv2016

Programmable Potentials: Approximate N-body potentials from coarse-level logic

This paper gives a systematic method for constructing an N-body potential, approximating the true potential, that accurately captures meso-scale behavior of the chemical or biological system using pairwise potentials coming from experimental data or ab initio methods. The meso-scale behavior is translated into logic rules for the dynamics. Each pairwise potential has an associated logic function that is constructed using the logic rules, a class of elementary logic functions, and AND, OR, and NOT gates. The effect of each logic function is to turn its associated potential on and off. The N-body potential is constructed as linear combination of the pairwise potentials, where the "coefficients" of the potentials are smoothed versions of the associated logic functions. These potentials allow a potentially low-dimensional description of complex processes while still accurately capturing the relevant physics at the meso-scale. We present the proposed formalism to construct coarse-grained potential models for three examples: an inhibitor molecular system, bond breaking in chemical reactions, and DNA transcription from biology. The method can potentially be used in reverse for design of molecular processes by specifying properties of molecules that can carry them out.

physics.chem-ph↗

arXiv2014

Construction of eigenfunctions for scalar-type operators via Laplace averages with connections to the Koopman operator

This paper extends Yosida's mean ergodic theorem in order to compute projections onto non-unitary eigenspaces for spectral operators of scalar-type on locally convex linear topological spaces. For spectral operators with dominating point spectrum, the projections take the form of Laplace averages, which are a generalization of the Fourier averages used when the spectrum is unitary. Inverse iteration and Laplace averages project onto eigenspaces of spectral operators with minimal point spectrum. Two classes of dynamical systems --- attracting fixed points in $\mathbb{C}^{d}$ and attracting limit cycles in $\mathbb{R}^{2}$ --- and their respective spaces of observables are given for which the associated composition operator is spectral. It is shown that the natural spaces of observables are completions with an $\ell^{2}$ polynomial norm of a space of polynomials over a normed unital commutative ring. These spaces are generalizations of the Hardy spaces $H^{2}(\mathbb{D})$ and $H^{2}(\mathbb{D}^{d})$. Elements of the ring are observables defined on the attractor --- the fixed point or the limit cycle, in our examples. Furthermore, we are able to provide a (semi)global spectral theorem for the composition operators associated with a large class of dissipative nonlinear dynamical systems; any sufficiently smooth dynamical system topologically conjugate to either of the two cases above admits an observable space on which the associated Koopman operator is spectral. It is conjectured that this is generically true for systems where the basin of attraction can be properly "coordinatized".

math.SP↗