arXiv subjects

Alex Townsend

Publications and source records attributed to Alex Townsend.

At least 37 records · Page 2Linked to original sources

arXiv2024

Numerical Instability of Algebraic Rootfinding Methods

We demonstrate that the most popular variants of all common algebraic multidimensional rootfinding algorithms are unstable by analyzing the conditioning of subproblems that are constructed at intermediate steps. In particular, we give multidimensional polynomial systems for which the conditioning of a subproblem can be worse than the conditioning of the original problem by a factor that grows exponentially with the number of variables.

math.NA↗

arXiv2024

How to reveal the rank of a matrix?

We study algorithms called rank-revealers that reveal a matrix's rank structure. Such algorithms form a fundamental component in matrix compression, singular value estimation, and column subset selection problems. While column-pivoted QR has been widely adopted due to its practicality, it is not always a rank-revealer. Conversely, Gaussian elimination (GE) with a pivoting strategy known as global maximum volume pivoting is guaranteed to estimate a matrix's singular values but its exponential complexity limits its interest to theory. We show that the concept of local maximum volume pivoting is a crucial and practical pivoting strategy for rank-revealers based on GE and QR. In particular, we prove that it is both necessary and sufficient; highlighting that all local solutions are nearly as good as the global one. This insight elevates Gu and Eisenstat's rank-revealing QR as an archetypal rank-revealer, and we implement a version that is observed to be at most $2\times$ more computationally expensive than CPQR. We unify the landscape of rank-revealers by considering GE and QR together and prove that the success of any pivoting strategy can be assessed by benchmarking it against a local maximum volume pivot.

math.NA↗

arXiv2023

A Mathematical Guide to Operator Learning

Operator learning aims to discover properties of an underlying dynamical system or partial differential equation (PDE) from data. Here, we present a step-by-step guide to operator learning. We explain the types of problems and PDEs amenable to operator learning, discuss various neural network architectures, and explain how to employ numerical PDE solvers effectively. We also give advice on how to create and manage training data and conduct optimization. We offer intuition behind the various neural network architectures employed in operator learning by motivating them from the point-of-view of numerical linear algebra.

math.NA↗

arXiv2023

Are sketch-and-precondition least squares solvers numerically stable?

Sketch-and-precondition techniques are efficient and popular for solving large least squares (LS) problems of the form $Ax=b$ with $A\in\mathbb{R}^{m\times n}$ and $m\gg n$. This is where $A$ is ``sketched" to a smaller matrix $SA$ with $S\in\mathbb{R}^{\lceil cn\rceil\times m}$ for some constant $c>1$ before an iterative LS solver computes the solution to $Ax=b$ with a right preconditioner $P$, where $P$ is constructed from $SA$. Prominent sketch-and-precondition LS solvers are Blendenpik and LSRN. We show that the sketch-and-precondition technique in its most commonly used form is not numerically stable for ill-conditioned LS problems. For provable and practical backward stability and optimal residuals, we suggest using an unpreconditioned iterative LS solver on $(AP)z=b$ with $x=Pz$. Provided the condition number of $A$ is smaller than the reciprocal of the unit round-off, we show that this modification ensures that the computed solution has a backward error comparable to the iterative LS solver applied to a well-conditioned matrix. Using smoothed analysis, we model floating-point rounding errors to argue that our modification is expected to compute a backward stable solution even for arbitrarily ill-conditioned LS problems. Additionally, we provide experimental evidence that using the sketch-and-solve solution as a starting vector in sketch-and-precondition algorithms (as suggested by Rokhlin and Tygert in 2008) should be highly preferred over the zero vector. The initialization often results in much more accurate solutions -- albeit not always backward stable ones.

math.NA↗

arXiv2023

Beyond expectations: Residual Dynamic Mode Decomposition and Variance for Stochastic Dynamical Systems

Koopman operators linearize nonlinear dynamical systems, making their spectral information of crucial interest. Numerous algorithms have been developed to approximate these spectral properties, and Dynamic Mode Decomposition (DMD) stands out as the poster child of projection-based methods. Although the Koopman operator itself is linear, the fact that it acts in an infinite-dimensional space of observables poses challenges. These include spurious modes, essential spectra, and the verification of Koopman mode decompositions. While recent work has addressed these challenges for deterministic systems, there remains a notable gap in verified DMD methods for stochastic systems, where the Koopman operator measures the expectation of observables. We show that it is necessary to go beyond expectations to address these issues. By incorporating variance into the Koopman framework, we address these challenges. Through an additional DMD-type matrix, we approximate the sum of a squared residual and a variance term, each of which can be approximated individually using batched snapshot data. This allows verified computation of the spectral properties of stochastic Koopman operators, controlling the projection error. We also introduce the concept of variance-pseudospectra to gauge statistical coherency. Finally, we present a suite of convergence results for the spectral information of stochastic Koopman operators. Our study concludes with practical applications using both simulated and experimental data. In neural recordings from awake mice, we demonstrate how variance-pseudospectra can reveal physiologically significant information unavailable to standard expectation-based dynamical models.

math.DS↗

arXiv2023

Elliptic PDE learning is provably data-efficient

PDE learning is an emerging field that combines physics and machine learning to recover unknown physical systems from experimental data. While deep learning models traditionally require copious amounts of training data, recent PDE learning techniques achieve spectacular results with limited data availability. Still, these results are empirical. Our work provides theoretical guarantees on the number of input-output training pairs required in PDE learning. Specifically, we exploit randomized numerical linear algebra and PDE theory to derive a provably data-efficient algorithm that recovers solution operators of 3D uniformly elliptic PDEs from input-output data and achieves an exponential convergence rate of the error with respect to the size of the training dataset with an exceptionally high probability of success.

cs.LG↗

arXiv2023

Structured matrix recovery from matrix-vector products

Can one recover a matrix efficiently from only matrix-vector products? If so, how many are needed? This paper describes algorithms to recover matrices with known structures, such as tridiagonal, Toeplitz, Toeplitz-like, and hierarchical low-rank, from matrix-vector products. In particular, we derive a randomized algorithm for recovering an $N \times N$ unknown hierarchical low-rank matrix from only $\mathcal{O}((k+p)\log(N))$ matrix-vector products with high probability, where $k$ is the rank of the off-diagonal blocks, and $p$ is a small oversampling parameter. We do this by carefully constructing randomized input vectors for our matrix-vector products that exploit the hierarchical structure of the matrix. While existing algorithms for hierarchical matrix recovery use a recursive "peeling" procedure based on elimination, our approach uses a recursive projection procedure.

math.NA↗

arXiv2023

Rigorous data-driven computation of spectral properties of Koopman operators for dynamical systems

Koopman operators are infinite-dimensional operators that globally linearize nonlinear dynamical systems, making their spectral information valuable for understanding dynamics. However, Koopman operators can have continuous spectra and infinite-dimensional invariant subspaces, making computing their spectral information a considerable challenge. This paper describes data-driven algorithms with rigorous convergence guarantees for computing spectral information of Koopman operators from trajectory data. We introduce residual dynamic mode decomposition (ResDMD), which provides the first scheme for computing the spectra and pseudospectra of general Koopman operators from snapshot data without spectral pollution. Using the resolvent operator and ResDMD, we compute smoothed approximations of spectral measures associated with general measure-preserving dynamical systems. We prove explicit convergence theorems for our algorithms, which can achieve high-order convergence even for chaotic systems when computing the density of the continuous spectrum and the discrete spectrum. Since our algorithms come with error control, ResDMD allows aposteri verification of spectral quantities, Koopman mode decompositions, and learned dictionaries. We demonstrate our algorithms on the tent map, circle rotations, Gauss iterated map, nonlinear pendulum, double pendulum, and Lorenz system. Finally, we provide kernelized variants of our algorithms for dynamical systems with a high-dimensional state space. This allows us to compute the spectral measure associated with the dynamics of a protein molecule with a 20,046-dimensional state space and compute nonlinear Koopman modes with error bounds for turbulent flow past aerofoils with Reynolds number $>10^5$ that has a 295,122-dimensional state space.

math.NA↗

arXiv2023

Avoiding discretization issues for nonlinear eigenvalue problems

The first step when solving an infinite-dimensional eigenvalue problem is often to discretize it. We show that one must be extremely careful when discretizing nonlinear eigenvalue problems. Using examples, we show that discretization can: (1) introduce spurious eigenvalues, (2) entirely miss spectra, and (3) bring in severe ill-conditioning. While there are many eigensolvers for solving matrix nonlinear eigenvalue problems, we propose a solver for general holomorphic infinite-dimensional nonlinear eigenvalue problems that avoids discretization issues, which we prove is stable and converges. Moreover, we provide an algorithm that computes the problem's pseudospectra with explicit error control, allowing verification of computed spectra. The algorithm and numerical examples are publicly available in $\texttt{infNEP}$, which is a software package written in MATLAB.

math.NA↗

arXiv2023

Leveraging the Hankel norm approximation and block-AAA algorithms in reduced order modeling

Large-scale linear, time-invariant (LTI) dynamical systems are widely used to characterize complicated physical phenomena. We propose a two-stage algorithm to reduce the order of a large-scale LTI system given samples of its transfer function for a target degree $k$ of the reduced system. In the first stage, a modified adaptive Antoulas--Anderson (AAA) algorithm is used to construct a degree $d$ rational approximation of the transfer function that corresponds to an intermediate system, which can be numerically stably reduced in the second stage using ideas from the theory on Hankel norm approximation (HNA). We also study the numerical issues of Glover's HNA algorithm and provide a remedy for its numerical instabilities. A carefully computed rational approximation of degree $d$ gives us a numerically stable algorithm for reducing an LTI system, which is more efficient than SVD-based algorithms and more accurate than moment-matching algorithms.

math.NA↗

arXiv2022

Probabilistic Missing Value Imputation for Mixed Categorical and Ordered Data

Many real-world datasets contain missing entries and mixed data types including categorical and ordered (e.g. continuous and ordinal) variables. Imputing the missing entries is necessary, since many data analysis pipelines require complete data, but this is challenging especially for mixed data. This paper proposes a probabilistic imputation method using an extended Gaussian copula model that supports both single and multiple imputation. The method models mixed categorical and ordered data using a latent Gaussian distribution. The unordered characteristics of categorical variables is explicitly modeled using the argmax operator. The method makes no assumptions on the data marginals nor does it require tuning any hyperparameters. Experimental results on synthetic and real datasets show that imputation with the extended Gaussian copula outperforms the current state-of-the-art for both categorical and ordered variables in mixed data.

stat.ME↗

arXiv2022

Tuning Frequency Bias in Neural Network Training with Nonuniform Data

Small generalization errors of over-parameterized neural networks (NNs) can be partially explained by the frequency biasing phenomenon, where gradient-based algorithms minimize the low-frequency misfit before reducing the high-frequency residuals. Using the Neural Tangent Kernel (NTK), one can provide a theoretically rigorous analysis for training where data are drawn from constant or piecewise-constant probability densities. Since most training data sets are not drawn from such distributions, we use the NTK model and a data-dependent quadrature rule to theoretically quantify the frequency biasing of NN training given fully nonuniform data. By replacing the loss function with a carefully selected Sobolev norm, we can further amplify, dampen, counterbalance, or reverse the intrinsic frequency biasing in NN training.

cs.LG↗

arXiv2022

Exploring the electric field around a loop of static charge: Rectangles, stadiums, ellipses, and knots

We study the electric field around a continuous one-dimensional loop of static charge, under the assumption that the charge is distributed uniformly along the loop. For rectangular or stadium-shaped loops in the plane, we find that the electric field can undergo a symmetry-breaking pitchfork bifurcation as the loop is elongated; the field can have either one or three zeros, depending on the loop's aspect ratio. For knotted charge distributions in three-dimensional space, we compute the electric field numerically and compare our results to previously published theoretical bounds on the number of equilibrium points around charged knots. Our computations reveal that the previous bounds are far from sharp. The numerics also suggest conjectures for the actual minimum number of equilibrium points for all charged knots with five or fewer crossings. In addition, we provide the first images of the equipotential surfaces around charged knots, and visualize their topological transitions as the level of the potential is varied.

math.DS↗

arXiv2022

Learning Green's functions associated with time-dependent partial differential equations

Neural operators are a popular technique in scientific machine learning to learn a mathematical model of the behavior of unknown physical systems from data. Neural operators are especially useful to learn solution operators associated with partial differential equations (PDEs) from pairs of forcing functions and solutions when numerical solvers are not available or the underlying physics is poorly understood. In this work, we attempt to provide theoretical foundations to understand the amount of training data needed to learn time-dependent PDEs. Given input-output pairs from a parabolic PDE in any spatial dimension $n\geq 1$, we derive the first theoretically rigorous scheme for learning the associated solution operator, which takes the form of a convolution with a Green's function $G$. Until now, rigorously learning Green's functions associated with time-dependent PDEs has been a major challenge in the field of scientific machine learning because $G$ may not be square-integrable when $n>1$, and time-dependent PDEs have transient dynamics. By combining the hierarchical low-rank structure of $G$ together with randomized numerical linear algebra, we construct an approximant to $G$ that achieves a relative error of $\smash{\mathcal{O}(Γ_ε^{-1/2}ε)}$ in the $L^1$-norm with high probability by using at most $\smash{\mathcal{O}(ε^{-\frac{n+2}{2}}\log(1/ε))}$ input-output training pairs, where $Γ_ε$ is a measure of the quality of the training dataset for learning $G$, and $ε>0$ is sufficiently small.

math.NA↗

arXiv2022

Data-driven discovery of Green's functions with human-understandable deep learning

There is an opportunity for deep learning to revolutionize science and technology by revealing its findings in a human interpretable manner. To do this, we develop a novel data-driven approach for creating a human-machine partnership to accelerate scientific discovery. By collecting physical system responses under excitations drawn from a Gaussian process, we train rational neural networks to learn Green's functions of hidden linear partial differential equations. These functions reveal human-understandable properties and features, such as linear conservation laws and symmetries, along with shock and singularity locations, boundary effects, and dominant modes. We illustrate the technique on several examples and capture a range of physics, including advection-diffusion, viscous shocks, and Stokes flow in a lid-driven cavity.

cs.LG↗

arXiv2022

A global synchronization theorem for oscillators on a random graph

Consider $n$ identical Kuramoto oscillators on a random graph. Specifically, consider \ER random graphs in which any two oscillators are bidirectionally coupled with unit strength, independently and at random, with probability $0\leq p\leq 1$. We say that a network is globally synchronizing if the oscillators converge to the all-in-phase synchronous state for almost all initial conditions. Is there a critical threshold for $p$ above which global synchrony is extremely likely but below which it is extremely rare? It is suspected that a critical threshold exists and is close to the so-called connectivity threshold, namely, $p\sim \log(n)/n$ for $n \gg 1$. Ling, Xu, and Bandeira made the first progress toward proving a result in this direction: they showed that if $p\gg \log(n)/n^{1/3}$, then \ER networks of Kuramoto oscillators are globally synchronizing with high probability as $n\rightarrow\infty$. Here we improve that result by showing that $p\gg \log^2(n)/n$ suffices. Our estimates are explicit: for example, we can say that there is more than a $99.9996\%$ chance that a random network with $n = 10^6$ and $p>0.01117$ is globally synchronizing.

math.DS↗

arXiv2022

On the stability of unevenly spaced samples for interpolation and quadrature

Unevenly spaced samples from a periodic function are common in signal processing and can often be viewed as a perturbed equally spaced grid. In this paper, we analyze how the uneven distribution of the samples impacts the quality of interpolation and quadrature. Starting with equally spaced nodes on $[-π,π)$ with grid spacing $h$, suppose the unevenly spaced nodes are obtained by perturbing each uniform node by an arbitrary amount $\leq αh$, where $0 \leq α< 1/2$ is a fixed constant. We prove a discrete version of the Kadec-1/4 theorem, which states that the nonuniform discrete Fourier transform associated with perturbed nodes has a bounded condition number independent of $h$, for any $α< 1/4$. We go on to show that unevenly spaced quadrature rules converge for all continuous functions and interpolants converge uniformly for all differentiable functions whose derivative has bounded variation when $0 \leq α< 1/4$. Though, quadrature rules at perturbed nodes can have negative weights for any $α> 0$, we provide a bound on the absolute sum of the quadrature weights. Therefore, we show that perturbed equally spaced grids with small $α$ can be used without numerical woes. While our proof techniques work primarily when $0 \leq α< 1/4$, we show that a small amount of oversampling extends our results to the case when $1/4 \leq α< 1/2$.

math.NA↗

arXiv2022

A generalization of the randomized singular value decomposition

The randomized singular value decomposition (SVD) is a popular and effective algorithm for computing a near-best rank $k$ approximation of a matrix $A$ using matrix-vector products with standard Gaussian vectors. Here, we generalize the randomized SVD to multivariate Gaussian vectors, allowing one to incorporate prior knowledge of $A$ into the algorithm. This enables us to explore the continuous analogue of the randomized SVD for Hilbert--Schmidt (HS) operators using operator-function products with functions drawn from a Gaussian process (GP). We then construct a new covariance kernel for GPs, based on weighted Jacobi polynomials, which allows us to rapidly sample the GP and control the smoothness of the randomly generated functions. Numerical examples on matrices and HS operators demonstrate the applicability of the algorithm.

math.NA↗