Applications of AAA rational approximation
The AAA algorithm for rational approximation is employed to illustrate applications of rational functions all across numerical analysis.
math.NA↗
arXiv subjects
Yuji Nakatsukasa
Publications and source records attributed to Yuji Nakatsukasa.
Randomized flexible Krylov methods for $\ell_p$ regularization
The computation of sparse solutions of large-scale linear discrete ill-posed problems remains a computationally demanding task. A powerful framework in this context is the use of iteratively reweighted schemes, which are based on constructing a sequence of quadratic tangent majorants of the $\ell_2$-$\ell_1$ regularization functional (with additional smoothing to ensure differentiability at the origin), and solving them successively. Recently, flexible Krylov-Tikhonov methods have been used to partially solve each problem in the sequence efficiently. However, in order to guarantee convergence, the complexity of the algorithm at each iteration increases with respect to more traditional methods. We propose a randomized flexible Krylov method to alleviate the increase of complexity, which leverages the adaptability of the flexible Krylov subspaces with the efficiency of `sketch-and-solve' methods. A possible caveat of the mentioned methods is their memory requirements. In this case, one needs to rely instead on inner-outer schemes. In these scenarios, we propose a `sketch-to-precondition' method to speed up the convergence of each of the subproblems in the sequence. The performance of these algorithms is shown through a variety of numerical examples.
math.NA↗
SubApSnap: Solving parameter-dependent linear systems with a snapshot and subsampling
A growing number of problems in computational mathematics can be reduced to the solution of many linear systems that are related, often depending smoothly or slowly on a parameter $p$, that is, $A(p)x(p)=b(p)$. We introduce an efficient algorithm for solving such parameter-dependent linear systems for many values of $p$. The algorithm, which we call SubApSnap (for \emph{Sub}sampled $A(p)$ times \emph{Snap}shot), is based on combining ideas from model order reduction and randomised linear algebra: namely, taking a snapshot matrix, and solving the resulting tall-skinny least-squares problems using a subsampling-based dimension-reduction approach. We show that SubApSnap is a strict generalisation of the popular DEIM algorithm in nonlinear model order reduction. SubApSnap is a sublinear-time algorithm, and once the snapshot and subsampling are determined, it solves $A(p_*)x(p_*)=b(p_*)$ for a new value of $p_*$ at a dramatically improved speed: it does not even need to read the whole matrix $A(p_*)$ to solve the linear system for a new value of $p_*$. We prove under natural assumptions that, given a good subsampling and snapshot, SubApSnap yields solutions with small residual for all parameter values of interest. We illustrate the efficiency and performance of the algorithm with problems arising in PDEs, model reduction, and kernel ridge regression, where SubApSnap achieves speedups of many orders of magnitude over a standard solution; for example over $20,000\times$ for a $10^7\times 10^7$ problem, while providing good accuracy.
math.NA↗
Sharp error bounds for approximate eigenvalues and singular values from subspace methods
Subspace methods are commonly used for finding approximate eigenvalues and singular values of large-scale matrices. Once a subspace is found, the Rayleigh-Ritz method (for symmetric eigenvalue problems) and Petrov-Galerkin projection (for singular values) are the de facto method for extraction of eigenvalues and singular values. In this work we derive quadratic error bounds for approximate eigenvalues of symmetric matrices obtained via the Rayleigh-Ritz process. Our bounds take advantage of the fact that extremal eigenpairs tend to converge faster than the rest, hence having smaller residuals $\|A\widehat x_i-θ_i\widehat x_i\|_2$, where $(θ_i,\widehat x_i)$ is a Ritz pair (approximate eigenpair). The proof uses the structure of the perturbation matrix underlying the Rayleigh-Ritz method to bound the components of its eigenvectors. In this way, we obtain a bound of the form $c\frac{\|A\widehat x_i-θ_i\widehat x_i\|_2^2}{\mbox{Gap}_i}$, where $\mbox{Gap}_i$ is roughly the gap between the $i$th Ritz value and the eigenvalues that are not approximated by the Ritz process, and $c> 1$ is a modest scalar. Our bound is adapted to each Ritz value and is robust to clustered Ritz values, which is a key improvement over existing results. We further show that the bound is asymptotically sharp, and generalize it to singular values of arbitrary real matrices. Finally, we apply these bounds to several methods for computing eigenvalues and singular values, and illustrate the sharpness of our bounds in a number of computational settings, including Krylov methods and randomized algorithms.
math.NA↗
Instability of the Sherman-Morrison formula and stabilization by iterative refinement
Owing to its simplicity and efficiency, the Sherman-Morrison (SM) formula has seen widespread use across various scientific and engineering applications for solving rank-one perturbed linear systems of the form $(A+uv^T)x = b$. Although the formula dates back at least to 1944, its numerical stability properties have remained an open question and continue to be a topic of current research. We analyze the backward stability of the SM, demonstrate its instability in a scenario increasingly common in scientific computing and address an open question posed by Nick Higham on the proportionality of the backward error bound to the condition number of $A$. We then incorporate fixed-precision iterative refinement into the SM framework reusing the previously computed decompositions and prove that, under reasonable assumptions, it achieves backward stability without sacrificing the efficiency of the SM formula. While our theory does not prove the SM formula with iterative refinement always outputs a backward stable solution, empirically it is observed to eventually produce a backward stable solution in all our numerical experiments. We conjecture that with iterative refinement, the SM formula yields a backward stable solution provided that $κ_2(A), κ_2(A+uv^T)$ are both bounded safely away from $ε_M^{-1}$, where $ε_M$ is the unit roundoff.
math.NA↗
Computing the Zeros of a Holomorphic Function Using Quadrature-Based Subdivision and Rational Approximation of the Logarithmic Derivative
We introduce a new method that uses AAA approximation to reliably compute all the zeros of a holomorphic function in a specified search region in the complex plane. Specifically, the method is based on rational approximation of the logarithmic derivative in combination with subdivision of the search region based on Cauchy's argument principle. This is motivated by the fact that, while it is straightforward to compute the zeros of a AAA rational approximation, there is no guarantee that all of the zeros of the function being approximated will be found. Many of the ideas presented are also applicable to computing both the zeros and the poles of a meromorphic function. A implementation of the method is provided by the Python package skzeros.
math.NA↗
Fast randomized least-squares solvers can be just as accurate and stable as classical direct solvers
One of the greatest success stories of randomized algorithms for linear algebra has been the development of fast, randomized algorithms for highly overdetermined linear least-squares problems. However, none of the existing algorithms is backward stable, preventing them from being deployed as drop-in replacements for existing QR-based solvers. This paper introduces sketch-and-precondition with iterative refinement (SPIR) and FOSSILS, two provably backward stable randomized least-squares solvers. SPIR and FOSSILS combine iterative refinement with a preconditioned iterative method applied to the normal equations and converge at the same rate as existing randomized least-squares solvers. This work offers the promise of incorporating randomized least-squares solvers into existing software libraries while maintaining the same level of accuracy and stability as classical solvers.
math.NA↗
Efficient Function Approximation Under Heteroskedastic Noise
Approximating a function $f(x)$ on $[-1,1]$ based on $N+1$ samples is a classical problem in numerical analysis. If the samples come with heteroskedastic noise depending on $x$ of variance $σ(x)^2$, an $O(N\log N)$ algorithm for this problem has not yet been found in the current literature. In this paper, we propose a method called HeteroChebtrunc, adapted from an algorithm named NoisyChebtrunc. Using techniques in high-dimensional probability, we show that with high probability, HeteroChebtrunc achieves a tighter infinity-norm error bound than NoisyChebtrunc under heteroskedastic noise. This algorithm runs in $O(N+\hat{N}\log \hat{N})$ operations, where $\hat{N}\ll N$ is a chosen parameter. While investigating the properties of HeteroChebtrunc, we also derive a high-probability non-asymptotic relative error bound on the sample variance estimator for subgaussian variables, which is potentially another result of broader interest. We provide numerical experiments to demonstrate the improved uniform error of our algorithm.
math.NA↗
A minimax method for the spectral fractional Laplacian and related evolution problems
We present a numerical method for the approximation of the inverse of the fractional Laplacian $(-Δ)^{s}$, based on its spectral definition, using rational functions to approximate the fractional power $A^{-s}$ of a matrix $A$, for $0<s<1$. The proposed numerical method is fast and accurate, benefiting from the fact that the matrix $A$ arises from a finite element approximation of the Laplacian $-Δ$, which makes it applicable to a wide range of domains with potentially irregular shapes. We make use of state-of-the-art software to compute the best rational approximation of a fractional power. We analyze the convergence rate of our method and validate our findings through a series of numerical experiments with a range of exponents $s \in (0,1)$. Additionally, we apply the proposed numerical method to different evolution problems that involve the fractional Laplacian through an interaction potential: the fractional porous medium equation and the fractional Keller-Segel equation. We then investigate the accuracy of the resulting numerical method, focusing in particular on the accurate reproduction of qualitative properties of the associated analytical solutions to these partial differential equations.
math.NA↗
Polynomial approximation of noisy functions
Approximating a univariate function on the interval $[-1,1]$ with a polynomial is among the most classical problems in numerical analysis. When the function evaluations come with noise, a least-squares fit is known to reduce the effect of noise as more samples are taken. The generic algorithm for the least-squares problem requires $O(Nn^2)$ operations, where $N+1$ is the number of sample points and $n$ is the degree of the polynomial approximant. This algorithm is unstable when $n$ is large, for example $n\gg \sqrt{N}$ for equispaced sample points. In this study, we blend numerical analysis and statistics to introduce a stable and fast $O(N\log N)$ algorithm called NoisyChebtrunc based on the Chebyshev interpolation. It has the same error reduction effect as least-squares and the convergence is spectral until the error reaches $O(σ\sqrt{{n}/{N}})$, where $σ$ is the noise level, after which the error continues to decrease at the Monte-Carlo $O(1/\sqrt{N})$ rate. To determine the polynomial degree, NoisyChebtrunc employs a statistical criterion, namely Mallows' $C_p$. We analyze NoisyChebtrunc in terms of the variance and concentration in the infinity norm to the underlying noiseless function. These results show that with high probability the infinity-norm error is bounded by a small constant times $σ\sqrt{{n}/{N}}$, when the noise {is} independent and follows a subgaussian or subexponential distribution. We illustrate the performance of NoisyChebtrunc with numerical experiments.
math.NA↗
Convergence and Near-optimal Sampling for Multivariate Function Approximations in Irregular Domains via Vandermonde with Arnoldi
Vandermonde matrices are usually exponentially ill-conditioned and often result in unstable approximations. In this paper, we introduce and analyze the \textit{multivariate Vandermonde with Arnoldi (V+A) method}, which is based on least-squares approximation together with a Stieltjes orthogonalization process, for approximating continuous, multivariate functions on $d$-dimensional irregular domains. The V+A method addresses the ill-conditioning of the Vandermonde approximation by creating a set of discrete orthogonal bases with respect to a discrete measure. The V+A method is simple and general, relying only on the domain's sample points. This paper analyzes the sample complexity of {the least-squares approximation that uses the V+A method}. We show that, for a large class of domains, this approximation gives a well-conditioned and near-optimal $N$-dimensional least-squares approximation using $M=O(N^2)$ equispaced sample points or $M=O(N^2\log N)$ random sample points, independently of $d$. We provide a comprehensive analysis of the error estimates and the rate of convergence of the least-squares approximation that uses the V+A method. Based on the multivariate V+A techniques, we propose a new variant of the weighted V+A least-squares algorithm that uses only $M=O(N\log N)$ sample points to achieve a near-optimal approximation. {Our initial numerical results validate that the V+A least-squares approximation method provides well-conditioned and near-optimal approximations for multivariate functions on (irregular) domains. Additionally, the (weighted) least-squares approximation that uses the V+A method performs competitively with state-of-the-art orthogonalization techniques and can serve as a practical tool for selecting near-optimal distributions of sample points in irregular domains.
math.NA↗
Stable algorithms for general linear systems by preconditioning the normal equations
This paper studies the solution of nonsymmetric linear systems by preconditioned Krylov methods based on the normal equations, LSQR in particular. On some examples, preconditioned LSQR is seen to produce errors many orders of magnitude larger than classical direct methods; this paper demonstrates that the attainable accuracy of preconditioned LSQR can be greatly improved by applying iterative refinement or restarting when the accuracy stalls. This observation is supported by rigorous backward error analysis. This paper also provides a discussion of the relative merits of GMRES and LSQR for solving nonsymmetric linear systems, demonstrates stability for left-preconditioned LSQR without iterative refinement, and shows that iterative refinement can also improve the accuracy of preconditioned conjugate gradient.
math.NA↗
Low-rank approximation of parameter-dependent matrices via CUR decomposition
A low-rank approximation of a parameter-dependent matrix $A(t)$ is an important task in the computational sciences appearing for example in dynamical systems and compression of a series of images. In this work, we introduce AdaCUR, an efficient algorithm for computing a low-rank approximation of parameter-dependent matrices via CUR decompositions. The key idea for this algorithm is that for nearby parameter values, the column and row indices for the CUR decomposition can often be reused. AdaCUR is rank-adaptive, provides error control, and has complexity that compares favorably against existing methods. A faster algorithm which we call FastAdaCUR that prioritizes speed over accuracy is also given, which is rank-adaptive and has complexity which is at most linear in the number of rows or columns, but without error control.
math.NA↗
Eigen-componentwise convergence of SGD on quadratic programming
Stochastic gradient descent (SGD) is a workhorse algorithm for solving large-scale optimization problems in data science and machine learning. Understanding the convergence of SGD is hence of fundamental importance. In this work we examine the SGD convergence (with various step sizes) when applied to unconstrained convex quadratic programming (essentially least-squares (LS) problems), and in particular analyze the error components respect to the eigenvectors of the Hessian. The main message is that the convergence depends largely on the corresponding eigenvalues (singular values of the coefficient matrix in the LS context), namely the components for the large singular values converge faster in the initial phase. We then show there is a phase transition in the convergence where the convergence speed of the components, especially those corresponding to the larger singular values, will decrease. Finally, we show that the convergence of the overall error (in the solution) tends to decay as more iterations are run, that is, the initial convergence is faster than the asymptote.
math.NA↗
Accuracy and Stability of CUR decompositions with Oversampling
This work investigates the accuracy and numerical stability of CUR decompositions with oversampling. The CUR decomposition approximates a matrix using a subset of columns and rows of the matrix. When the number of columns and the rows are the same, the CUR decomposition can become unstable and less accurate due to the presence of the matrix inverse in the core matrix. Nevertheless, we demonstrate that the CUR decomposition can be implemented in a numerical stable manner and illustrate that oversampling, which increases either the number of columns or rows in the CUR decomposition, can enhance its accuracy and stability. Additionally, this work devises an algorithm for oversampling motivated by the theory of the CUR decomposition and the cosine-sine decomposition, whose competitiveness is illustrated through experiments.
math.NA↗
A Sherman--Morrison--Woodbury approach to solving least squares problems with low-rank updates
We present a simple formula to update the pseudoinverse of a full-rank rectangular matrix that undergoes a low-rank modification, and demonstrate its utility for solving least squares problems. The resulting algorithm can be dramatically faster than solving the modified least squares problem from scratch, just like the speedup enabled by Sherman--Morrison--Woodbury for solving linear systems with low-rank modifications.
math.NA↗
Efficient Bounds and Estimates for Canonical Angles in Randomized Subspace Approximations
Randomized subspace approximation with "matrix sketching" is an effective approach for constructing approximate partial singular value decompositions (SVDs) of large matrices. The performance of such techniques has been extensively analyzed, and very precise estimates on the distribution of the residual errors have been derived. However, our understanding of the accuracy of the computed singular vectors (measured in terms of the canonical angles between the spaces spanned by the exact and the computed singular vectors, respectively) remains relatively limited. In this work, we present practical bounds and estimates for canonical angles of randomized subspace approximation that can be computed efficiently either a priori or a posteriori, without assuming prior knowledge of the true singular subspaces. Under moderate oversampling in the randomized SVD, our prior probabilistic bounds are asymptotically tight and can be computed efficiently, while bringing a clear insight into the balance between oversampling and power iterations given a fixed budget on the number of matrix-vector multiplications. The numerical experiments demonstrate the empirical effectiveness of these canonical angle bounds and estimates on different matrices under various algorithmic choices for the randomized SVD.
math.NA↗
Fast randomized numerical rank estimation for numerically low-rank matrices
Matrices with low-rank structure are ubiquitous in scientific computing. Choosing an appropriate rank is a key step in many computational algorithms that exploit low-rank structure. However, estimating the rank has been done largely in an ad-hoc fashion in large-scale settings. In this work we develop a randomized algorithm for estimating the numerical rank of a (numerically low-rank) matrix. The algorithm is based on sketching the matrix with random matrices from both left and right; the key fact is that with high probability, the sketches preserve the orders of magnitude of the leading singular values. We prove a result on the accuracy of the sketched singular values and show that gaps in the spectrum are detected. For an $m\times n$ $(m\geq n)$ matrix of numerical rank $r$, the algorithm runs with complexity $O(mn\log n+r^3)$, or less for structured matrices. The steps in the algorithm are required as a part of many low-rank algorithms, so the additional work required to estimate the rank can be even smaller in practice. Numerical experiments illustrate the speed and robustness of our rank estimator.
math.NA↗