arXiv subjects

Erin Carson

Publications and source records attributed to Erin Carson.

At least 37 records · Page 2Linked to original sources

arXiv2025

Mixed precision sketching for least-squares problems and its application in GMRES-based iterative refinement

Sketching-based preconditioners have been shown to accelerate the solution of dense least-squares problems with coefficient matrices having substantially more rows than columns. The cost of generating these preconditioners can be reduced by employing low precision floating-point formats for all or part of the computations. We perform finite precision analysis of a mixed precision algorithm that computes the $R$-factor of a QR factorization of the sketched coefficient matrix. Two precisions can be chosen and the analysis allows understanding how to set these precisions to exploit the potential benefits of low precision formats and still guarantee an effective preconditioner. If the nature of the least-squares problem requires a solution with a small forward error, then mixed precision iterative refinement (IR) may be needed. For ill-conditioned problems the GMRES-based IR approach can be used, but good preconditioner is crucial to ensure convergence. We theoretically show when the sketching-based preconditioner can guarantee that the GMRES-based IR reduces the relative forward error of the least-squares solution and the residual to the level of the working precision unit roundoff. Small numerical examples illustrate the analysis.

math.NA↗

arXiv2025

A stable one-synchronization variant of reorthogonalized block classical Gram--Schmidt

The block classical Gram--Schmidt (BCGS) algorithm and its reorthogonalized variant are widely-used methods for computing the economic QR factorization of block columns $X$ due to their lower communication cost compared to other approaches such as modified Gram--Schmidt and Householder QR. To further reduce communication, i.e., synchronization, there has been a long ongoing search for a variant of reorthogonalized BCGS variant that achieves $O(u)$ loss of orthogonality while requiring only \emph{one} synchronization point per block column, where $u$ represents the unit roundoff. Utilizing Pythagorean inner products and delayed normalization techniques, we propose the first provably stable one-synchronization reorthogonalized BCGS variant, demonstrating that it has $O(u)$ loss of orthogonality under the condition $O(u) κ^2(X) \leq 1/2$, where $κ(\cdot)$ represents the condition number. By incorporating one additional synchronization point, we develop a two-synchronization reorthogonalized BCGS variant which maintains $O(u)$ loss of orthogonality under the improved condition $O(u) κ(X) \leq 1/2$. An adaptive strategy is then proposed to combine these two variants, ensuring $O(u)$ loss of orthogonality while using as few synchronization points as possible under the less restrictive condition $O(u) κ(X) \leq 1/2$. As an example of where this adaptive approach is beneficial, we show that using the adaptive orthogonalization variant, $s$-step GMRES achieves a backward error comparable to $s$-step GMRES with BCGSI+, also known as BCGS2, both theoretically and numerically, but requires fewer synchronization points.

math.NA↗

arXiv2025

A comparison of mixed precision iterative refinement approaches for least-squares problems

Various approaches to iterative refinement (IR) for least-squares problems have been proposed in the literature and it may not be clear which approach is suitable for a given problem. We consider three approaches to IR for least-squares problems when two precisions are used and review their theoretical guarantees, known shortcomings and when the method can be expected to recognize that the correct solution has been found, and extend uniform precision analysis for an IR approach based on the semi-normal equations to the two-precision case. We focus on the situation where it is desired to refine the solution to the working precision level. It is shown that the IR methods exhibit different sensitivities to the conditioning of the problem and the size of the least-squares residual, which should be taken into account when choosing the IR approach. We also discuss a new approach that is based on solving multiple least-squares problems.

math.NA↗

arXiv2025

Mixed precision HODLR matrices

Hierarchical matrix computations have attracted significant attention in the science and engineering community as exploiting data-sparse structures can significantly reduce the computational complexity of many important kernels. One particularly popular option within this class is the Hierarchical Off-Diagonal Low-Rank (HODLR) format. In this paper, we show that the off-diagonal blocks of HODLR matrices that are approximated by low-rank matrices can be represented in low precision without degenerating the quality of the overall approximation (with the error growth bounded by a factor of $2$). We also present an adaptive-precision scheme for constructing and storing HODLR matrices, and we prove that the use of mixed precision does not compromise the numerical stability of the resulting HODLR matrix--vector product and LU factorization. That is, the resulting error in these computations is not significantly greater than the case where we use one precision (say, double) for constructing and storing the HODLR matrix. Our analyses further give insight on how one must choose the working precision in HODLR matrix computations relative to the approximation error in order to not observe the effects of finite precision. Intuitively, when a HODLR matrix is subject to a high degree of approximation error, subsequent computations can be performed in a lower precision without detriment. We demonstrate the validity of our theoretical results through a range of numerical experiments.

math.NA↗

arXiv2024

Reorthogonalized Pythagorean variants of block classical Gram-Schmidt

Block classical Gram-Schmidt (BCGS) is commonly used for orthogonalizing a set of vectors $X$ in distributed computing environments due to its favorable communication properties relative to other orthogonalization approaches, such as modified Gram-Schmidt or Householder. However, it is known that BCGS (as well as recently developed low-synchronization variants of BCGS) can suffer from a significant loss of orthogonality in finite-precision arithmetic, which can contribute to instability and inaccurate solutions in downstream applications such as $s$-step Krylov subspace methods. A common solution to improve the orthogonality among the vectors is reorthogonalization. Focusing on the "Pythagorean" variant of BCGS, introduced in [E. Carson, K. Lund, & M. Rozložník. SIAM J. Matrix Anal. Appl. 42(3), pp. 1365--1380, 2021], which guarantees an $O(\varepsilon)κ^2(X)$ bound on the loss of orthogonality as long as $O(\varepsilon)κ^2(X)<1$, where $\varepsilon$ denotes the unit roundoff, we introduce and analyze two reorthogonalized Pythagorean BCGS variants. These variants feature favorable communication properties, with asymptotically two synchronization points per block column, as well as an improved $O(\varepsilon)$ bound on the loss of orthogonality. Our bounds are derived in a general fashion to additionally allow for the analysis of mixed-precision variants. We verify our theoretical results with a panel of test matrices and experiments from a new version of the \texttt{BlockStab} toolbox.

math.NA↗

arXiv2024

The effect of approximate coarsest-level solves on the convergence of multigrid V-cycle methods

The multigrid V-cycle method is a popular method for solving systems of linear equations. It computes an approximate solution by using smoothing on fine levels and solving a system of linear equations on the coarsest level. Solving on the coarsest level depends on the size and difficulty of the problem. If the size permits, it is typical to use a direct method based on LU or Cholesky decomposition. In settings with large coarsest-level problems, approximate solvers such as iterative Krylov subspace methods, or direct methods based on low-rank approximation, are often used. The accuracy of the coarsest-level solver is typically determined based on the experience of the users with the concrete problems and methods. In this paper we present an approach to analyzing the effects of approximate coarsest-level solves on the convergence of the V-cycle method for symmetric positive definite problems. Using these results, we derive coarsest-level stopping criterion through which we may control the difference between the approximation computed by a V-cycle method with approximate coarsest-level solver and the approximation which would be computed if the coarsest-level problems were solved exactly. The coarsest-level stopping criterion may thus be set up such that the V-cycle method converges to a chosen finest-level accuracy in (nearly) the same number of V-cycle iterations as the V-cycle method with exact coarsest-level solver. We also utilize the theoretical results to discuss how the convergence of the V-cycle method may be affected by the choice of a tolerance in a coarsest-level stopping criterion based on the relative residual norm.

math.NA↗

arXiv2024

Towards understanding CG and GMRES through examples

When the CG method for solving linear algebraic systems was formulated about 70 years ago by Lanczos, Hestenes, and Stiefel, it was considered an iterative process possessing a mathematical finite termination property. CG was placed into a rich mathematical context, including links with Gauss quadrature and continued fractions. The optimality property of CG was described via a normalized weighted polynomial least squares approximation to zero. This highly nonlinear problem explains the adaptation of CG iterates to the given data. Karush and Hayes immediately considered CG in infinite dimensional Hilbert spaces and investigated its superlinear convergence. Since then, the view of CG and other Krylov subspace methods has changed. Today these methods are primarily used as computational tools, and their behavior is typically characterized using linear upper bounds or heuristics based on clustering of eigenvalues. Such simplifications limit the mathematical understanding and also negatively affect their practical application. This paper offers a different perspective. Focusing on CG and GMRES, it presents mathematically important and practically relevant phenomena that uncover their behavior through a discussion of computed examples. These examples provide an easily accessible approach that enables understanding of the methods, while pointers to more detailed analyses in the literature are given. This approach allows readers to choose the level of depth and thoroughness appropriate for their intentions. Some of the points made in this paper illustrate well known facts. Others challenge mainstream views and explain existing misunderstandings. Several points refer to recent results leading to open problems. We consider CG and GMRES crucially important for the mathematical understanding, further development, and practical applications also of other Krylov subspace methods.

math.NA↗

arXiv2024

Mixed Precision FGMRES-Based Iterative Refinement for Weighted Least Squares

With the recent emergence of mixed precision hardware, there has been a renewed interest in its use for solving numerical linear algebra problems fast and accurately. The solution of least squares (LS) problems $\min_x\|b-Ax\|_2$, where $A \in \mathbb{R}^{m\times n}$, arise in numerous application areas. Overdetermined standard least squares problems can be solved by using mixed precision within the iterative refinement method of Björck, which transforms the least squares problem into an $(m+n)\times(m+n)$ ''augmented'' system. It has recently been shown that mixed precision GMRES-based iterative refinement can also be used, in an approach termed GMRES-LSIR. In practice, we often encounter types of least squares problems beyond standard least squares, including weighted least squares (WLS), $\min_x\|D^{1/2}(b-Ax)\|_2$, where $D^{1/2}$ is a diagonal matrix of weights. In this paper, we discuss a mixed precision FGMRES-WLSIR algorithm for solving WLS problems using two different preconditioners.

math.NA↗

arXiv2023

Mixed Precision Rayleigh Quotient Iteration for Total Least Squares Problems

With the recent emergence of mixed precision hardware, there has been a renewed interest in its use for solving numerical linear algebra problems fast and accurately. The solution of total least squares problems, i.e., solving $\min_{E,r} \| [E, r]\|_F$ subject to $(A+E)x=b+r$, arises in numerous applications. Solving this problem requires finding the smallest singular value and corresponding right singular vector of $[A,b]$, which is challenging when $A$ is large and sparse. An efficient algorithm for this case due to Björck et al. [SIAM J. Matrix Anal. Appl. 22(2), 2000], called RQI-PCGTLS, is based on Rayleigh quotient iteration coupled with the preconditioned conjugate gradient method. We develop a mixed precision variant of this algorithm, RQI-PCGTLS-MP, in which up to three different precisions can be used. We assume that the lowest precision is used in the computation of the preconditioner, and give theoretical constraints on how this precision must be chosen to ensure stability. In contrast to standard least squares, for total least squares, the resulting constraint depends not only on the matrix $A$, but also on the right-hand side $b$. We perform a number of numerical experiments on model total least squares problems used in the literature, which demonstrate that our algorithm can attain the same accuracy as RQI-PCGTLS albeit with a potential convergence delay due to the use of low precision. Performance modeling shows that the mixed precision approach can achieve up to a $4\times$ speedup depending on the size of the matrix and the number of Rayleigh quotient iterations performed.

math.NA↗

arXiv2023

The stability of split-preconditioned FGMRES in four precisions

We consider the split-preconditioned FGMRES method in a mixed precision framework, in which four potentially different precisions can be used for computations with the coefficient matrix, application of the left preconditioner, application of the right preconditioner, and the working precision. Our analysis is applicable to general preconditioners. We obtain bounds on the backward and forward errors in split-preconditioned FGMRES. Our analysis further provides insight into how the various precisions should be chosen; under certain assumptions, a suitable selection guarantees a backward error on the order of the working precision.

math.NA↗

arXiv2023

Single-pass Nyström approximation in mixed precision

Low rank matrix approximations appear in a number of scientific computing applications. We consider the Nyström method for approximating a positive semidefinite matrix $A$. In the case that $A$ is very large or its entries can only be accessed once, a single-pass version may be necessary. In this work, we perform a complete rounding error analysis of the single-pass Nyström method in two precisions, where the computation of the expensive matrix product with $A$ is assumed to be performed in the lower of the two precisions. Our analysis gives insight into how the sketching matrix and shift should be chosen to ensure stability, implementation aspects which have been commented on in the literature but not yet rigorously justified. We further develop a heuristic to determine how to pick the lower precision, which confirms the general intuition that the lower the desired rank of the approximation, the lower the precision we can use without detriment. We also demonstrate that our mixed precision Nyström method can be used to inexpensively construct limited memory preconditioners for the conjugate gradient method and derive a bound the condition number of the resulting preconditioned coefficient matrix. We present numerical experiments on a set of matrices with various spectral decays and demonstrate the utility of our mixed precision approach.

math.NA↗

arXiv2023

Mixed Precision Iterative Refinement with Adaptive Precision Sparse Approximate Inverse Preconditioning

Hardware trends have motivated the development of mixed precision algo-rithms in numerical linear algebra, which aim to decrease runtime while maintaining acceptable accuracy. One recent development is the development of an adaptive precision sparse matrix-vector produce routine, which may be used to accelerate the solution of sparse linear systems by iterative methods. This approach is also applicable to the application of inexact preconditioners, such as sparse approximate inverse preconditioners used in Krylov subspace methods. In this work, we develop an adaptive precision sparse approximate inverse preconditioner and demonstrate its use within a five-precision GMRES-based iterative refinement method. We call this algorithm variant BSPAI-GMRES-IR. We then analyze the conditions for the convergence of BSPAI-GMRES-IR, and determine settings under which BSPAI-GMRES-IR will produce similar backward and forward errors as the existing SPAI-GMRES-IR method, the latter of which does not use adaptive precision in preconditioning. Our numerical experiments show that this approach can potentially lead to a reduction in the cost of storing and applying sparse approximate inverse preconditioners, although a significant reduction in cost may comes at the expense of increasing the number of GMRES iterations required for convergence.

math.NA↗

arXiv2023

Iterated Gauss-Seidel GMRES

The GMRES algorithm of Saad and Schultz (1986) is an iterative method for approximately solving linear systems $A{\bf x}={\bf b}$, with initial guess ${\bf x}_0$ and residual ${\bf r}_0 = {\bf b} - A{\bf x}_0$. The algorithm employs the Arnoldi process to generate the Krylov basis vectors (the columns of $V_k$). It is well known that this process can be viewed as a $QR$ factorization of the matrix $B_k = [\: {\bf r}_0, AV_k\:]$ at each iteration. Despite an ${O}(ε)κ(B_k)$ loss of orthogonality, for unit roundoff $ε$ and condition number $κ$, the modified Gram-Schmidt formulation was shown to be backward stable in the seminal paper by Paige et al. (2006). We present an iterated Gauss-Seidel formulation of the GMRES algorithm (IGS-GMRES) based on the ideas of Ruhe (1983) and Świrydowicz et al. (2020). IGS-GMRES maintains orthogonality to the level ${O}(ε)κ(B_k)$ or ${O}(ε)$, depending on the choice of one or two iterations; for two Gauss-Seidel iterations, the computed Krylov basis vectors remain orthogonal to working precision and the smallest singular value of $V_k$ remains close to one. The resulting GMRES method is thus backward stable. We show that IGS-GMRES can be implemented with only a single synchronization point per iteration, making it relevant to large-scale parallel computing environments. We also demonstrate that, unlike MGS-GMRES, in IGS-GMRES the relative Arnoldi residual corresponding to the computed approximate solution no longer stagnates above machine precision even for highly non-normal systems.

math.NA↗

arXiv2022

Using Mixed Precision in Low-Synchronization Reorthogonalized Block Classical Gram-Schmidt

Using lower precision in algorithms can be beneficial in terms of reducing both computation and communication costs. Motivated by this, we aim to further the state-of-the-art in developing and analyzing mixed precision variants of iterative methods. In this work, we focus on the block variant of low-synchronization classical Gram-Schmidt with reorthogonalization, which we call BCGSI+LS. We demonstrate that the loss of orthogonality produced by this orthogonalization scheme can exceed $O(u)κ(\mathcal{X})$, where $u$ is the unit roundoff and $κ(\mathcal{X})$ is the condition number of the matrix to be orthogonalized, and thus we can not in general expect this to result in a backward stable block GMRES implementation. We then develop a mixed precision variant of this algorithm, called BCGSI+LS-MP, which uses higher precision in certain parts of the computation. We demonstrate experimentally that for a number of challenging test problems, our mixed precision variant successfully maintains a loss of orthogonality below $O(u)κ(\mathcal{X})$. This indicates that we can achieve a backward stable block GMRES algorithm that requires only one synchronization per iteration.

math.NA↗

arXiv2022

Mixed Precision Iterative Refinement with Sparse Approximate Inverse Preconditioning

With the commercial availability of mixed precision hardware, mixed precision GMRES-based iterative refinement schemes have emerged as popular approaches for solving sparse linear systems. Existing analyses of these approaches, however, are based on using full LU factorizations to construct preconditioners for use within GMRES in each refinement step. In practical applications, inexact preconditioning techniques, such as incomplete LU or sparse approximate inverses, are often used for performance reasons. In this work, we investigate the use of sparse approximate inverse preconditioners based on Frobenius norm minimization within GMRES-based iterative refinement. We analyze the computation of sparse approximate inverses in finite precision and derive constraints under which user-specified stopping criteria will be satisfied. We then analyze the behavior of and convergence constraints for a five-precision GMRES-based iterative refinement scheme that uses sparse approximate inverse preconditioning, which we call SPAI-GMRES-IR. Our numerical experiments confirm the theoretical analysis and illustrate the resulting tradeoffs between preconditioner sparsity and GMRES-IR convergence rate.

math.NA↗

arXiv2022

Mixed Precision GMRES-based Iterative Refinement with Recycling

With the emergence of mixed precision capabilities in hardware, iterative refinement schemes for solving linear systems $Ax=b$ have recently been revisited and reanalyzed in the context of three or more precisions. These new analyses show that under certain constraints on condition number, the LU factorization of the matrix can be computed in low precision without affecting the final accuracy. Another promising technique is GMRES-based iterative refinement, which, in contrast to the standard approach, use GMRES preconditioned by the low-precision triangular factors to solve for the approximate solution update in each refinement step. This more accurate solution method extends the range of problems which can be solved with a given combination of precisions. However, in certain settings, GMRES may require too many iterations per refinement step, making it potentially more expensive than simply recomputing the LU factors in a higher precision. Krylov subspace recycling is a well-known technique for reusing information across sequential invocations of a Krylov subspace method on systems with the same or a slowly changing coefficient matrix. In this work, we incorporate the idea of Krylov subspace recycling into a mixed precision GMRES-based iterative refinement solver. The insight is that in each refinement step, we call preconditioned GMRES on a linear system with the same coefficient matrix $A$, with only the right-hand side changing. In this way, the GMRES solves in subsequent refinement steps can be accelerated by recycling information obtained from the first step. We perform extensive numerical experiments on various random dense problems, Toeplitz problems (prolate matrices), and problems from real applications, which confirm the benefits of the recycling approach.

math.NA↗

arXiv2021

Multistage Mixed Precision Iterative Refinement

Low precision arithmetic, in particular half precision floating point arithmetic, is now available in commercial hardware. Using lower precision can offer significant savings in computation and communication costs with proportional savings in energy. Motivated by this, there has been a renewed interest in mixed precision iterative refinement for solving linear systems $Ax=b$, and new variants of GMRES-based iterative refinement have been developed. Each particular variant with a given combination of precisions leads to different condition number-based constraints for convergence of the backward and forward errors, and each has different performance costs. The constraints for convergence given in the literature are, as an artifact of the analyses, often overly strict in practice, and thus could lead a user to select a more expensive variant when a less expensive one would have sufficed. In this work, we develop a multistage mixed precision iterative refinement solver which aims to combine existing mixed precision approaches to balance performance and accuracy and improve usability. For an initial combination of precisions, the algorithm begins with the least expensive approach and convergence is monitored via inexpensive computations with quantities produced during the iteration. If slow convergence or divergence is detected using particular stopping criteria, the algorithm switches to use a more expensive, but more reliable variant. A novel aspect of our approach is that, unlike existing implementations, our algorithm first attempts to use ``stronger'' solvers for the solution update before resorting to increasing the precision(s). In some scenarios, this can avoid the need to refactorize the matrix in higher precision. We perform extensive numerical experiments on random dense problems and problems from real applications which confirm the benefits of the multistage approach.

math.NA↗

arXiv2021

Mixed Precision $s$-step Lanczos and Conjugate Gradient Algorithms

Compared to the classical Lanczos algorithm, the $s$-step Lanczos variant has the potential to improve performance by asymptotically decreasing the synchronization cost per iteration. However, this comes at a cost. Despite being mathematically equivalent, the $s$-step variant is known to behave quite differently in finite precision, with potential for greater loss of accuracy and a decrease in the convergence rate relative to the classical algorithm. It has previously been shown that the errors that occur in the $s$-step version follow the same structure as the errors in the classical algorithm, but with the addition of an amplification factor that depends on the square of the condition number of the $O(s)-$dimensional Krylov bases computed in each outer loop. As the condition number of these $s$-step bases grows (in some cases very quickly) with $s$, this limits the parameter $s$ that can be chosen and thus limits the performance that can be achieved. In this work we show that if a select few computations in $s$-step Lanczos are performed in double the working precision, the error terms then depend only linearly on the conditioning of the $s$-step bases. This has the potential for drastically improving the numerical behavior of the algorithm with little impact on per-iteration performance. Our numerical experiments demonstrate the improved numerical behavior possible with the mixed precision approach, and also show that this improved behavior extends to the $s$-step CG algorithm in mixed precision.

math.NA↗