arXiv subjects

Laura Grigori

Publications and source records attributed to Laura Grigori.

At least 19 recordsLinked to original sources

arXiv2026

Randomized Block Davidson Eigensolvers for Plane-Wave Density-Functional Theory

Iterative diagonalization is the dominant cost of plane-wave density-functional theory (DFT), with search-space orthogonalization scaling particularly quickly with problem size and the number of target states. We present a randomized block Davidson-type eigensolver that replaces Euclidean orthogonalization with randomized Gram-Schmidt in a sketched inner product, requiring only a single pass over the basis while keeping its conditioning bounded independently of the input vectors. This modification changes only the Rayleigh-Ritz step, which becomes a definite generalized Hermitian eigenproblem. Ritz extraction remains exact, preserving true Ritz pairs and the interlacing property that makes each band energy an upper bound on the true one. The method is implemented in mixed precision for CPUs and GPUs from a single Julia code, interfaces matrix-free with DFTK, and is released in the open-source RandESC library. On sparse test problems with a fixed number of eigenpairs, the sketched solver overtakes its deterministic counterpart beyond matrix dimensions of about $2\times 10^4$ and is $25\%$ faster at $5\times 10^5$. In full self-consistent field DFT calculations, however, both Davidson variants outperform the locally optimal block preconditioned conjugate gradient (LOBPCG) reference only by $5$ to $11\%$ in total time, while the additional benefit of sketching is limited. As the number of requested states grows with system size, orthogonalization savings are offset by the generalized eigenproblem. Therefore, the regime in which sketching pays off is set by how the number of wanted states scales with the problem dimension, not by the eigensolver as such.

math.NA↗

arXiv2026

Accelerating the Canonical Polyadic Alternating Least Squares Optimization via a Randomized Interpolative Decomposition

We present a novel leverage score-based sampling strategy for the randomized alternating least squares optimization (ALS) of the canonical polyadic decomposition (CPD-ALS). Unlike previous strategies, we determine row-wise samples for the CPD-ALS problem from the leverage scores of the target tensor which is being decomposed. We demonstrate that, when rows are sampled according to the leverage score distribution of the matricized target tensor, each least squares subproblem of the CPD-ALS problem achieves $(1+\epsilon)-$relative accuracy in the residual norm with probability at least $1-\delta$ using a sampling $s=\frac{R\gamma}{\beta} \max\left(\frac{4}{\delta \epsilon}, \frac{144\ln(2R/\delta)}{\epsilon_{0}^{2}}\right)$, where $\epsilon_{0}$ is a constant, $\beta$ is leverage score's approximation constant, $R$ is the target rank and $\gamma$ captures the coherence between the Khatri Rao product (KRP) of the CPD factor matrices and the exact KRP; $\gamma$ decreases as the ALS iterates converge. To efficiently approximate the leverage score distribution for each matricization of the target tensor without explicitly computing leverage scores we use a randomized strong rank-revealing QR (sRRQR) factorizations, SE-QRCS. By construction, this QR-based leverage score sampling method outperforms previously published schemes as it does not, in principle, require the resampling of the target tensor or recomputing the leverage scores of the KRP, minimizing the computational and storage overhead of the CPD-ALS procedure.

math.NA↗

arXiv2026

Sketch low-rank dynamics: orthogonal vs. oblique projections

We study how sketching techniques from randomized numerical linear algebra can be incorporated into the dynamical low-rank approximation (DLRA) of large-scale matrix differential equations. A natural approach is to sketch the Galerkin condition that defines the DLRA, which leads to an oblique tangent space projection. We show that this oblique projection approximately reproduces the standard DLRA only under restrictive conditions on the vector field, and that it fails on problems with a large perpendicular residual. As an alternative, we propose an orthogonal sketch DLRA that evolves sketch-orthogonal bases while using standard orthogonal projections for the dynamics. This approach preserves the geometric structure of the classical DLRA and is numerically stable. The computational advantage of randomized Gram--Schmidt over Householder QR lies in fewer global synchronizations on a row-distributed basis, at a comparable flop count; when the basis is well conditioned, randomized Gram--Schmidt can be replaced by randomized Cholesky QR, which additionally shifts the basis update from BLAS-2 to BLAS-3 kernels, making it well-suited to modern accelerators. We derive sketch versions of the projector-splitting and BUG integrators, and demonstrate the approach on the Allen--Cahn, Fokker--Planck, and Vlasov--Poisson equations.

math.NA↗

arXiv2026

Attention Mechanisms Through the Lens of Numerical Methods: Approximation Methods and Alternative Formulations

The attention mechanism is the computational core of modern Transformer architectures, but its quadratic complexity in the input sequence length is the bottleneck for large-scale inference. This has motivated a rapidly growing body of work aimed at accelerating attention through approximation and reformulation. In this survey, we revisit attention mechanisms through the lens of numerical analysis, with a particular emphasis on tools and perspectives from numerical linear algebra. Our goal is twofold: first, we aim to systematically review and classify fast approximation methods according to the numerical principles they exploit. These include sparsity and clustering approaches, low-rank and subspace projection techniques, randomized sketching methods, and tensor-based decompositions. We also discuss kernel-inspired reformulations of attention and recent architectural variants, such as Latent Attention, that modify the standard softmax formulation to improve efficiency. Second, by presenting these developments within a unified mathematical framework, we aim to bridge the gap between disciplines and highlight opportunities for further contributions from computational mathematics, particularly numerical linear algebra, to the design of scalable attention mechanisms.

math.NA↗

arXiv2026

Communication Lower Bounds and Algorithms for Sketching with Random Dense Matrices

Sketching is widely used in randomized linear algebra for low-rank matrix approximation, column subset selection, and many other problems, and it has gained significant traction in machine learning applications. However, sketching large matrices often necessitates distributed memory algorithms, where communication overhead becomes a critical bottleneck on modern supercomputing clusters. Despite its growing relevance, distributed-memory parallel strategies for sketching remain largely unexplored. In this work, we establish communication lower bounds for sketching using dense matrices that determine how much data movement is required to perform it in parallel. One important observation of our lower bounds is that no communication is required for a small number of processors. We show that our lower bounds are tight by presenting communication optimal algorithms. Furthermore, we extend our approach to determine communication lower bounds for computations of Nystr\"om approximation where sketching is applied twice. We also introduce novel parallel algorithms whose communication costs are close to the lower bounds. Finally, we implement our algorithms on modern state-of-the-art supercomputing infrastructures which have both CPU- and GPU-equipped systems and demonstrate their parallel scalability.

cs.DC↗

arXiv2026

Restoring similarity in randomized Krylov methods with applications to eigenvalue problems and matrix functions

The randomized Arnoldi process has been used in large-scale scientific computing because it produces a well-conditioned basis for the Krylov subspace more quickly than the standard Arnoldi process. However, the resulting Hessenberg matrix is generally not similar to the one produced by the standard Arnoldi process, which can lead to delays or spike-like irregularities in convergence. In this paper, we introduce a modification of the randomized Arnoldi process that restores similarity with the Hessenberg matrix generated by the standard Arnoldi process. This is accomplished by enforcing orthogonality between the last Arnoldi vector and the previously generated subspace, which requires solving only one additional least-squares problem. When applied to eigenvalue problems and matrix function evaluations, the modified randomized Arnoldi process produces approximations that are identical to those obtained with the standard Arnoldi process. Numerical experiments demonstrate that our approach is as fast as the randomized Arnoldi process and as robust as the standard Arnoldi process.

math.NA↗

arXiv2025

Randomized orthogonalization and Krylov subspace methods: principles and algorithms

We present an overview of randomized orthogonalization techniques that construct a well-conditioned basis whose sketch is orthonormal. Randomized orthogonalization has recently emerged as a powerful paradigm for reducing the computational and communication cost of state-of-the-art orthogonalization procedures on parallel architectures, while preserving, and in some cases improving, their numerical stability. This approach can be employed within Krylov subspace methods to mitigate the cost of orthogonalization, yielding a randomized Arnoldi relation. We review the main variants of the randomized Gram--Schmidt and Householder QR algorithms, and discuss their application to Krylov methods for the solution of large-scale linear algebra problems, such as linear systems of equations, eigenvalue problems, the evaluation of matrix functions, and matrix equations.

math.NA↗

arXiv2025

Adaptive Randomized Tensor Train Rounding using Khatri-Rao Products

Approximating a tensor in the tensor train (TT) format has many important applications in scientific computing. Rounding a TT tensor involves further compressing a tensor that is already in the TT format. This paper proposes new randomized algorithms for TT-rounding that uses sketches based on Khatri-Rao products (KRP). When the TT-ranks are known in advance, the proposed methods are comparable in cost to the sketches that used a sketching matrix in the TT-format~\cite{al2023randomized}. However, the use of KRP sketches enables adaptive algorithms to round the tensor in the TT-format within a fixed user-specified tolerance. An important component of the adaptivity is the estimation of error using KRP sketching, for which we develop theoretical guarantees. We report numerical experiments on synthetic tensors, parametric low-rank kernel approximations, and the solution of parametric partial differential equations. The numerical experiments show that we obtain speed-ups of up to $50\times$ compared to deterministic TT-rounding. Both the computational cost analysis and numerical experiments verify that the adaptive algorithms are competitive with the fixed rank algorithms, suggesting the adaptivity introduces only a low overhead.

math.NA↗

arXiv2025

Randomized biorthogonalization through a two-sided Gram-Schmidt process

We propose and analyze a randomized two-sided Gram-Schmidt process for the biorthogonalization of two given matrices $X, Y \in\mathbb{R}^{n\times m}$. The algorithm aims to find two matrices $Q, P \in\mathbb{R}^{n\times m}$ such that ${\rm range}(X) = {\rm range}(Q)$, ${\rm range}(Y) = {\rm range}(P)$ and $(\Omega Q)^T \Omega P = I$, where $\Omega \in\mathbb{R}^{s \times n}$ is a sketching matrix satisfying an oblivious subspace $\varepsilon$-embedding property; in other words, the biorthogonality condition on the columns of $Q$ and $P$ is replaced by an equivalent condition on their sketches. This randomized approach is computationally less expensive than the classical two-sided Gram-Schmidt process, has better numerical stability, and the condition number of the computed bases $Q, P$ is often smaller than in the deterministic case. Several different implementations of the randomized algorithm are analyzed and compared numerically. The randomized two-sided Gram-Schmidt process is applied to the nonsymmetric Lancozs algorithm for the approximation of eigenvalues and both left and right eigenvectors.

math.NA↗

arXiv2025

Efficient QR-based Column Subset Selection through Randomized Sparse Embeddings

In this paper, we introduce an efficient algorithm for column subset selection that combines the column-pivoted QR factorization with sparse subspace embeddings. The proposed method, SE-QRCS, is particularly effective for wide matrices with significantly more columns than rows. Starting from a matrix $A$, the algorithm selects $k$ columns from the sketched matrix $B = A \Omega^T$, where $\Omega$ is a sparse oblivious subspace embedding for a subspace of dimension $rank(A)$. The sparsity structure of $\Omega$ is then exploited to map the selected pivots back to the corresponding columns of $A$, which are then used to produce the final subset of selected columns. We prove that this procedure yields a factorization with strong rank-revealing properties, thus revealing the spectrum of $A$. The resulting bounds exhibit a reduced dependence on the number of columns of $A$ compared to those obtained from the strong rank-revealing QR factorization of $A$. For general matrices, the algorithm can be extended by first applying an additional subspace embedding of $range(A)$.

math.NA↗

arXiv2025

Randomized Krylov-Schur eigensolver with deflation

This work introduces a novel algorithm to solve large-scale eigenvalue problems and seek a small set of eigenpairs. The method, called randomized Krylov-Schur (rKS), has a simple implementation and benefits from fast and efficient operations in low-dimensional spaces, such as sketch-orthogonalization processes and stable reordering of Schur factorizations. It also includes a practical deflation technique for converged eigenpairs, enabling the computation of the eigenspace associated with a given part of the spectrum. Numerical experiments are provided to demonstrate the scalability and accuracy of the method.

math.NA↗

arXiv2025

Minimizing Communication for Parallel Symmetric Tensor Times Same Vector Computation

In this article, we focus on the parallel communication cost of multiplying the same vector along two modes of a $3$-dimensional symmetric tensor. This is a key computation in the higher-order power method for determining eigenpairs of a $3$-dimensional symmetric tensor and in gradient-based methods for computing a symmetric CP decomposition. We establish communication lower bounds that determine how much data movement is required to perform the specified computation in parallel. The core idea of the proof relies on extending a key geometric inequality for $3$-dimensional symmetric computations. We demonstrate that the communication lower bounds are tight by presenting an optimal algorithm where the data distribution is a natural extension of the triangle block partition scheme for symmetric matrices to 3-dimensional symmetric tensors.

cs.DC↗

arXiv2025

An additive two-level parallel variant of the DMRG algorithm with coarse-space correction

The density matrix renormalization group (DMRG) algorithm is a popular alternating minimization scheme for solving high-dimensional optimization problems in the tensor train format. Classical DMRG, however, is based on sequential minimization, which raises challenges in its implementation on parallel computing architectures. To overcome this, we propose a novel additive two-level DMRG algorithm that combines independent, local minimization steps with a global update step using a subsequent coarse-space minimization. Our proposed algorithm, which is directly inspired by additive Schwarz methods from the domain decomposition literature, is particularly amenable to implementation on parallel, distributed architectures since both the local minimization steps and the construction of the coarse-space can be performed in parallel. Numerical experiments on strongly correlated molecular systems demonstrate that the method achieves competitive convergence rates while achieving significant parallel speedups.

math.NA↗

arXiv2025

Randomized strong rank-revealing QR for column subset selection and low-rank matrix approximation

We discuss a randomized strong rank-revealing QR factorization that effectively reveals the spectrum of a matrix $\textbf{M}$. This factorization can be used to address problems such as selecting a subset of the columns of $\textbf{M}$, computing its low-rank approximation, estimating its rank, or approximating its null space. Given a random sketching matrix $\pmb{\Omega}$ that satisfies the $\epsilon$-embedding property for a subspace within the range of $\textbf{M}$, the factorization relies on selecting columns that allow to reveal the spectrum via a deterministic strong rank-revealing QR factorization of $\textbf{M}^{sk} = \pmb{\Omega}\textbf{M}$, the sketch of $\textbf{M}$. We show that this selection leads to a factorization with strong rank-revealing properties, making it suitable for approximating the singular values of $\textbf{M}$.

math.NA↗

arXiv2024

Communication Lower Bounds and Optimal Algorithms for Symmetric Matrix Computations

In this article, we focus on the communication costs of three symmetric matrix computations: i) multiplying a matrix with its transpose, known as a symmetric rank-k update (SYRK) ii) adding the result of the multiplication of a matrix with the transpose of another matrix and the transpose of that result, known as a symmetric rank-2k update (SYR2K) iii) performing matrix multiplication with a symmetric input matrix (SYMM). All three computations appear in the Level 3 Basic Linear Algebra Subroutines (BLAS) and have wide use in applications involving symmetric matrices. We establish communication lower bounds for these kernels using sequential and distributed-memory parallel computational models, and we show that our bounds are tight by presenting communication-optimal algorithms for each setting. Our lower bound proofs rely on applying a geometric inequality for symmetric computations and analytically solving constrained nonlinear optimization problems. The symmetric matrix and its corresponding computations are accessed and performed according to a triangular block partitioning scheme in the optimal algorithms.

cs.DC↗

arXiv2024

Randomized Implicitly Restarted Arnoldi method for the non-symmetric eigenvalue problem

In this paper, we introduce a randomized algorithm for solving the non-symmetric eigenvalue problem, referred to as randomized Implicitly Restarted Arnoldi (rIRA). This method relies on using a sketch-orthogonal basis during the Arnoldi process while maintaining the Arnoldi relation and exploiting a restarting scheme to focus on a specific part of the spectrum. We analyze this method and show that it retains useful properties of the Implicitly Restarted Arnoldi (IRA) method, such as restarting without adding errors to the Ritz pairs and implicitly applying polynomial filtering. Experiments are presented to validate the numerical efficiency of the proposed randomized eigenvalue solver.

math.NA↗

arXiv2024

Randomized Householder QR

This paper introduces a randomized Householder QR factorization (RHQR). This factorization can be used to obtain a well conditioned basis of a vector space and thus can be employed in a variety of applications. The RHQR factorization of the input matrix $W$ is equivalent to the standard Householder QR factorization of matrix $ΨW$, where $Ψ$ is a sketching matrix that can be obtained from any subspace embedding technique. For this reason, the RHQR factorization can also be reconstructed from the Householder QR factorization of the sketched problem, yielding a single-synchronization randomized QR factorization (recRHQR). In most contexts, left-looking RHQR requires a single synchronization per iteration, with half the computational cost of Householder QR, and a similar cost to Randomized Gram-Schmidt (RGS) overall. We discuss the usage of RHQR factorization in the Arnoldi process and then in GMRES, showing thus how it can be used in Krylov subspace methods to solve systems of linear equations. Based on Charles Sheffield's connection between Householder QR and Modified Gram-Schmidt (MGS), a BLAS2-RGS is also derived. A finite precision analysis shows that, under mild probabilistic assumptions, the RHQR factorization of the input matrix $W$ inherits the stability of the Householder QR factorization, producing a well-conditioned basis and a columnwise backward stable factorization, all independently of the condition number of the input $W$, and with the accuracy of the sketching step. We study the subsampled randomized Hadamard transform (SRHT) as a very stable sketching technique. Numerical experiments show that RHQR produces a well conditioned basis whose sketch is numerically orthogonal and an accurate factorization, even for the most difficult inputs and with high-dimensional operations made in half-precision.

math.NA↗

arXiv2024

Fast Exact Leverage Score Sampling from Khatri-Rao Products with Applications to Tensor Decomposition

We present a data structure to randomly sample rows from the Khatri-Rao product of several matrices according to the exact distribution of its leverage scores. Our proposed sampler draws each row in time logarithmic in the height of the Khatri-Rao product and quadratic in its column count, with persistent space overhead at most the size of the input matrices. As a result, it tractably draws samples even when the matrices forming the Khatri-Rao product have tens of millions of rows each. When used to sketch the linear least squares problems arising in CANDECOMP / PARAFAC tensor decomposition, our method achieves lower asymptotic complexity per solve than recent state-of-the-art methods. Experiments on billion-scale sparse tensors validate our claims, with our algorithm achieving higher accuracy than competing methods as the decomposition rank grows.

math.NA↗