arXiv subjects

Stanislav Budzinskiy

Publications and source records attributed to Stanislav Budzinskiy.

15 recordsLinked to original sources

arXiv2026

LAMP: Look-Ahead Mixed-Precision Inference of Large Language Models

Mixed-precision computations are a hallmark of the current stage of AI, driving the progress in large language models towards efficient, locally deployable solutions. This article addresses the floating-point computation of compositionally-rich functions, concentrating on transformer inference. Based on the rounding error analysis of a composition $f(g(\mathrm{x}))$, we provide an adaptive strategy that selects a small subset of components of $g(\mathrm{x})$ to be computed more accurately while all other computations can be carried out with lower accuracy. We then explain how this strategy can be applied to different compositions within a transformer and illustrate its overall effect on transformer inference. We study the effectiveness of this algorithm numerically on GPT-2 models and demonstrate that already very low recomputation rates allow for improvements of up to two orders of magnitude in accuracy.

cs.LG↗

arXiv2025

Low-rank cross approximation of function-valued tensors for reduced-order modeling of parametric PDEs

The paper considers function-valued tensors, viewed as multidimensional arrays with entries in an abstract Hilbert space. Despite the absence of the algebraic structure of a field, the geometric inner-product structure suffices to introduce the Tucker rank, higher-order SVD, and Tucker-cross decomposition for function-valued tensors. An adaptive cross-approximation algorithm is developed to compute low-rank approximations of such tensors. The framework is motivated by, and applied to, model order reduction of the parameter-to-solution map for a parametric PDE. The resulting reduced-order model can be interpreted as an encoder-decoder scheme with a nonlinear encoder and a multilinear decoder. The performance of the proposed non-intrusive approximation method is demonstrated in numerical examples for two nonlinear parametric PDE systems.

math.NA↗

arXiv2025

Matrices over a Hilbert space and their low-rank cross approximation

Motivated by applications in reduced-order modeling (ROM) of parametric partial differential equations, we investigate the algebraic properties of Bochner matrices -- matrices with entries in an abstract Hilbert space. Low-rank cross approximation is extended to Bochner matrices and its approximation guarantees are derived. The high-dimensional nature of the entries is shown to manifest itself in maximum-volume bounds, making them smaller than in the classical setting. An analogue of adaptive cross approximation is proposed and validated as a non-intrusive ROM method in numerical experiments with parametric nonlinear Stokes equations.

math.NA↗

arXiv2025

Numerical stability analysis of large language models

Transformers are the state-of-the-art architecture for large language models, and a key to their scalability is the strategic usage of low-precision arithmetic. We develop a mixed-precision analysis of transformer inference, deriving bounds for the condition numbers and forward error of the architecture's constituent parts. Notably, we compare the numerical stability of LayerNorm and RMSNorm in the massive-outlier regime, tighten the error bound of softmax in the presence of attention sinks, and quantify the impact of its shifted evaluation on the sensitivity to perturbations. Furthermore, we derive novel sequence-length-independent bounds on the local Lipschitz constant of self-attention. Our worst-case error bound for transformer inference suggests that its numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream. Crucially, and as validated by experiments with GPT-2, our analysis establishes that the scaling of residual-projection weights preserves the propagation of the relative rounding error unless it forces a qualitative transition in the dynamics of the residual stream.

math.NA↗

arXiv2024

When big data actually are low-rank, or entrywise approximation of certain function-generated matrices

The article concerns low-rank approximation of matrices generated by sampling a smooth function of two $m$-dimensional variables. We identify several misconceptions surrounding a claim that, for a specific class of analytic functions, such $n \times n$ matrices admit accurate entrywise approximation of rank that is independent of $m$ and grows as $\log(n)$ -- colloquially known as ''big-data matrices are approximately low-rank''. We provide a theoretical explanation of the numerical results presented in support of this claim, describing three narrower classes of functions for which function-generated matrices can be approximated within an entrywise error of order $\varepsilon$ with rank $\mathcal{O}(\log(n) \varepsilon^{-2} \log(\varepsilon^{-1}))$ that is independent of the dimension $m$: (i) functions of the inner product of the two variables, (ii) functions of the Euclidean distance between the variables, and (iii) shift-invariant positive-definite kernels. We extend our argument to tensor-train approximation of tensors generated with functions of the ''higher-order inner product'' of their multiple variables. We discuss our results in the context of low-rank approximation of (a) growing datasets and (b) attention in transformer neural networks.

math.NA↗

arXiv2024

Entrywise tensor-train approximation of large tensors via random embeddings

The theory of low-rank tensor-train approximation is well understood when the approximation error is measured in the Frobenius norm. The entrywise maximum norm is equally important but is significantly weaker for large tensors, making the estimates obtained via the Frobenius norm and norm equivalence pessimistic or even meaningless. In this article, we derive a direct estimate of the entrywise approximation error that is applicable in some of these cases. The estimate is given in terms of the higher-order generalization of the matrix factorization norm, and its proof is based on the tensor-structured Hanson--Wright inequality. The theoretical results are accompanied by numerical experiments carried out with the method of alternating projections.

math.NA↗

arXiv2023

On the distance to low-rank matrices in the maximum norm

Every sufficiently big matrix with small spectral norm has a nearby low-rank matrix if the distance is measured in the maximum norm (Udell & Townsend, SIAM J Math Data Sci, 2019). We use the Hanson--Wright inequality to improve the estimate of the distance for matrices with incoherent column and row spaces. In numerical experiments with several classes of matrices we study how well the theoretical upper bound describes the approximation errors achieved with the method of alternating projections.

math.NA↗

arXiv2023

Quasioptimal alternating projections and their use in low-rank approximation of matrices and tensors

We study the convergence of specific inexact alternating projections for two non-convex sets in a Euclidean space. The $\sigma$-quasioptimal metric projection ($\sigma \geq 1$) of a point $x$ onto a set $A$ consists of points in $A$ the distance to which is at most $\sigma$ times larger than the minimal distance $\mathrm{dist}(x,A)$. We prove that quasioptimal alternating projections, when one or both projections are quasioptimal, converge locally and linearly for super-regular sets with transversal intersection. The theory is motivated by the successful application of alternating projections to low-rank matrix and tensor approximation. We focus on two problems -- nonnegative low-rank approximation and low-rank approximation in the maximum norm -- and develop fast alternating-projection algorithms for matrices and tensor trains based on cross approximation and acceleration techniques. The numerical experiments confirm that the proposed methods are efficient and suggest that they can be used to regularise various low-rank computational routines.

math.OC↗

arXiv2023

Low-rank nonnegative tensor approximation via alternating projections and sketching

We show how to construct nonnegative low-rank approximations of nonnegative tensors in Tucker and tensor train formats. We use alternating projections between the nonnegative orthant and the set of low-rank tensors, using STHOSVD and TTSVD algorithms, respectively, and further accelerate the alternating projections using randomized sketching. The numerical experiments on both synthetic data and hyperspectral images show the decay of the negative elements and that the error of the resulting approximation is close to the initial error obtained with STHOSVD and TTSVD. The proposed method for the Tucker case is superior to the previous ones in terms of computational complexity and decay of negative elements. The tensor train case, to the best of our knowledge, has not been studied before.

math.NA↗

arXiv2023

Sketching for low-rank nonnegative matrix approximation: Numerical study

We propose new approximate alternating projection methods, based on randomized sketching, for the low-rank nonnegative matrix approximation problem: find a low-rank approximation of a nonnegative matrix that is nonnegative, but whose factors can be arbitrary. We calculate the computational complexities of the proposed methods and evaluate their performance in numerical experiments. The comparison with the known deterministic alternating projection methods shows that the randomized approaches are faster and exhibit similar convergence properties.

math.NA↗

arXiv2022

Variational Bayesian inference for CP tensor completion with side information

We propose a message passing algorithm, based on variational Bayesian inference, for low-rank tensor completion with automatic rank determination in the canonical polyadic format when additional side information (SI) is given. The SI comes in the form of low-dimensional subspaces the contain the fiber spans of the tensor (columns, rows, tubes, etc.). We validate the regularization properties induced by SI with extensive numerical experiments on synthetic and real-world data and present the results about tensor recovery and rank determination. The results show that the number of samples required for successful completion is significantly reduced in the presence of SI. We also discuss the origin of a bump in the phase transition curves that exists when the dimensionality of SI is comparable with that of the tensor.

cs.LG↗

arXiv2021

Tensor train completion: local recovery guarantees via Riemannian optimization

In this work, we estimate the number of randomly selected elements of a tensor that with high probability guarantees local convergence of Riemannian gradient descent for tensor train completion. We derive a new bound for the orthogonal projections onto the tangent spaces based on the harmonic mean of the unfoldings' singular values and introduce a notion of core coherence for tensor trains. We also extend the results to tensor train completion with auxiliary subspace information and obtain the corresponding local convergence guarantees.

math.NA↗

arXiv2020

Note: low-rank tensor train completion with side information based on Riemannian optimization

We consider the low-rank tensor train completion problem when additional side information is available in the form of subspaces that contain the mode-$k$ fiber spans. We propose an algorithm based on Riemannian optimization to solve the problem. Numerical experiments show that the proposed algorithm requires far fewer known entries to recover the tensor compared to standard tensor train completion methods.

math.NA↗

arXiv2019

Pulsating and rotating spirals in a delayed feedback diffractive nonlinear optical system

We study spiral waves in a mathematical model of a nonlinear optical system with a feedback loop. Starting from a delayed scalar diffusion equation in a thin annulus with oblique derivative boundary conditions, we shrink the annulus and derive the limiting equation on a circle. Based on the explicitly constructed normal form of the Hopf bifurcation for the one-dimensional delayed scalar diffusion equation, we make predictions about the existence and stability of two-dimensional spirals that we verify in direct numerical simulations, observing pulsating and rotating spiral waves.

nlin.PS↗

arXiv2019

Zeros of Bessel cross-products coming from oblique derivative boundary value problems

The paper is devoted to (combinations of) Bessel cross-products that arise from oblique derivative boundary value problems for the Laplacian in a circular annulus. We show that like their Neumann-Laplacian counterpart (and unlike the Dirichlet-Laplacian), they possess two kinds of zeros: those that can be derived by McMahon series and diverge to infinity in the limit, and exceptional ones that remain finite. For both cases we find asymptotic expressions for a fixed oblique angle and vanishing thickness of the annulus. We further present plots of numerically computed zeros and discuss their behaviour when the oblique angle changes and the thickness remains fixed.

math.CA↗