arXiv subjects

Josh Alman

Publications and source records attributed to Josh Alman.

At least 19 recordsLinked to original sources

arXiv2026

Improving the matrix multiplication exponent with modern optimization and AlphaEvolve

The current best bounds on the matrix multiplication exponent $\omega$ are obtained through a refinement of the laser method called combination loss analysis (Duan et al., 2022; Williams et al., 2024; Alman et al., 2025). In this note, we address the optimization problem at the core of this approach and propose several improvements. First, we reformulate the optimization problem allowing us to solve it in a larger setting than was previously possible. Second, we leverage recent advances in machine learning to design a new optimization algorithm for this problem. Finally, we refine the resulting optimization algorithm with AlphaEvolve. Our combined approach yields an upper bound of $\omega$ < 2.371177, improving the previous best bound of 2.371339.

cs.DS↗

arXiv2026

An algorithm for $k$-set cover

We show that set cover on a universe of size $n$ and with sets of size at most $k$ can be solved in time $2^{(1-1/k+O(1/k^{3/2}))n}$. This improves on a $2^{(1-0.929/k)n}$-time algorithm of Bj\"orklund (STACS 2010) for all sufficiently large $k$.

cs.DS↗

arXiv2026

Superlogarithmic-Rank Matrix Rigidity for the Walsh-Hadamard Transform

For sufficiently large $N$ which is a power of 2, we prove that changing at most one percent of the entries of the $N\times N$ Walsh-Hadamard Transform cannot reduce its rank over $\mathbb{F}_3$ to $\lfloor \log^2 N/80\rfloor$ or below. To the best of our knowledge, this is the first constant-fraction rigidity lower bound for an explicit matrix family at a superlogarithmic target rank over any choice of field, inching toward the parameters in Razborov's program for communication complexity lower bounds.

cs.CC↗

arXiv2026

Asymptotic Rank Speedup Theorems, Revisited

Motivated by fast matrix multiplication and recent connections between asymptotic tensor rank and fine-grained complexity, we revisit classical tools from the matrix multiplication literature and develop a framework for obtaining improved asymptotic rank upper bounds for tensors beyond matrix multiplication. In the 1980s, Coppersmith-Winograd and Strassen discovered a series of speedup theorems for asymptotic rank: in certain regimes, one can extract additional terms from a border rank upper bound on a tensor $T$, and then use these terms to obtain an improved asymptotic rank of $T$. We establish general speedup theorems that subsume these results and enable quantitative improvements. Two representative applications are: (1) The asymptotic rank of the small Coppersmith-Winograd tensor $\mathrm{cw}_q$ is less than its border rank. For instance, we prove the asymptotic rank of $\mathrm{cw}_2$ is smaller than $3.931$, improving on $\underline{\mathrm{R}}(\mathrm{cw}_2)=4$. It is known that if the asymptotic rank of $\mathrm{cw}_2$ equals $3$, this would imply $\omega=2$. (2) A general improvement over Strassen's bound: we obtain an upper bound below $d^{2\omega/3}$ on the asymptotic rank of any $d\times d\times d$ tensor. To make full use of speedups, we analyze degenerations in which both sides are nontrivial direct sums, a setting where the optimal quantitative bound one can achieve was previously unclear. We do so via an approach we call Strassen calculus: a systematic method for converting such degeneration data into explicit asymptotic rank bounds using Strassen's theory of the asymptotic spectrum.

cs.CC↗

arXiv2026

The edge of the asymptotic spectrum of tensors

Strassen founded the theory of the asymptotic spectrum of tensors to study the complexity of matrix multiplication. A central challenge in this theory is to explicitly construct new spectral points. In Crelle 1991, Strassen proposed the upper support functionals $\zeta^\theta$ as candidate spectral points, where $\theta$ ranges over a triangle $\Theta$. Recent progress, involving tools and ideas from quantum information theory (Christandl-Vrana-Zuiddam, STOC 2018, JAMS 2021) and convex optimization (Hirai, 2025), culminated in the proof that the upper support functionals are indeed spectral points over the complex numbers (Sakabe-Do\u{g}an-Walter, 2026). In this paper, we give an even clearer picture of the situation for support functionals when $\theta$ lies along the edges of the triangle. We show that not only are these functionals spectral points, but that they are uniquely determined as spectral points by their behavior on matrix multiplication tensors. As our methods are algebraic, as a corollary this establishes for the first time the existence of nontrivial spectral points over arbitrary fields. As part of our argument, we show a close connection between the edge support functionals and Harder-Narasimhan filtrations from quiver representation theory. We thus show, using recent work in algorithmic invariant theory, that these support functionals can be computed in deterministic polynomial time. Other ingredients of our proof include a new criterion for abstractly characterizing asymptotic tensor ranks by spectral points, and a characterization of the edge support functionals in terms of matrix multiplication capacity. As another application of these tools, we prove the existence of spectral points for higher-mode tensors beyond those currently known.

cs.CC↗

arXiv2026

Learning Functions of Halfspaces

We give an algorithm that learns arbitrary Boolean functions of $k$ arbitrary halfspaces over $\mathbb{R}^n$, in the challenging distribution-free Probably Approximately Correct (PAC) learning model, running in time $2^{\sqrt{n} \cdot (\log n)^{O(k)}}$. This is the first algorithm that can PAC learn even intersections of two halfspaces in time $2^{o(n)}.$

cs.DS↗

arXiv2026

Poly-attention: a general scheme for higher-order self-attention

The self-attention mechanism, at the heart of the Transformer model, is able to effectively model pairwise interactions between tokens. However, numerous recent works have shown that it is unable to perform basic tasks involving detecting triples of correlated tokens, or compositional tasks where multiple input tokens need to be referenced to generate a result. Some higher-dimensional alternatives to self-attention have been proposed to address this, including higher-order attention and Strassen attention, which can perform some of these polyadic tasks in exchange for slower, superquadratic running times. In this work, we define a vast class of generalizations of self-attention, which we call poly-attention mechanisms. Our mechanisms can incorporate arbitrary higher-order (tensor) computations as well as arbitrary relationship structures between the input tokens, and they include the aforementioned alternatives as special cases. We then systematically study their computational complexity and representational strength, including giving new algorithms and matching complexity-theoretic lower bounds on the time complexity of computing the attention matrix exactly as well as approximately, and tightly determining which polyadic tasks they can each perform. Our results give interesting trade-offs between different desiderata for these mechanisms, including a tight relationship between how expressive a mechanism is, and how large the coefficients in the model may be so that the mechanism can be approximated in almost-linear time. Notably, we give a new attention mechanism which can be computed exactly in quadratic time, and which can perform function composition for any fixed number of functions. Prior mechanisms, even for just composing two functions, could only be computed in superquadratic time, and our new lower bounds show that faster algorithms for them are not possible.

cs.LG↗

arXiv2026

Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers

Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function {\Gamma}, inspired by the equality function, and prove that a one-layer softmax Transformer can compute {\Gamma}, with p bits of precision, but not with p-1 bits of precision. This result concretely explains the widely observed phenomenon of empirical loss of expressivity when quantization is used. Practically, it suggests that tasks requiring equality-like comparisons (exact match, membership, etc.) are especially sensitive to quantization. Dropping even one bit can cross a threshold where the model cannot represent the needed comparison reliably. Thus, it paves the way for developing heuristics that will help practitioners choose how much quantization is possible: the precision should be chosen as a function of the length of equality to be checked for the specific task. Our proofs combine explicit finite-precision Transformer constructions with communication-complexity lower bounds, yielding a tight "one-bit" threshold.

cs.LG↗

arXiv2025

Kronecker Powers, Orthogonal Vectors, and the Asymptotic Spectrum

We study circuits for computing depth-2 linear transforms defined by Kronecker power matrices. Recent works have improved on decades-old constructions in this area using a new ''rebalancing'' approach [Alman, Guan and Padaki, SODA'23; Sergeev'22], but it was unclear how to apply this approach optimally. We find that Strassen's theory of asymptotic spectra can be applied to capture the design of these circuits. In particular, in hindsight, we find that the techniques of recent work on rebalancing were proving special cases of the duality theorem, which is central to Strassen's theory. We carefully outline a collection of ''obstructions'' to designing small depth-2 circuits using a rebalancing approach, and apply Strassen's theory to show that our obstructions are complete. Using this connection, combined with other algorithmic techniques, we give new improved circuit constructions as well as other applications, including: - The $N \times N$ disjointness matrix has a depth-2 linear circuit of size $O(N^{1.2495})$ over any field. This also yield smaller circuits for many families of matrices using reductions to disjointness. - The Strong Exponential Time Hypothesis implies an $N^{1 + \Omega(1)}$ size lower bound for depth-2 linear circuits computing the Walsh--Hadamard transform (and the disjointness matrix with a technical caveat), and proving a $N^{1 + \Omega(1)}$ depth-2 size lower bound would also imply breakthrough threshold circuit lower bounds. - The Orthogonal Vectors (OV) problem in moderate dimension $d$ can be solved in deterministic time $\tilde{O}(n \cdot 1.155^d)$, derandomizing an algorithm of Nederlof and W\k{e}grzycki [STOC'21], and the counting problem can be solved in time $\tilde{O}(n \cdot 1.26^d)$, improving an algorithm of Williams [FOCS'24] which runs in time $\tilde{O}(n \cdot 1.35^d)$.

cs.DS↗

arXiv2025

Faster exact learning of k-term DNFs with membership and equivalence queries

In 1992 Blum and Rudich [BR92] gave an algorithm that uses membership and equivalence queries to learn $k$-term DNF formulas over $\{0,1\}^n$ in time $\textsf{poly}(n,2^k)$, improving on the naive $O(n^k)$ running time that can be achieved without membership queries [Val84]. Since then, many alternative algorithms [Bsh95, Kus97, Bsh97, BBB+00] have been given which also achieve runtime $\textsf{poly}(n,2^k)$. We give an algorithm that uses membership and equivalence queries to learn $k$-term DNF formulas in time $\textsf{poly}(n) \cdot 2^{\tilde{O}(\sqrt{k})}$. This is the first improvement for this problem since the original work of Blum and Rudich [BR92]. Our approach employs the Winnow2 algorithm for learning linear threshold functions over an enhanced feature space which is adaptively constructed using membership queries. It combines a strengthened version of a technique that effectively reduces the length of DNF terms from the original work of [BR92] with a range of additional algorithmic tools (attribute-efficient learning algorithms for low-weight linear threshold functions and techniques for finding relevant variables from junta testing) and analytic ingredients (extremal polynomials and noise operators) that are novel in the context of query-based DNF learning.

cs.DS↗

arXiv2025

Two Heads Are Better than One: Simulating Large Transformers with Small Ones

The quadratic complexity of self-attention prevents transformers from scaling effectively to long input sequences. On the other hand, modern GPUs and other specialized hardware accelerators are well-optimized for processing small input sequences in transformers during both training and inference. A natural question arises: can we take advantage of the efficiency of small transformers to deal with long input sequences? In this paper, we show that transformers with long input sequences (large transformers) can be efficiently simulated by transformers that can only take short input sequences (small transformers). Specifically, we prove that any transformer with input length $N$ can be efficiently simulated by only $O((N/M)^2)$ transformers with input length $M \ll N$, and that this cannot be improved in the worst case. However, we then prove that in various natural scenarios including average-case inputs, sliding window masking and attention sinks, the optimal number $O(N/M)$ of small transformers suffice.

cs.LG↗

arXiv2025

DNF Learning via Locally Mixing Random Walks

We give two results on PAC learning DNF formulas using membership queries in the challenging "distribution-free" learning framework, where learning algorithms must succeed for an arbitrary and unknown distribution over $\{0,1\}^n$. (1) We first give a quasi-polynomial time "list-decoding" algorithm for learning a single term of an unknown DNF formula. More precisely, for any target $s$-term DNF formula $f = T_1 \vee \cdots \vee T_s$ over $\{0,1\}^n$ and any unknown distribution $D$ over $\{0,1\}^n$, our algorithm, which uses membership queries and random examples from $D$, runs in $\textsf{quasipoly}(n,s)$ time and outputs a list $L$ of candidate terms such that with high probability some term $T_i$ of $f$ belongs to $L$. (2) We then use result (1) to give a $\textsf{quasipoly}(n,s)$-time algorithm, in the distribution-free PAC learning model with membership queries, for learning the class of size-$s$ DNFs in which all terms have the same size. Our algorithm learns using a DNF hypothesis. The key tool used to establish result (1) is a new result on "locally mixing random walks," which, roughly speaking, shows that a random walk on a graph that is covered by a small number of expanders has a non-negligible probability of mixing quickly in a subset of these expanders.

cs.DS↗

arXiv2025

Only Large Weights (And Not Skip Connections) Can Prevent the Perils of Rank Collapse

Attention mechanisms lie at the heart of modern large language models (LLMs). Straightforward algorithms for forward and backward (gradient) computation take quadratic time, and a line of work initiated by [Alman and Song NeurIPS 2023] and [Alman and Song NeurIPS 2024] has shown that quadratic time is necessary unless the model weights are small, in which case almost linear time algorithms are possible. In this paper, we show that large weights are necessary to avoid a strong preclusion to representational strength we call layer collapse, which means that the entire network can be approximated well by a network with only a single layer. Thus, the quadratic running time of attention is unavoidable for expressive transformers. The notion of layer collapse that we introduce is a variant on the notion of rank collapse from the work of [Dong, Cordonnier, and Loukas ICML 2021]. They showed that in Self Attention Networks with small weights and with skip connections, rank collapse must occur. This is typically interpreted as justifying the necessity of skip connections in expressive networks. However, our result shows that even with skip connections, if the weights are small, then layer collapse still occurs. Thus, only large weights, and not skip connections, can prevent these representational weaknesses.

cs.LG↗

arXiv2025

Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transform

The transformer architecture has been widely applied to many machine learning tasks. A main bottleneck in the time to perform transformer computations is a task called attention computation. [Alman and Song, NeurIPS 2023] have shown that in the bounded entry regime, there is an almost linear time algorithm to approximate the attention computation. They also proved that the bounded entry assumption is necessary for a fast algorithm assuming the popular Strong Exponential Time Hypothesis. A new version of transformer which uses position embeddings has recently been very successful. At a high level, position embedding enables the model to capture the correlations between tokens while taking into account their position in the sequence. Perhaps the most popular and effective version is Rotary Position Embedding (RoPE), which was proposed by [Su, Lu, Pan, Murtadha, Wen, and Liu, Neurocomputing 2024]. A main downside of RoPE is that it complicates the attention computation problem, so that previous techniques for designing almost linear time algorithms no longer seem to work. In this paper, we show how to overcome this issue, and give a new algorithm to compute the RoPE attention in almost linear time in the bounded entry regime. (Again, known lower bounds imply that bounded entries are necessary.) Our new algorithm combines two techniques in a novel way: the polynomial method, which was used in prior fast attention algorithms, and the Fast Fourier Transform.

cs.LG↗

arXiv2025

Low Rank Matrix Rigidity: Tight Lower Bounds and Hardness Amplification

For an $N \times N$ matrix $A$, its rank-$r$ rigidity, denoted $\mathcal{R}_A(r)$, is the minimum number of entries of $A$ that one must change to make its rank become at most $r$. Determining the rigidity of interesting explicit families of matrices remains a major open problem, and is central to understanding the complexities of these matrices in many different models of computation and communication. We focus in this paper on the Walsh-Hadamard transform and on the `distance matrix', whose rows and columns correspond to binary vectors, and whose entries calculate whether the row and column are close in Hamming distance. Our results also generalize to other Kronecker powers and `Majority powers' of fixed matrices. We prove two new results about such matrices. First, we prove new rigidity lower bounds in the low-rank regime where $r < \log N$. For instance, we prove that over any finite field, there are constants $c_1, c_2 > 0$ such that the $N \times N$ Walsh-Hadamard matrix $H_n$ satisfies $$\mathcal{R}_{H_n}(c_1 \log N) \geq N^2 \left( \frac12 - N^{-c_2} \right),$$ and a similar lower bound for the other aforementioned matrices. This is tight, and is the new best rigidity lower bound for an explicit matrix family at this rank; the previous best was $\mathcal{R}(c_1 \log N) \geq c_3 N^2$ for a small constant $c_3>0$. Second, we give new hardness amplification results, showing that rigidity lower bounds for these matrices for slightly higher rank would imply breakthrough rigidity lower bounds for much higher rank. For instance, if one could prove $$\mathcal{R}_{H_n}(\log^{1 + \varepsilon} N) \geq N^2 \left( \frac12 - N^{-1/2^{(\log \log N)^{o(1)}}} \right)$$ over any finite field for some $\varepsilon>0$, this would imply that $H_n$ is Razborov rigid, giving a breakthrough lower bound in communication complexity.

cs.CC↗

arXiv2024

Faster Algorithms for Average-Case Orthogonal Vectors and Closest Pair Problems

We study the average-case version of the Orthogonal Vectors problem, in which one is given as input $n$ vectors from $\{0,1\}^d$ which are chosen randomly so that each coordinate is $1$ independently with probability $p$. Kane and Williams [ITCS 2019] showed how to solve this problem in time $O(n^{2 - \delta_p})$ for a constant $\delta_p > 0$ that depends only on $p$. However, it was previously unclear how to solve the problem faster in the hardest parameter regime where $p$ may depend on $d$. The best prior algorithm was the best worst-case algorithm by Abboud, Williams and Yu [SODA 2014], which in dimension $d = c \cdot \log n$, solves the problem in time $n^{2 - \Omega(1/\log c)}$. In this paper, we give a new algorithm which improves this to $n^{2 - \Omega(\log\log c /\log c)}$ in the average case for any parameter $p$. As in the prior work, our algorithm uses the polynomial method. We make use of a very simple polynomial over the reals, and use a new method to analyze its performance based on computing how its value degrades as the input vectors get farther from orthogonal. To demonstrate the generality of our approach, we also solve the average-case version of the closest pair problem in the same running time.

cs.DS↗

arXiv2024

Improving the Leading Constant of Matrix Multiplication

Algebraic matrix multiplication algorithms are designed by bounding the rank of matrix multiplication tensors, and then using a recursive method. However, designing algorithms in this way quickly leads to large constant factors: if one proves that the tensor for multiplying $n \times n$ matrices has rank $\leq t$, then the resulting recurrence shows that $M \times M$ matrices can be multiplied using $O(n^2 \cdot M^{\log_n t})$ operations, where the leading constant scales proportionally to $n^2$. Even modest increases in $n$ can blow up the leading constant too much to be worth the slight decrease in the exponent of $M$. Meanwhile, the asymptotically best algorithms use very large $n$, such that $n^2$ is larger than the number of atoms in the visible universe! In this paper, we give new ways to use tensor rank bounds to design matrix multiplication algorithms, which lead to smaller leading constants than the standard recursive method. Our main result shows that, if the tensor for multiplying $n \times n$ matrices has rank $\leq t$, then $M \times M$ matrices can be multiplied using only $n^{O(1/(\log n)^{0.33})} \cdot M^{\log_n t}$ operations. In other words, we improve the leading constant in general from $O(n^2)$ to $n^{O(1/(\log n)^{0.33})} < n^{o(1)}$. We then apply this and further improve the leading constant in a number of situations of interest. We show that, in the popularly-conjectured case where $\omega=2$, a new, different recursive approach can lead to an improvement. We also show that the leading constant of the current asymptotically fastest matrix multiplication algorithm, and any algorithm designed using the group-theoretic method, can be further improved by taking advantage of additional structure of the underlying tensor identities.

cs.CC↗

arXiv2024

Fundamental Limitations on Subquadratic Alternatives to Transformers

The Transformer architecture is widely deployed in many popular and impactful Large Language Models. At its core is the attention mechanism for calculating correlations between pairs of tokens. Performing an attention computation takes quadratic time in the input size, and had become the time bottleneck for transformer operations. In order to circumvent this, researchers have used a variety of approaches, including designing heuristic algorithms for performing attention computations faster, and proposing alternatives to the attention mechanism which can be computed more quickly. For instance, state space models such as Mamba were designed to replace attention with an almost linear time alternative. In this paper, we prove that any such approach cannot perform important tasks that Transformer is able to perform (assuming a popular conjecture from fine-grained complexity theory). We focus on document similarity tasks, where one is given as input many documents and would like to find a pair which is (approximately) the most similar. We prove that Transformer is able to perform this task, and we prove that this task cannot be performed in truly subquadratic time by any algorithm. Thus, any model which can be evaluated in subquadratic time - whether because of subquadratic-time heuristics for attention, faster attention replacements like Mamba, or any other reason - cannot perform this task. In other words, in order to perform tasks that (implicitly or explicitly) involve document similarity, one may as well use Transformer and cannot avoid its quadratic running time.

cs.LG↗