arXiv · 2611.12186
NearFormer: Linear-Size Sparse Attention via Randomized Near Neighbors
Abstract
Graph transformers can capture long-range interactions, but their quadratic attention cost limits their use on large graphs. We introduce \emph{NearFormer}, a graph transformer whose global attention is restricted to a randomized near-neighbor (NN) graph. NearFormer uses an efficient randomized construction to connect each node to a small number of near neighbors from a broader neighborhood defined by a label-independent feature or spectral geometry. For a fixed retained budget \(k\), the resulting attention support has linear size \(\mathcal{O}(|E|+nk)\). The NN graph is built once by within-bin search and then held fixed. We use these edges as sparse global-attention supports, with optional learned attention biases, while preserving the original graph for local message passing. Experiments on long-range graph benchmarks and synthetic structural tasks show that NearFormer provides a favorable accuracy--scalability trade-off relative to local message passing, dense and sparse attention.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Ali Parviz, Alex Cloninger, Gal Mishne. 2026-11-09. NearFormer: Linear-Size Sparse Attention via Randomized Near Neighbors. https://searcharxiv.com/abs/2611.12186
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.