arXiv subjects

Ran Ben-Basat

Publications and source records attributed to Ran Ben-Basat.

11 recordsLinked to original sources

arXiv2026

Quantizing With Randomized Hadamard Transforms: Efficient Heuristic Now Proven

Uniform random rotations (URRs) are a common preprocessing step in modern quantization approaches used for gradient compression, inference acceleration, KV-cache compression, model weight quantization, and approximate nearest-neighbor search in vector databases. In practice, URRs are often replaced by randomized Hadamard transforms (RHTs), which preserve orthogonality while admitting fast implementations. The remaining issue is the performance for worst-case inputs. With a URR, each coordinate is individually distributed as a shifted beta distribution, which converges to a Gaussian distribution in high dimensions. Generally, one RHT is not suitable in the worst case, as individual coordinates can be far from these distributions. We show that after composing two RHTs on any $d$-sized input vector, the marginal distribution of every fixed coordinate of the normalized rotated vector is within $O(d^{-1/2})$ of a standard Gaussian both in Kolmogorov distance and in $1$-Wasserstein distance. We then plug these bounds into the analyses of modern compression schemes, namely DRIVE and QUIC-FL, and show that two RHTs achieve performance that asymptotically matches URRs. However, we show that two RHTs may not be sufficient for Vector Quantization (VQ), which often requires weak correlation across fixed-size blocks of coordinates (as opposed to only marginal distribution convergence for single coordinates). We prove that a composition of three RHTs leads to decaying coordinate covariance. This ensures that any fixed, bounded, multi-dimensional VQ codebook optimized for URRs has the same expected error when using three RHTs, up to an additive term that vanishes with the dimension. Finally, because practical inputs are rarely adversarial, we propose a linear-time ${O}(d)$ check on the input's moments to dynamically adapt the number of RHTs used at runtime to improve performance.

cs.LG↗

arXiv2026

A Note on TurboQuant and the Earlier DRIVE/EDEN Line of Work

This note clarifies the relationship between the recent TurboQuant work and the earlier DRIVE (NeurIPS 2021) and EDEN (ICML 2022) schemes. DRIVE is a 1-bit quantizer that EDEN extended to any $b>0$ bits per coordinate; we refer to them collectively as EDEN. First, TurboQuant$_{\text{mse}}$ is a special case of EDEN obtained by fixing EDEN's scalar scale parameter to $S=1$. EDEN supports both biased and unbiased quantization, each optimized by a different $S$ (chosen via methods described in the EDEN works). The fixed choice $S=1$ used by TurboQuant is generally suboptimal, although the optimal $S$ for biased EDEN converges to $1$ as the dimension grows; accordingly TurboQuant$_{\text{mse}}$ approaches EDEN's behavior for large $d$. Second, TurboQuant$_{\text{prod}}$ combines a biased $(b-1)$-bit EDEN step with an unbiased 1-bit QJL quantization of the residual. It is suboptimal in three ways: (1) its $(b-1)$-bit step uses the suboptimal $S=1$; (2) its 1-bit unbiased residual quantization has worse MSE than (unbiased) 1-bit EDEN; (3) chaining a biased $(b-1)$-bit step with a 1-bit unbiased residual step is inferior to unbiasedly quantizing the input directly with $b$-bit EDEN. Third, some of the analysis in the TurboQuant work mirrors that of the EDEN works: both exploit the connection between random rotations and the shifted Beta distribution, use the Lloyd-Max algorithm, and note that Randomized Hadamard Transforms can replace uniform random rotations. Experiments support these claims: biased EDEN (with optimized $S$) is more accurate than TurboQuant$_{\text{mse}}$, and unbiased EDEN is markedly more accurate than TurboQuant$_{\text{prod}}$, often by more than a bit (e.g., 2-bit EDEN beats 3-bit TurboQuant$_{\text{prod}}$). We also repeat all accuracy experiments from the TurboQuant paper, showing that EDEN outperforms it in every setup we have tried.

cs.LG↗

arXiv2024

Optimal and Near-Optimal Adaptive Vector Quantization

Quantization is a fundamental optimization for many machine-learning use cases, including compressing gradients, model weights and activations, and datasets. The most accurate form of quantization is \emph{adaptive}, where the error is minimized with respect to a given input, rather than optimizing for the worst case. However, optimal adaptive quantization methods are considered infeasible in terms of both their runtime and memory requirements. We revisit the Adaptive Vector Quantization (AVQ) problem and present algorithms that find optimal solutions with asymptotically improved time and space complexity. We also present an even faster near-optimal algorithm for large inputs. Our experiments show our algorithms may open the door to using AVQ more extensively in a variety of machine learning applications.

cs.LG↗

arXiv2022

SQUID: Faster Analytics via Sampled Quantile Estimation

Streaming algorithms are fundamental in the analysis of large and online datasets. A key component of many such analytic tasks is $q$-MAX, which finds the largest $q$ values in a number stream. Modern approaches attain a constant runtime by removing small items in bulk and retaining the largest $q$ items at all times. Yet, these approaches are bottlenecked by an expensive quantile calculation. This work introduces a quantile-sampling approach called SQUID and shows its benefits in multiple analytic tasks. Using this approach, we design a novel weighted heavy hitters data structure that is faster and more accurate than the existing alternatives. We also show SQUID's practicality for improving network-assisted caching systems with a hardware-based cache prototype that uses SQUID to implement the cache policy. The challenge here is that the switch's dataplane does not allow the general computation required to implement many cache policies, while its CPU is orders of magnitude slower. We overcome this issue by passing just SQUID's samples to the CPU, thus bridging this gap. In software implementations, we show that our method is up to 6.6x faster than the state-of-the-art alternatives when using real workloads. For switch-based caching, SQUID enables a wide spectrum of data-plane-based caching policies and achieves higher hit ratios than the state-of-the-art P4LRU.

cs.DS↗

arXiv2022

FRANCIS: Fast Reaction Algorithms for Network Coordination In Switches

Optimizing the reaction to network events, which is critical in tasks such as clock synchronization, multicast, and routing, becomes increasingly challenging as networks grow larger. To improve the reaction time compared to centralized solutions, the theory community has made significant progress in the design of message-passing algorithms that leverage all nodes for distributed computation, and the advent of programmable switches makes it now possible to materialize them. We propose FRANCIS, a framework and associated libraries for running message-passing algorithms on programmable switches. It features primitives that allow easy integration of such algorithms for quickly reacting to network events while optimizing resource consumption. We use FRANCIS to implement event reaction solutions that improve clock synchronization, source-routed multicast, and routing and demonstrate up to 18x reduction in reaction time.

cs.NI↗

arXiv2021

How to send a real number using a single bit (and some shared randomness)

We consider the fundamental problem of communicating an estimate of a real number $x\in[0,1]$ using a single bit. A sender that knows $x$ chooses a value $X\in\set{0,1}$ to transmit. In turn, a receiver estimates $x$ based on the value of $X$. We consider both the biased and unbiased estimation problems and aim to minimize the cost. For the biased case, the cost is the worst-case (over the choice of $x$) expected squared error, which coincides with the variance if the algorithm is required to be unbiased. We first overview common biased and unbiased estimation approaches and prove their optimality when no shared randomness is allowed. We then show how a small amount of shared randomness, which can be as low as a single bit, reduces the cost in both cases. Specifically, we derive lower bounds on the cost attainable by any algorithm with unrestricted use of shared randomness and propose near-optimal solutions that use a small number of shared random bits. Finally, we discuss open problems and future directions.

cs.DS↗

arXiv2019

Optimal Distributed Covering Algorithms

We present a time-optimal deterministic distributed algorithm for approximating a minimum weight vertex cover in hypergraphs of rank $f$. This problem is equivalent to the Minimum Weight Set Cover problem in which the frequency of every element is bounded by $f$. The approximation factor of our algorithm is $(f+ε)$. Our algorithm runs in the CONGEST model and requires $O(\logΔ/ \log\logΔ)$ rounds, for constants $ε\in(0,1]$ and $f\in N^+$. This is the first distributed algorithm for this problem whose running time does not depend on the vertex weights nor the number of vertices. For constant values of $f$ and $ε$, our algorithm improves over the $(f+ε)$-approximation algorithm of KMW06 whose running time is $O(\log Δ+ \log W)$, where $W$ is the ratio between the largest and smallest vertex weights in the graph. Our algorithm also achieves an $f$-approximation for the problem in $O(f\log n)$ rounds, improving over the classical result of KVY94 that achieves a running time of $O(f\log^2 n)$. Finally, for weighted vertex cover ($f=2$) our algorithm achieves a \emph{deterministic} running time of $O(\log n)$, matching the \emph{randomized} previously best result of KY11. We also show that integer covering-programs can be reduced to the Minimum Weight Set Cover problem in the distributed setting. This allows us to achieve an $(f+ε)$-approximate integral solution in $O(\frac{\logΔ}{\log\logΔ}+(f\cdot\log M)^{1.01}\logε^{-1}(\logΔ)^{0.01})$ rounds, where $f$ bounds the number of variables in a constraint, $Δ$ bounds the number of constraints a variable appears in, and $M=\max \{1, 1/a_{\min}\}$, where $a_{min}$ is the smallest normalized constraint coefficient. This improves over the results of KMW06 for the integral case, which runs in $O(ε^{-4}\cdot f^4\cdot \log f\cdot\log(M\cdotΔ))$ rounds.

cs.DC↗

arXiv2018

Optimal Distributed Weighted Set Cover Approximation

cs.DC↗

arXiv2018

Parameterized Distributed Algorithms

In this work, we initiate a thorough study of parameterized graph optimization problems in the distributed setting. In a parameterized problem, an algorithm decides whether a solution of size bounded by a \emph{parameter} $k$ exists and if so, it finds one. We study fundamental problems, including Minimum Vertex Cover (MVC), Maximum Independent Set (MaxIS), Maximum Matching (MaxM), and many others, in both the LOCAL and CONGEST distributed computation models. We present lower bounds for the round complexity of solving parameterized problems in both models, together with optimal and near-optimal upper bounds. Our results extend beyond the scope of parameterized problems. We show that any LOCAL $(1+ε)$-approximation algorithm for the above problems must take $Ω(ε^{-1})$ rounds. Joined with the algorithm of [GKM17] and the $Ω(\sqrt{\frac{\log n}{\log\log n}})$ lower bound of [KMW16], this settles the complexity of $(1+ε)$-approximating MVC, MaxM and MaxIS at $(ε^{-1}\log n)^{Θ(1)}$. We also show that our parameterized approach reduces the runtime of exact and approximate CONGEST algorithms for MVC and MaxM if the optimal solution is small, without knowing its size beforehand. Finally, we propose the first deterministic $o(n^2)$ rounds CONGEST algorithms that approximate MVC and MaxM within a factor strictly smaller than $2$.

cs.DC↗

arXiv2018

A Deterministic Distributed $2$-Approximation for Weighted Vertex Cover in $O(\log n\logΔ/ \log^2\logΔ)$ Rounds

We present a deterministic distributed $2$-approximation algorithm for the Minimum Weight Vertex Cover problem in the CONGEST model whose round complexity is $O(\log n \log Δ/ \log^2 \log Δ)$. This improves over the currently best known deterministic 2-approximation implied by [KVY94]. Our solution generalizes the $(2+ε)$-approximation algorithm of [BCS17], improving the dependency on $ε^{-1}$ from linear to logarithmic. In addition, for every $ε=(\log Δ)^{-c}$, where $c\geq 1$ is a constant, our algorithm computes a $(2+ε)$-approximation in $O(\log Δ/ \log \log Δ)$~rounds (which is asymptotically optimal).

cs.DC↗

arXiv2014

Relations between automata and the simple k-path problem

Let $G$ be a directed graph on $n$ vertices. Given an integer $k<=n$, the SIMPLE $k$-PATH problem asks whether there exists a simple $k$-path in $G$. In case $G$ is weighted, the MIN-WT SIMPLE $k$-PATH problem asks for a simple $k$-path in $G$ of minimal weight. The fastest currently known deterministic algorithm for MIN-WT SIMPLE $k$-PATH by Fomin, Lokshtanov and Saurabh runs in time $O(2.851^k\cdot n^{O(1)}\cdot \log W)$ for graphs with integer weights in the range $[-W,W]$. This is also the best currently known deterministic algorithm for SIMPLE k-PATH- where the running time is the same without the $\log W$ factor. We define $L_k(n)\subseteq [n]^k$ to be the set of words of length $k$ whose symbols are all distinct. We show that an explicit construction of a non-deterministic automaton (NFA) of size $f(k)\cdot n^{O(1)}$ for $L_k(n)$ implies an algorithm of running time $O(f(k)\cdot n^{O(1)}\cdot \log W)$ for MIN-WT SIMPLE $k$-PATH when the weights are non-negative or the constructed NFA is acyclic as a directed graph. We show that the algorithm of Kneis et al. and its derandomization by Chen et al. for SIMPLE $k$-PATH can be used to construct an acylic NFA for $L_k(n)$ of size $O^*(4^{k+o(k)})$. We show, on the other hand, that any NFA for $L_k(n)$ must be size at least $2^k$. We thus propose closing this gap and determining the smallest NFA for $L_k(n)$ as an interesting open problem that might lead to faster algorithms for MIN-WT SIMPLE $k$-PATH. We use a relation between SIMPLE $k$-PATH and non-deterministic xor automata (NXA) to give another direction for a deterministic algorithm with running time $O^*(2^k)$ for SIMPLE $k$-PATH.

cs.DS↗