arXiv subjects

Brian Bushnell

Publications and source records attributed to Brian Bushnell.

3 recordsLinked to original sources

arXiv2026

Arithmetic Variable LogLog: Advancing the Memory-Variance Frontier

Cardinality estimation - counting the number of distinct elements in a data stream - requires a tradeoff between memory and accuracy. ExaLogLog recently established the state of the art for this tradeoff by combining wide registers with a Fisher-information-optimal maximum likelihood (ML) estimator, achieving the best known memory-variance product (MVP) among HyperLogLog variants. Here we present Arithmetic Variable LogLog (AVLL), which surpasses ExaLogLog at every memory point tested using arithmetic encoding and eliminating uncommon states to consume 64-bit words completely with 11 registers each, yielding a 5.5x register-count advantage. Its four-component blended estimator, HLDLC, exploits this density advantage to surpass ExaLogLog's ML accuracy without iterative solving. At 1 KB, AVLL achieves 1.63% width-weighted mean absolute error compared to ExaLogLog's 1.71% - a 4.7% improvement. The corresponding empirical MVP is 3.4, surpassing ExaLogLog's practical MVP of 3.78 and its theoretical optimum of 3.67. This holds at every tested size from 0.25 to 4 KB. AVLL inherits DynamicLogLog's early exit mechanism, which filters most elements before any register is touched. With thousands of simultaneous sketches per thread, AVLL is 2.7-4.5x faster than ExaLogLog due to the reduced memory bandwidth from early exits. Like DynamicLogLog, AVLL stores relative NLZ values with a shared offset, so its memory scales as O(B + log log C) rather than O(B x log log C) - decoupling maximum representable cardinality from register width. These results hold under both high-complexity (all-unique) and low-complexity (nonuniformly high duplication rate) data distributions, with zero accuracy degradation from duplication. AVLL is implemented as a single self-contained Java class with all correction formulas embedded, available in the BBTools suite at https://bbmap.org.

cs.DS↗

arXiv2026

IndelFreeAligner: A Streaming Aligner for Comprehensive Gapless Alignment Against Terabase-Scale References

The comparison of short sequences to massive reference databases is a cornerstone of modern genomics, but it presents a significant scalability challenge. Traditional alignment tools rely on time- and memory-intensive pre-indexing of the reference, creating a substantial bottleneck for applications involving small query sets against terabase-scale data, such as CRISPR spacer analysis. Here we present IndelFreeAligner, a streaming, indel-free alignment tool that eliminates the preprocessing bottleneck. It operates in two modes: an indexed mode for larger query sets and a brute-force mode for maximum speed on small query sets. By processing reference sequences on-the-fly, IndelFreeAligner supports user-specified mismatch thresholds up to the full query length and maintains memory usage independent of total reference size. A novel MinHitsCalculator component uses Monte Carlo simulation to determine adaptive seed-hit thresholds for indexed mode. Benchmarks against Bowtie1 and BLAST+ demonstrate that IndelFreeAligner aligns a single query against a 4 Gbp reference in 1.7 seconds versus 17 minutes for Bowtie1 (including index construction at optimal thread count), a 607-fold speedup. Against RefSeq Bacteria (560 GB compressed), IndelFreeAligner completes a 10-query search in 12 minutes using 8 GB of RAM, while BLAST+ requires 3 hours and 17 minutes to build its database alone and 506 GB of RAM to query it. Indexed mode achieves 0% false negatives through 4 substitutions (99.84-99.90% mapped at 8-16); brute-force mode is exhaustive by design. IndelFreeAligner provides a scalable and efficient solution for alignment tasks that were previously computationally prohibitive. It is distributed open-source as part of BBTools (Bushnell, 2014).

q-bio.GN↗

arXiv2026

DynamicLogLog: Faster, Smaller, and More Accurate Cardinality Estimation

Cardinality estimation - calculating the number of distinct elements in a stream - is a longstanding problem with applications from networking to bioinformatics. HyperLogLog (HLL), the prevailing standard, has a well-known error spike in its transition region and requires 6 bits per bucket, with data structure size scaling as B*log(log(cardinality)). We present DynamicLogLog (DLL), which uses a shared exponent across all buckets, storing only relative leading-zero counts. This yields three benefits: (1) only 4 bits per bucket (33% memory reduction), (2) an early exit mask that rejects >99.9% of elements at high cardinality before any bucket access (over 10x faster than HLL when bandwidth-constrained), and (3) a flat error profile via Dynamic Linear Counting (DLC) and a Logarithmic Hybrid Blend that eliminates HLL's transition artifact. Squaring the maximum representable cardinality requires only a single additional bit of global state. At 2,048 buckets with 512k simulations, DLL4's hybrid estimate achieves 1.830% mean and 1.834% peak absolute error using 1,024 bytes, compared to 1.84% mean and 34.1% peak for HLL using 1,536 bytes. DLC achieves 1.90% mean without correction factors. DynamicUltraLogLog (UDLL6), a fusion of DLL and UltraLogLog, achieves ULL-level accuracy at 75% of the memory. History-corrected variants (Hybrid+n) and Layered DLC (LDLC) provide further improvements using per-state correction tables and anti-phase error cancellation.

cs.DS↗