Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.13704

Benchmarking Content-Defined Chunking Algorithms for Data Deduplication

Marcel Gregoriadis·Leonhard Balduf·Björn Scheuermann·Johan Pouwelse
View original source ↗Read source document ↗Export metadata ↓

Abstract

Data deduplication relies on content-defined chunking (CDC) algorithms to decide where data is split. We observe that most CDC algorithms are modular: a base chunking algorithm fixes the byte-level cut-point condition, while high-level techniques (size normalization, min/max thresholds, caching) layer on top. Existing benchmarks conflate the two, obscuring which design choices actually drive performance. We isolate and evaluate eight state-of-the-art base chunking algorithms across four realistic workloads and four metrics: throughput, deduplication ratio, mean chunk size, and chunk-size variance. Further, we derive new formulas relating algorithm parameters to expected chunk size for AE, RAM, and MII. In doing so, we uncover a longstanding error in the widely-used AE parameterization reproduced across multiple papers and evaluation tools that produces chunks 13-39% smaller than intended and inflates AE's reported deduplication. Across our evaluation, AE is the most balanced base chunking algorithm across all four metrics, RAM's similarity to AE does not carry over to low-entropy data, and MII and PCI show limited advantages over simpler alternatives in storage-oriented workloads. We release all implementations, datasets, and analysis scripts as open source.

Explore related subjects

cs.DBcs.DC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Marcel Gregoriadis, Leonhard Balduf, Björn Scheuermann, Johan Pouwelse. 2026-10-09. Benchmarking Content-Defined Chunking Algorithms for Data Deduplication. https://arxiv.org/abs/2610.13704

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Access Paths for Efficient Ordering with Large Language Models

In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.

cs.DB↗
arXiv2026

MetaJoin: Adaptive Join Selections for Heterogeneous Tables

Joining heterogeneous tables requires different forms of correspondence, including exact agreement, syntactic transformation, semantic similarity, and entity-level matching. The appropriate join method is often unknown in advance, while running every candidate method incurs unnecessary cost. We introduce MetaJoin, a label-free framework for adaptive join selection. MetaJoin uses input evidence to choose an initial specialist, then uses specialist-specific output evidence to determine whether that result should be retained, compared with an alternative, escalated, or repaired. This allows join selection to adapt after execution rather than committing to a one-shot routing decision. We also introduce MNN Join, a transformation specialist that searches for string transformations by maximizing mutual-nearest-neighbor coverage between two unaligned columns and uses the same reciprocal structure to validate its output. We evaluate MetaJoin on 670 datasets from six benchmarks. It achieves 0.817 macro row-F1, slightly exceeding a benchmark-level oracle that is given the best standalone specialist for each benchmark (0.812), and outperforms label-free selection policies built from the same specialist portfolio by more than 0.13. Across the 609 datasets in the four transformation-dominant benchmarks, MNN Join outperforms every evaluated transformation baseline. Finally, MetaJoin's accuracy varies by only 0.015 as the controller backbone ranges from 3B to 14B parameters, showing that output validation and repair substantially reduce dependence on the initial routing decision.

cs.DB↗
arXiv2026

The Stretto Execution Engine for Semantic Data Systems

Semantic data systems enable querying over both structured and unstructured data using LLMs, but a central challenge is navigating the performance-accuracy trade-off when selecting among alternative strategies for executing semantic operators. In this paper, we present Stretto, a new execution engine for semantic query processing that combines a novel optimizer with a rich set of physical execution strategies for semantic operators. Unlike prior work, Stretto provides global - rather than per-operator - accuracy guarantees while optimizing query performance. To this end, Stretto formulates query planning as a constrained optimization problem and employs a gradient-based optimizer to jointly select operator implementations and allocate error budgets across the full query plan. Furthermore, to enable diverse execution choices, Stretto introduces a new technique that builds on KV-cache compression to realize a spectrum of physical execution strategies for semantic operators with controllable runtime-accuracy trade-offs. Experimental results demonstrate that Stretto outperforms state-of-the-art systems while consistently satisfying quality guarantees.

cs.DB↗