Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.13823

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

Oktie Hassanzadeh·Dharmashankar Subramanian
View original source ↗Read source document ↗Export metadata ↓

Abstract

Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executions at temperature 0. We measure how much another execution changes. We re-execute the text and tabular workloads of SemBench, LRO-Bench, and UDA-Bench through their own harnesses, log every one of 1.2 million model calls, and reanalyze SemBench's published executions. At temperature 0, between 0.07% and 10.2% of rows change their operator output over five executions, depending on the model, and up to 24.0% on one task. Plans amplify these changes exponentially in the number of rows an answer exposes, as a possible-worlds model predicts: on UDA-Bench, a few percent of extracted values change, but about two thirds of the answers do. In SemBench's published results, 21% of text results change quality between executions, including on engines whose plans are deterministic. Several per-query system comparisons, and 16 of 28 tests of LRO-Bench's implementation recommendations, are not settled by the executions the benchmarks run. Changing rows are rows on which models disagree with each other; identical requests within one execution disagree as often as across executions; and one execution's decision margin or reasoning length predicts which rows change. Voting reduces the variance but does not remove it, and a calibrated cascade keeps its guarantee while the rows it returns change. We derive reporting rules for benchmarks and design rules for engines.

Explore related subjects

cs.DBcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Oktie Hassanzadeh, Dharmashankar Subramanian. 2026-10-09. Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing. https://arxiv.org/abs/2610.13823

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Access Paths for Efficient Ordering with Large Language Models

In this work, we present the \texttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical implementations. First, we propose several improvements to existing semantic sorting algorithms and introduce a semantic-aware external merge sort algorithm. Our extensive evaluation reveals that no single implementation offers universal optimality on all datasets. From our evaluations, we observe a general scaling relationship between sorting cost and the ordering quality for comparison-based algorithms. Building on these insights, we design a budget-aware optimizer that utilizes heuristic rules, LLM-as-Judge evaluation, and consensus aggregation to dynamically select the near-optimal access path for LLM ORDER BY. In our extensive evaluations, our optimizer consistently achieves ranking accuracy on par with or superior to the best static methods across all benchmarks. We believe that this work provides foundational insights into the principled optimization of semantic operators essential for building robust, large-scale LLM-powered analytic systems.

cs.DB↗
arXiv2026

MetaJoin: Adaptive Join Selections for Heterogeneous Tables

Joining heterogeneous tables requires different forms of correspondence, including exact agreement, syntactic transformation, semantic similarity, and entity-level matching. The appropriate join method is often unknown in advance, while running every candidate method incurs unnecessary cost. We introduce MetaJoin, a label-free framework for adaptive join selection. MetaJoin uses input evidence to choose an initial specialist, then uses specialist-specific output evidence to determine whether that result should be retained, compared with an alternative, escalated, or repaired. This allows join selection to adapt after execution rather than committing to a one-shot routing decision. We also introduce MNN Join, a transformation specialist that searches for string transformations by maximizing mutual-nearest-neighbor coverage between two unaligned columns and uses the same reciprocal structure to validate its output. We evaluate MetaJoin on 670 datasets from six benchmarks. It achieves 0.817 macro row-F1, slightly exceeding a benchmark-level oracle that is given the best standalone specialist for each benchmark (0.812), and outperforms label-free selection policies built from the same specialist portfolio by more than 0.13. Across the 609 datasets in the four transformation-dominant benchmarks, MNN Join outperforms every evaluated transformation baseline. Finally, MetaJoin's accuracy varies by only 0.015 as the controller backbone ranges from 3B to 14B parameters, showing that output validation and repair substantially reduce dependence on the initial routing decision.

cs.DB↗
arXiv2026

The Stretto Execution Engine for Semantic Data Systems

Semantic data systems enable querying over both structured and unstructured data using LLMs, but a central challenge is navigating the performance-accuracy trade-off when selecting among alternative strategies for executing semantic operators. In this paper, we present Stretto, a new execution engine for semantic query processing that combines a novel optimizer with a rich set of physical execution strategies for semantic operators. Unlike prior work, Stretto provides global - rather than per-operator - accuracy guarantees while optimizing query performance. To this end, Stretto formulates query planning as a constrained optimization problem and employs a gradient-based optimizer to jointly select operator implementations and allocate error budgets across the full query plan. Furthermore, to enable diverse execution choices, Stretto introduces a new technique that builds on KV-cache compression to realize a spectrum of physical execution strategies for semantic operators with controllable runtime-accuracy trade-offs. Experimental results demonstrate that Stretto outperforms state-of-the-art systems while consistently satisfying quality guarantees.

cs.DB↗