Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons

Motti Goldberger; Nils Rudi

Home/Archive/arXiv

arXiv · 2610.26380

Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons

Motti Goldberger·Nils Rudi

View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper considers a decision-maker tasked with identifying the $k$ best items from a large pool of candidates, such as job applicants, LLMs, award nominees, or new product ideas. The decision-maker learns about the items' quality by soliciting human evaluations. Each evaluation can be costly, so the decision-maker aims to use the limited evaluation budget efficiently by allocating evaluations adaptively. A key consideration is the form of evaluation to solicit, which should be something evaluators are comfortable providing. Two popular forms are ratings and pairwise comparisons, and we allow for both, as they may be especially useful in combination given the limited budget. Ratings can be used early on to get a coarse picture of which items are good and which are not. As that picture forms, comparisons can refine the ordering of the good items, narrowing in on the ultimate top-$k$ selection. To solve the allocation problem, we develop an adaptive algorithm built from two allocation policies. The first, our primary contribution, is based on a surrogate objective that is naturally forward-looking, valuing evaluations that help inform what to evaluate in future rounds, not just those that directly improve the selection decision. The decision rule is interpretable, and we prove that it is asymptotically consistent. Late in the evaluation process, when being forward-looking is less productive, we construct a myopic policy and a criterion for switching to it. In simulation, the method gives insight into how to combine these two types of evaluations, and beats the (limited) existing benchmarks.

Explore related subjects

stat.ML cs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Motti Goldberger, Nils Rudi. 2026-10-20. Selecting the Best Few with Human Evaluators: Adaptively Combining Ratings and Pairwise Comparisons. https://arxiv.org/abs/2610.26380

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Entropic perspective on AI resilience, data-minimal learning and optimal control

Behind the general terms like adversarial attacks and generative adversarial networks (GANs), such criteria as AI resilience and AI robustness gained a crucial importance, in trying to make the emergent AI tools reliable, energy-efficient and safe. From mathematical perspective, these problems boil down to finding a mathematically-sound quantification of the smallest-sufficient amounts of the perturbations of function arguments that lead to the largest possible perturbations of the approximated function values. Here, the entropy-optimising perspective on adversarial algorithms from Artificial Intelligence (AI) is proposed to attack this problem. It is shown that adopting this perspective helps proving computational conditions for the global optimality and uniqueness of adversarial attacks based on cheaply-verifiable mathematical criteria. Further, it is shown how this perspective can be used for developing self-attacking learning algorithms, that generate optimal new data points for training, replacing one of the trained agents with this mathematical criterium. On a broad selection of various synthetic and real-life problems from hydrodynamics and biomedicine, it is shown that such self-attacking learning algorithms allow training orders-of-magnitude simpler and cheaper models with superior performance - and can be used to directly train the optimal controls in complex systems, requiring only a small fraction of the training data and outperforming the as-comprehensive-as-the-author-was-able-to-find set of contemporary AI tools (like boosted Random Forests, deep neural networks, foundational models based on transformer architectures, etc.) in complexity (measured as the model descriptor length) and predictive performance.}

stat.ML↗

arXiv2026

Help Me Help You: The Aggregate Value of Source and Target Data in Transfer Learning

Transfer learning, in regression or classification, concerns problems where one aims to use imperfect data from a source distribution P to improve prediction w.r.t. a target distribution Q. While there is now much theoretical understanding of the unsupervised setting of this problem (where no labeled target data is available), the supervised setting with labeled target data remains under-studied. We identify and characterize an interesting dichotomy in the rates achievable in the supervised setting: there is a weak transfer regime where optimal rates are determined by information in either the source or target dataset but not both, and a strong transfer regime where much faster target rates are achievable, accounting for complementary information in source and target data. Furthermore, while previous theoretical tools for the unsupervised setting can be extended to capture the weak transfer regime, we argue that they cannot capture the strong regime due to limitations inherent in the various notions of relatedness they rely on. Instead, we provide an appropriate refinement of relatedness notions---termed weak and strong moduli of transfer---that reveal the two regimes and lead to a nontrivial characterization of which pairs of distributions P and Q adhere to which regime. Finally, the above dichotomy reveals a new categorization of existing algorithmic approaches in terms of their optimality or lack-thereof across supervised regimes, and we provide a generic approach that remains nearly optimal in all regimes without prior distributional knowledge.

stat.ML↗

arXiv2026

Bayesian Nonparametric Dynamical Clustering of Time Series

We present a method that models the evolution of an unbounded number of time series clusters by switching among an unknown number of regimes with linear dynamics. We develop a Bayesian non-parametric approach using a hierarchical Dirichlet process as a prior on the parameters of a Switching Linear Dynamical System and a Gaussian process prior to model the statistical variations in amplitude and temporal alignment within each cluster. By modeling the evolution of time series patterns, the method avoids unnecessary proliferation of clusters in a principled manner. We perform inference by formulating a variational lower bound for off-line and on-line scenarios, enabling efficient learning through optimization. We illustrate the versatility and effectiveness of the approach through several case studies of electrocardiogram analysis using publicly available databases.

stat.ML↗