arXiv subjects

Hela Momand

Publications and source records attributed to Hela Momand.

2 recordsLinked to original sources

arXiv2026

AI Research Preference Models

AI research agents (AIRA) can now carry machine learning experiments from proposal through implementation and evaluation. Yet progress on frontier tasks is throttled by the cost of evaluations that can consume days of GPU time. When an agent can propose far more candidates than it can afford to run, progress depends on its research preference: how it allocates a fixed execution budget across many candidates. We introduce AI Research Preference Models (RPMs) that predict which candidate solution is most promising, without paying the cost of running them all. We build RPMs from frozen pretrained language models in two variants: an inference-only model that reasons over candidate plans, code, and previously executed solutions, and an agentic model that additionally runs small-scale pilot experiments. Integrated into the AIRA-dojo research agent and evaluated on the machine learning research benchmark AIRS-Bench, the two variants increase the average normalized score from 0.684 to 0.711 and 0.729, respectively. Both reach the unguided agent's 24-hour performance in roughly 15 hours, using less than two-thirds of its execution budget, and together yield new state-of-the-art results on two AIRS-Bench tasks.

cs.AI↗

arXiv2026

AIRA_2: Overcoming Bottlenecks in AI Research Agents

Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA$_2$, which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA$^{\dagger}_{2}$ achieves a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, outperforming the strongest baseline, which achieves 72.7%. On AIRS-Bench, AIRA$_2$ exceeds human state-of-the-art on 6 out of 20 diverse research tasks. Ablations confirm that each architectural component is necessary, that performance follows a predictable scaling law that transfers across LLM backbones, and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.

cs.AI↗