Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2608.28232

HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software

Aadarsh Singh·Sudhir K. Vempati
View original source ↗Read source document ↗Export metadata ↓

Abstract

Scientists increasingly want to drive research software by natural-language request, but fluent output helps only if it becomes a correct machine-readable artifact. We introduce HEPToolBench, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension. We evaluate 42 deployments, from small locally served open-weight models to hosted frontier systems. The central experiment compares direct generation of native HEP-tool syntax with a schema-mediated interface where the model returns a typed representation that deterministic software serializes. Across five matched requests, the mean score rises from 0.418 to 0.902 and task passes from 21/210 to 159/210, with 41 of 42 deployments improving; eleven deployments, seven locally served open-weight models, go from 0/5 passes under native syntax to 5/5 under the structured interface. Values use the corrected v1.2.1 contract, in which the two native scorers that had enforced operational conventions absent from their own prompts are rescored prompt-faithfully cohort-wide. One asymmetry remains: native scorers score the recorded response verbatim and reject Markdown-fenced replies, whereas structured scorers recover JSON from such wrappers; no full-cohort rescoring under a single symmetric extraction rule has been done. An audit of archived responses from 17 locally served deployments shows this does not explain their gains: passes remain six to ten times more frequent under the structured interface when both sides use the same extraction rule. A task pass alone does not guarantee runtime or scientific viability. Within this scope, moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models. Prompts, scorers, responses, and regeneration scripts are released for independent evaluation and extension.

Explore related subjects

hep-ph

Keep this discovery

BibTeXRIS

Aadarsh Singh, Sudhir K. Vempati. 2026-08-28. HEPToolBench 1.2: Testing How Reliably Language Models Can Drive Particle Physics Software. https://arxiv.org/abs/2608.28232

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Axionic Wormholes in Metric-Affine Gravity

The axion is a promising candidate for solving the strong CP problem. To solve this problem, the global U(1) symmetry must be preserved to a high degree of accuracy. However, it is well known that global symmetries are explicitly violated by quantum gravity effects, giving rise to what is referred to as the axion quality problem. In this paper, we investigate axionic wormholes as a source of explicit U(1) violation in Metric-Affine Gravity. This framework allows for spacetime torsion and non-metricity, which accommodate additional curvature-like and topological terms, such as the Holst and Nieh--Yan terms, that are absent from the metric and Palatini formalisms. We show that non-minimal couplings to these terms modify the wormhole dynamics and enhance the Euclidean wormhole action, thereby alleviating the axion quality problem. We also find that the viable parameter space is enlarged when two of these couplings are simultaneously present. We further identify representative parameter regions where the alleviation of the axion quality problem is compatible with inflationary constraints.

hep-ph↗
arXiv2026

Qubit-Qutrit Quantum Tomography of hadronic $\Lambda\phi$ and $\Lambda K^{\ast 0}$ systems

Quantum-information observables have emerged in recent years as new tools in nuclear and particle physics, from entanglement in top-quark pairs to spin correlations in $\Lambda\bar{\Lambda}$ production. Extending these studies to unequal-spin hadronic final states poses a fundamental challenge: the $6\times6$ density matrix of a qubit-qutrit system contains 35 independent spin parameters, but the decays of $\Lambda V$ pairs, with $V=\phi$ or $K^{*0}$, provide access to only 23 due to the hidden vector polarization from the strong decay. In this Letter, we formulate a qubit-qutrit quantum tomography (QQQT) technique for these spin-$\tfrac{1}{2}\otimes1$ systems and establish exact criteria for entanglement certification from the \textit{incomplete} density matrix. Compared with the $\Lambda\bar{\Lambda}$ system, QQQT of $\Lambda\phi$ and $\Lambda K^{*0}$ provides a new probe of nonperturbative QCD hadronization, enabling a direct comparison of the spin evolution of entangled quark pairs produced from the vacuum as they hadronize into a baryon or a vector meson.

hep-ph↗
arXiv2026

Twist decomposition of exclusive heavy meson production cross sections

We study the twist decomposition of the total cross sections for exclusive heavy vector meson electroproduction and photoproduction in the $\gamma^\ast p$ processes, within the leading logarithmic $1/x$ BFKL formalism. The Mellin transforms of the impact factors of the vector meson are calculated. We show that the higher twist contributions are strongly suppressed in the low-$x$ kinematical regime. Possible enhancement of the higher twists effects for nuclei targets is discussed.

hep-ph↗