arXiv subjects

Sam Miller

Publications and source records attributed to Sam Miller.

3 recordsLinked to original sources

arXiv2026

WorkBench Revisited: Workplace Agents Two Years On

The best agent on WorkBench in March 2024, GPT-4, completed just 43% of tasks. We revisit the benchmark in June 2026 and find that the best agent to date, Claude Fable 5, now completes 98%. Beyond this considerable progress in frontier agent performance, three things stand out. First, unintended harmful actions, such as emailing the wrong person, fell from 26% of tasks for GPT-4 to 1.9% for Claude Fable 5; capability and safety go together on WorkBench rather than trade off, so the models that finish the most tasks also do the least unintended damage. Second, the rise of open-weight models has drastically lowered costs for a performance level that was only accessible to proprietary models, while frontier costs have stayed stable. Third, while several classes of error have been eliminated, frontier models still make some basic mistakes that occasionally result in irreversible harm. We release an updated version of the benchmark with data and code quality improvements, new model scores, and analysis of agent progress on WorkBench since 2024.

cs.AI↗

arXiv2025

Improved high-gradient performance for medium-velocity superconducting half-wave resonators: Surface preparation and trapped flux mitigation

A development effort to improve the performance of superconducting radio-frequency half-wave resonators (SRF HWRs) is underway at the Facility for Rare Isotope Beams (FRIB), where 220 such resonators are in operation. Our goal was to achieve an intrinsic quality factor (Q0) of >= 2E10 at an accelerating gradient (Ea) of 12 MV/m. FRIB production resonators were prepared with buffered chemical polishing (BCP). First trials on electropolishing (EP) and post-EP low temperature baking (LTB) of FRIB HWRs allowed us to reach higher gradient (15 MV/m, limited by quench) with a higher quality factor at high gradient, but Q0 was still below our goal. Trapped magnetic flux during the Dewar test was found to be a source of Q0 reduction. Three strategies were used to reduce the trapped flux: (i) adding a local magnetic shield (LMGS) to supplement the ``global'' magnetic shield around the Dewar for reduction of the ambient magnetic field; (ii) performing a ``uniform cool-down'' (UC) to reduce the thermoelectric currents; and (iii) using a compensation coil to further reduce the ambient field with active field cancellation (AFC). The LMGS improved the Q0, but not enough to reach our goal. With UC and AFC, we exceeded our goal, reaching Q0 = 2.8E10 at Ea = 12 MV/m.

physics.acc-ph↗

arXiv2024

WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting

We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

cs.CL↗