Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents

Abstract

Automated evaluation of tool-using large language model (LLM) agents is widely assumed to be reliable, yet this assumption is rarely validated against human annotation. We present AgentProp-Bench, a diagnostic benchmark of 14,750 execution traces from thirteen LLM agents (nine proprietary, four open-weight) across four domains, and use it to audit three questions. First, substring-heuristic judging of agent outputs agrees with human annotation only at chance level (Cohen's kappa = 0.049 against each of two annotators), while a three-LLM ensemble reaches moderate agreement (kappa = 0.432) and a single GPT-4o-mini judge is in fact the strongest (kappa = 0.567); dual-annotator agreement is almost perfect (kappa = 0.835). Second, under validated judging a parameter-level error propagates to a wrong final answer with human-calibrated probability approximately 0.62, replicated across proprietary and open-weight models, and a model's ability to reject corrupted inputs and to recover from them are statistically indepen1). Third, severalagents fabricate tool executions -- asserting tool-derived renever obtained (up toilure invisible toend-to-end scores, and a lightweight runtime interceptor reduhallucination on everto 24 percentagepoints) at a tunable operating point that is net-positive on open-weight models. Ini-2.0-Flash, whichrarely emits an injectable tool call. All code, data, traces, and human labels are relehttps://github.com/bhaskargurram-ai/agenthallu-bench