Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.19596

JevCoder: When Does Decision Delegation Pay Off in Code Repair?

Jiahao Zhang·Yifan Zhang·Yu Huang
View original source ↗Read source document ↗Export metadata ↓

Abstract

A coding agent repeatedly makes decisions as it repairs code. These decisions shape repair trajectories and affect the token cost of successive model requests. We ask which repair decisions can be delegated to a low-cost probabilistic judge to reduce token use while maintaining repair quality. The judges answer questions with probabilities over predefined answers, without generating explanations. We evaluate judgment quality across repair stages and test how individual interventions change subsequent repair work. We compare these interventions with unguided repair, which uses no judge, and with the same workflow using the repair model itself as the judge. The benefit depends on the work an intervention changes, rather than the cost of the judgment alone. Based on these studies, we propose JevCoder, a workflow built around Jev, a low-cost probabilistic judge. It combines code-snippet recommendations with action-type guidance and uses judgments to determine when to submit. We evaluate three repair models, each on the same 400 tasks: 200 from SWE-bench Verified and 200 from SWE-Bench Pro Verified. Compared to unguided repair, JevCoder reduces mean repair-model token use by 21.6%-50.6%, with higher observed repair rates across all three models.

Explore related subjects

cs.SEcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Jiahao Zhang, Yifan Zhang, Yu Huang. 2026-10-14. JevCoder: When Does Decision Delegation Pay Off in Code Repair?. https://arxiv.org/abs/2610.19596

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths

Large Language Models (LLMs) are central to automated code generation, yet most approaches use one model at every stage, from writing code to repairing and optimizing it. LLMs have complementary strengths: their failures do not fully overlap across programming languages, algorithmic categories, and development stages. For example, in Python, the model with the highest code-generation accuracy ranks last at code repair. Using one model to generate code, another to repair it, and a third to refine it can therefore achieve higher correctness and better execution performance than a single model throughout. We propose PerfOrch, a multi-LLM multi-agent system with four agents for categorization, generation, debugging, and refinement. The generation, debugging, and refinement agents select models through a Memory module that ranks candidate LLMs by their measured performance for each stage, programming language, and algorithmic category. We evaluate PerfOrch with five candidate LLMs on 4,640 problems from four benchmarks covering five programming languages (HumanEval-X, EffiBench-X, BigCodeBench, and APPS) against single-model direct generation, single-model generate-debug-refine pipelines (PerfCodeGen), and the state-of-the-art multi-agent systems MapCoder and CodeSIM. PerfOrch achieves the highest pass@1 in all twelve benchmark-language settings, exceeding the strongest baseline by an average of 8.19 percentage points, and reduces execution time for an average of 75.03% of solved problems with an average speedup of 1.18x. The Memory rankings remain effective on three unseen benchmarks without re-profiling. Sequential refinement matches exhaustive execution-time refinement coverage with 66% fewer refinement-stage tokens (57% for average-memory optimization). The complementary strengths persist after all five LLMs are replaced with their current-generation successors.

cs.SE↗
arXiv2026

Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibration

Code from LLM-assisted development may contain defects like API hallucinations, motivating the development of code evaluation metrics to assess such code before deployment. As conventional rule-based metrics merely score code based on similarity to reference programs, and execution-based metrics require expensive manual preparation for tests, researchers have developed LLM-as-a-judge metrics for both behavioral and nonbehavioral code evaluation, and curated multiple benchmarks to validate their effectiveness. However, such benchmarks suffer from critical limitations: Some feature coarse-grained binary labels of passing/failing unit tests, unable to distinguish between fundamental and minor flaws; others utilize fine-grained criteria with manually annotated scores as ground-truth, but such criteria are often vaguely defined, introducing subjectivity and unreliability in human scores. Furthermore, they often directly prompt LLMs to generate code for evaluation, which may lead to unbalanced code quality distributions, failing to test the metrics' discriminative capability across varying quality levels. To curate a diverse benchmark with a balanced program quality distribution and streamline manual annotation, we propose AXIOM, a novel perturbation-based framework. It reframes program scores as the refinement effort needed for deployment, with two stages: (1) Quality-constrained semantic perturbation, prompting LLMs to apply sequences of predefined perturbation rules to existing programs, precisely controlling each program's target quality to achieve a balanced distribution. (2) Hybrid quality calibration, selecting semantically distinct programs for benchmark diversity, and performing streamlined manual annotation with multisource context. With AXIOM, we curate AXIOMBENCH, a large-scale multilingual benchmark with 1,962 code evaluation problems in a more balanced score distribution...

cs.SE↗
arXiv2026

LLMON: An LLM-native Markup Language to Leverage Structure and Semantics at the LLM Interface

Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often has a richer structure and semantics, which is not conveyed in a string. For example, most prompts contain both instructions ("Summarize this paper into a paragraph") and data (the paper to summarize), but these are usually not distinguished when passed to the model. This can lead to model confusion and security risks, such as prompt injection attacks. This work addresses this shortcoming by introducing an LLM-native mark-up language, LLMON (LLM Object Notation, pronounced "Lemon"), that enables the structure and semantic metadata of the text to be communicated in a natural way to an LLM. This information can then be used during model training, model prompting, and inference implementation, leading to improvements in model accuracy, safety, and security. This is analogous to how programming language types can be used for many purposes, such as static checking, code generation, dynamic checking, and IDE highlighting. We discuss the general design requirements of an LLM-native markup language, introduce the LLMON markup language and show how it meets these design requirements, describe how the information contained in a LLMON artifact can benefit model training and inference implementation, and provide some preliminary empirical evidence of its value for both of these use cases. We also discuss broader issues and research opportunities that are enabled with an LLM-native approach.

cs.SE↗