Zero2Repo: Can Coding Agents Build Repositories from Scratch?

Pei Yang; Tianyu Shi; Yuhang Yao; Wanyi Chen; Tongyun Yang; Dun Pei; Haonan Wang; Pengbin Feng; Guanxu Yu; Jingchun Huang; Zeyu Zhang; Shuhan Sun; Hao Li; Alex Gu; Xiang Li; Jie Xiao; Xinyu Wang; Hanxin Chen; Daqi Li; Qi Jia; Hongshan Lin; Zhizhou Gu; Zijun Tian; Weizhi Du; Lynn Ai; Eric Yang

arXiv · 2609.38269

Zero2Repo: Can Coding Agents Build Repositories from Scratch?

View original source ↗Read source document ↗Export metadata ↓

Abstract

Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limited to a single language and depend on manually curated tasks. We introduce Zero2Repo, a benchmark in which an agent receives a product requirements document, an interface contract, and an empty workspace, and must deliver a complete repository in the project's native ecosystem. Tasks are produced by a language-agnostic authoring pipeline that converts real, version-pinned open-source projects into behavioral specifications, reproducible environments, and hidden acceptance tests. Each task is validated by execution: a reference implementation derived from the upstream project must pass, and adversarial validation must show that the tests reject incorrect implementations. Evaluation runs production coding agents in isolated containers, withholds the acceptance tests until an explicit submission, and assigns a binary reward only when every test passes, with no LLM judge. The pipeline and harness make no language-specific assumptions and apply to mainstream programming ecosystems; the current release contains Python, TypeScript, Go, and C++ tasks. Even on 11 tasks drawn from repositories that frontier models have very likely seen during training, the strongest agent solves only 10, and every failing submission passes 90-99% of the hidden tests; for the two strongest agents, 67-100% of failed tests trace to a single omission or a low-frequency rule stated in the specification rather than to a missing subsystem, so each failure is a concrete target for improvement.

Explore related subjects

cs.SE cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Pei Yang, Tianyu Shi, Yuhang Yao, Wanyi Chen, Tongyun Yang, Dun Pei, Haonan Wang, Pengbin Feng, Guanxu Yu, Jingchun Huang, Zeyu Zhang, Shuhan Sun, Hao Li, Alex Gu, Xiang Li, Jie Xiao, Xinyu Wang, Hanxin Chen, Daqi Li, Qi Jia, Hongshan Lin, Zhizhou Gu, Zijun Tian, Weizhi Du, Lynn Ai, Eric Yang. 2026-10-01. Zero2Repo: Can Coding Agents Build Repositories from Scratch?. https://arxiv.org/abs/2609.38269

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Theory building in software engineering: Operationalization

This work is part of a research project whose ultimate goal is to systematize a procedure for constructing theories in software engineering. The proposed methodology involves four phases: conceptualization, operationalization, testing, and application. In previous work, we described the conceptualization process. This paper presents a set of procedures for systematizing the operationalization phase in theory building. Specifically, it translates the concepts and propositions obtained from the previous conceptualization into constructs and empirically testable hypotheses. The operationalization phase is structured across three distinct dimensions: the practical procedural steps, their strict formal mathematical specification, and their evaluation through an illustrative empirical case study. Grounded in critical realism, we operationalize the qualitatively derived concepts and propositions into explicit constructs and logical hypotheses. It is important to emphasize that a novel strategy has been designed that reduces the number of hypotheses resulting from this process. In the use case described, more than 30% of the initial hypotheses have been eliminated because they are redundant. This paper is a pioneering contribution in offering comprehensive guidelines for theory operationalization using logical implication and describes a strategy based on graph theory that tackles the problem of hypothesis explosion and maintains the principle of parsimony.

cs.SE↗

arXiv2026

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Large language models (LLMs) can translate natural-language problem descriptions into optimization code, but the code is prone to silent failures: it executes and returns a solver-feasible solution while encoding a semantically incorrect formulation. On compositional problems, the resulting feasibility-correctness gap reaches 90 percentage points. We introduce ReLoop, which combines two mechanisms. Structured generation decomposes code production into a four-stage reasoning chain (understand, formalize, synthesize, verify) to reduce formulation errors during generation. Behavioral verification detects the errors that remain by testing whether the formulation responds correctly to solver-based parameter perturbation, a signal that comes from the solver rather than from LLM self-review and requires no ground truth. The two mechanisms address different error structures: structured generation gives the largest gain on compositional problems (+8.5pp accuracy on RetailOpt-190 with Claude Opus 4.6), and behavioral verification gives its largest gain on localized defects (+4.4pp on MAMO-ComplexLP). With diagnostic execution recovery, ReLoop reaches 100% executable code on Claude Opus 4.6, and relative to direct generation it raises or preserves every reported metric of the three chat-tuned foundation models on all three benchmarks. For the narrowly fine-tuned SFT model we test, the chain-of-thought prompt conflicts with its learned output format and lowers its accuracy on MAMO-ComplexLP; we document and analyze this interaction. We release RetailOpt-190, 190 compositional retail optimization scenarios in which several constraints interact.

cs.SE↗

arXiv2026

Finding Memory Leaks in C/C++ Programs via Neuro-Symbolic Augmented Static Analysis

Memory leaks remain prevalent in real-world C/C++ software. Static analyzers such as CodeQL provide scalable program analysis but frequently miss such bugs because they cannot recognize project-specific custom memory-management functions and lack path-sensitive control-flow modeling. We present MemHint, a neuro-symbolic pipeline that addresses both limitations by combining LLMs' semantic understanding of code with Z3-based symbolic reasoning. MemHint parses the target codebase and applies an LLM to classify each function as a memory allocator, deallocator, or neither, producing function summaries that record which argument or return value carries memory ownership, extending the analyzer's built-in knowledge beyond standard primitives such as malloc and free. A Z3-based validation step checks each summary against the function's control-flow graph, discarding those whose claimed memory operation is unreachable on any feasible path. The validated summaries are injected into CodeQL and Infer via their respective extension mechanisms. Z3 path feasibility filtering then eliminates warnings on infeasible paths, and a final LLM-based validation step confirms whether each remaining warning is a genuine bug. On eight real-world C/C++ projects totaling over 3.6M lines of code, MemHint detects 54 unique memory leaks, all confirmed or fixed, at approximately $1.7 per detected bug, compared to 19 by vanilla CodeQL and 3 by vanilla Infer.

cs.SE↗