Searcharxiv⌕ Search

arXiv subjects

Mahi Begoug

Publications and source records attributed to Mahi Begoug.

3 recordsLinked to original sources

Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict

Studies of how long code from AI coding agents survives after it is merged reach conflicting verdicts: agent code is reported to be retained longer than human code, about as long, or modified sooner. Each study makes defensible measurement choices, but how much these choices decide the verdict is unknown. We revisit the survival of agent-attributed code in our earlier study: 854,139 lines that 2,802 merged pull requests (PRs) added to 215 repositories of the AIDev dataset. We vary nine measurement choices in five research questions and build a specification curve of 88 survival analyses, tested under three variance estimators, several observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, the choices change the size of the effect but not its direction. For code from the fully-autonomous agent Devin, they decide the verdict. Across repositories, 15 of 22 specifications report it modified sooner when lines are treated as independent, but only 3 when standard errors are clustered by repository, no more than permuted labels produce. Within repositories, no specification detects a difference. LLM intent classifiers agree with human labels about as well as two humans agree with each other, whereas keyword rules do not (kappa = 0.69 to 0.72 against 0.38), and the two reach different verdicts on why agent code is modified. Humans make most modifications to agent code under every unit and window. Re-run on our data, five published designs each change their result with a single choice. Whether agent code survives longer than human code is therefore decided as much by the measurement as by the code. We derive an operationalization checklist for post-merge studies.

cs.SE↗

Will It Survive? Deciphering the Fate of AI Agent-Authored Code in Open Source

AI coding agents such as GitHub Copilot, OpenAI Codex and Devin now open pull requests (PRs) that are merged into open-source projects. Industry reports suggest that AI-assisted code is revised or reverted soon after it is written, which raises the concern that agent code is "disposable". However, little is known about what happens to agent code after it is merged. Knowing how long this code survives, and why it is changed, can help maintainers decide how much review it needs. We performed a survival analysis of the 648,959 lines that 2,794 merged PRs added outside vendored and generated paths in 215 open-source repositories from the AIDev dataset, comparing agent-attributed and human code within the same repositories. Agent-attributed code is not modified sooner than human code. When each PR is weighted equally, agent-attributed lines are modified later (HR = 0.83, 95% CI [0.75, 0.93]). This difference comes from the supervised assistants, mainly Copilot and Codex (HR = 0.76). Code from Devin does not differ from human code over the whole period (HR = 0.95), but among PRs merged after mid-May 2025, code from both the supervised assistants and Devin is modified later. Without PR weighting, the difference is not statistically significant. When agent-attributed code is modified, the intent of the change is similar to that of human code (Cramer's V = 0.080). A file's size and recent changes predict whether agent-attributed code will be modified better than its code tokens do (AUC-ROC 0.78 vs. 0.70), and process features predict when it will be modified only weakly (Macro F1 of 0.31 against 0.25 for a random guess). Our findings suggest that agent code is not disposable, but whether it survives longer than human code depends on the agent, on when the code was merged and on how the comparison is computed.

cs.SE↗

SIEVE: A Parameterized GitHub Corpus Builder for Software Engineering Research

Software engineering research increasingly relies on large, richly annotated corpora of real-world code. Still, researchers typically assemble these with ad-hoc scripts or static, pre-built datasets that offer no control over corpus composition. This leaves a gap for a reusable tool that goes below the repository level and builds annotated, function- and class-level code corpora under researcher-specified criteria. We present SIEVE (Software Ingestion & Extraction for Verifiable Evaluation), an open-source, parameterized GitHub corpus builder that, unlike repository-level mining tools, works at code granularity: it locates individual functions, methods, and classes in repositories and extracts them as records, giving direct control over corpus construction based on several criteria, including widely used programming languages (Python, Java, JavaScript, C++), repository quality, and creation date. To support software engineering research and LLM evaluation on real-world code, SIEVE extracts CodeSearchNet- and OpenClassGen-compatible records, each annotated with 22 structural metrics and AST-derived features, and optionally with a CodeBERT-based P(AI-generated) score (F1=0.9478, AUROC=0.9902), alongside the third-party dependencies declared in each repository's package manifest. These annotations enable researchers to conduct structural analyses, investigate code dependencies, assess AI-generated code, and construct customized benchmarks for evaluating LLMs, including under temporal constraints. A manual validation of 400 records in four languages and two granularities shows 100% accuracy in code metrics and import detection, 99.0% in source completeness, and 99.5% in metadata accuracy. SIEVE is publicly available with a live demo at https://mrahman2025-sieve.hf.space.

cs.SE↗