Searcharxiv⌕ Search

arXiv subjects

Inase Kondo

Publications and source records attributed to Inase Kondo.

3 recordsLinked to original sources

SZZBench: Controlled Git Histories with Explicit Oracles for Diagnosing SZZ Implementations

The SZZ algorithm traces a bug-fixing commit back to the commits that introduced the bug. Its labels underlie defect-prediction data sets and many mining software repositories studies. SZZ implementations are mostly evaluated against oracles mined from real projects. Such oracles are realistic, but each case mixes the history factors that make an implementation succeed or fail, and an aggregate score cannot show why. We present SZZBench, a tool that shows why: it generates ordinary Git repositories from declarative scenarios with explicit oracles, scores the predictions of any SZZ implementation under declared evaluation policies, and reads each result against 26 naive baselines, such as always answering the second commit. SZZBench also checks the declarations of new scenarios, such as the tier, against the results of the naive baselines. Currently, SZZBench ships 279 scenarios in 17 categories and adapters for four SZZ tools. Running these tools in nine configurations without changing their code, we find that PyDriller passes 161 scenarios and a rule that always answers the second commit passes 160, yet only 97 scenarios are passed by both: the two succeed on different scenarios, which the number of passed scenarios hides and the per-scenario results reveal. SZZBench is available at https://github.com/higolab/SZZBench.

cs.SE↗

Rethinking SZZ Search Strategies: Lessons from Claude Code

Identifying bug-inducing commits (BICs) is important for studying software bugs and maintaining software, yet the accuracy of existing SZZ methods remains limited. Although various improvements have been proposed, many SZZ methods still rely on code tracing, which searches for BICs by tracing code changed in a bug-fixing commit (BFC) back to earlier commits. However, this approach may miss the actual BIC when the code responsible for introducing a bug differs from the code modified to fix it. To overcome this limitation, we investigate whether BIC identification can be improved by exploring search strategies that are not constrained to start with code tracing. We use Claude Code, a coding agent that has demonstrated high BIC identification accuracy in prior research, to explore such strategies. Rather than prescribing a search process or providing predefined candidate commits, we allow Claude Code to independently investigate a Git repository. We evaluate its identification accuracy and search behavior on 79 Java cases from 70 repositories. In 79.7% of runs, Claude Code reviewed commit histories before inspecting individual code diffs. It used commit messages to select changes for closer inspection. With this approach, Claude Code achieves an F1 score of 0.771, significantly outperforming all ten evaluated SZZ baselines. Our failure analysis also reveals incorrect oracle labels and ambiguities in BIC definitions in the studied evaluation dataset. After reconstructing the evaluation dataset to address these issues, Claude Code achieves an F1 score of 0.938. A key lesson from Claude Code for improving SZZ search strategies is to use semantic information to find relevant commits and then examine their code changes, rather than starting with code tracing.

cs.SE↗

Why Are Agentic Pull Requests Merged or Rejected? An Empirical Study

AI coding agents increasingly submit pull requests (Agentic-PRs) to open-source repositories, yet their performance is commonly assessed using merge and rejection outcomes alone. We hypothesized that these outcome labels do not reliably reflect agent capability without considering review interactions. To test this, we conducted a decision-oriented analysis of 11,048 closed Agentic Pull Requests, refined to 9,799 human-reviewed PRs, and manually inspected 717 representative cases to recover decision rationale from interaction artifacts. We found that rejection outcomes substantially overstate agent error: only 35.7% of rejected PRs reflected clear agentic failures, while 31.2% were driven by workflow constraints and 33.1% lacked observable decision rationale. Among merged PRs, 15.4% required explicit reviewer involvement through feedback or direct commits, and 5.5% showed no visible interaction trace. We further observed systematic differences across agents, with Copilot and Devin more often embedded in reviewer-mediated workflows, while Codex and Cursor PRs were typically merged with minimal interaction. These results reject the assumption that PR outcomes alone capture agent performance and demonstrate the need for interaction-aware evaluation grounded in review behavior.

cs.SE↗