arXiv subjects

Zhaonan Wu

Publications and source records attributed to Zhaonan Wu.

3 recordsLinked to original sources

arXiv2026

Agent-Based Software Artifact Evaluation

Artifact Evaluation (AE) has become a standard mechanism for linking software engineering papers to their supporting artifacts, but growing submission volume makes purely manual AE increasingly difficult to sustain. Although recent advances in LLM-based agents make AE automation increasingly plausible, current descriptive badge policies define badge semantics but provide no executable verification criteria, leaving both human reviewers and agents without a detailed basis for judgment. To address this gap, we construct ArtifactGuide, a structured scoring rubric grounded in ACM policy, expert-informed calibration, and artifact-based validation, and we design ArtifactCopilot, an agent collecting review evidence following a fixed sequence under ArtifactGuide and deriving the final badge decision from accumulated evidence. We evaluate our framework on 60 real artifacts from recent software engineering conferences using human-adjudicated badges as reference. The results show that ArtifactGuide improves the AE performance of coding agents over official ACM badge-policy prompts, increasing three-run mean exact badge agreement by 10.55 to 28.34 percentage points. Across all evaluated systems and prompting protocols, ArtifactCopilot achieves the highest badge-level agreement at 70.56% and is the only system that completed all repeated runs successfully while producing a review report in every run. A controlled user study with 8 experienced researchers suggests that ArtifactCopilot reports improve reviewer confidence, help reviewers locate evidence, and understand evaluation scope more clearly. Further analysis translates insights from automated AE into practical guidance for designing higher-quality artifacts with clearer review routes, more explicit claim-to-output links, and more concrete reuse paths.

cs.SE↗

arXiv2026

CommitSuite: A Comprehensive Benchmark for Commit Classification and Message Generation

High-quality commit messages are critical for maintaining software projects, yet ensuring their consistency and informativeness remains a practical challenge. While the Conventional Commits Specification (CCS) provides a structured format for commit messages, research on CCS-based commit classification and commit message generation (CMG) is limited by the absence of large-scale benchmarks, semantic annotations, and reliable evaluation methods. In this paper, we introduce CommitSuite, a benchmark comprising 63,533 CCS-compliant commits from 243 open-source repositories across seven programming languages. Each commit is labeled with its CCS type and enriched with AST-level code changes, along with LLM-assisted semantic annotations that capture the "what" and "why" behind the change. To evaluate CMG systems, we propose a reference-free framework based on five binary metrics: rationality, comprehensiveness, non-redundancy, authenticity, and logicality, enabling semantic-level assessment without relying on human-written references. Our experiments show that LLMs can effectively support both generation and evaluation, with evaluation achieving 0.849 Cohen's Kappa agreement against human judgments. CommitSuite offers a unified resource for structured commit understanding and facilitates reproducible research on commit classification and generation.

cs.SE↗

arXiv2025

CommitShield: Tracking Vulnerability Introduction and Fix in Version Control Systems

Version control systems are commonly used to manage open-source software, in which each commit may introduce new vulnerabilities or fix existing ones. Researchers have developed various tools for detecting vulnerabilities in code commits, but their performance is limited by factors such as neglecting descriptive data and challenges in accurately identifying vulnerability introductions. To overcome these limitations, we propose CommitShield, which combines the code analysis capabilities of static analysis tools with the natural language and code understanding capabilities of large language models (LLMs) to enhance the accuracy of vulnerability introduction and fix detection by generating precise descriptions and obtaining rich patch contexts. We evaluate CommitShield using the newly constructed vulnerability repair dataset, CommitVulFix, and a cleaned vulnerability introduction dataset. Experimental results indicate that CommitShield improves recall by 76%-87% over state-of-the-art methods in the vulnerability fix detection task, and its F1-score improves by 15%-27% in the vulnerability introduction detection task.

cs.SE↗