arXiv subjects

Shengyi Pan

Publications and source records attributed to Shengyi Pan.

6 recordsLinked to original sources

arXiv2026

Answer Is Cheap, Show Me the Evidence! Augmenting Automated Vulnerability Assessment with Evidence

Software vulnerability (SV) assessment helps prioritize remediation by characterizing reported vulnerabilities. Existing automated methods predict assessment results from SV reports (SVRs), but often overlook information in rich text, such as screenshots and code snippets, as well as contextual information about vulnerable projects. They also focus on prediction accuracy without providing explanations or supporting evidence, limiting their practical use when analysts must validate imperfect predictions. We propose EAVA, a framework that uses large language models (LLMs) to assess SVs and provide supporting evidence. EAVA employs specialized LLM agents to process rich-text content and project information, and builds a dedicated assessment model through a two-stage training pipeline. It first uses supervised instruction tuning on automatically annotated reasoning trajectories to inject domain knowledge, and then applies reinforcement learning to improve intrinsic reasoning. EAVA also retrieves similar historical vulnerabilities as supplementary evidence. Experiments on a newly collected SVR dataset show that EAVA outperforms the strongest baseline by 5.3 to 35.2 percent across multiple metrics. Ablation studies confirm the effectiveness of assessment-specific model training and information enrichment. A user study with security experts further demonstrates that the evidence provided by EAVA is useful and practical for real-world SV assessment.

cs.SE↗

arXiv2026

Mitigating Implicit Inconsistencies in Patch Porting

Promptly porting patches from a source codebase to its variants (e.g., forks and branches) is essential for mitigating propagated defects and vulnerabilities. Recent studies have explored automated patch porting to reduce manual effort and delay, but existing approaches mainly handle inconsistencies visible in a patch's local context and struggle with those requiring global mapping knowledge between codebases. We refer to such non-local inconsistencies as implicit inconsistencies. Implicit inconsistencies pose greater challenges for developers to resolve due to their non-local nature. To address them, we propose MIP, which enables collaboration among an LLM, a compiler, and code analysis utilities. MIP adopts different strategies for different cases: when source identifiers exist in the target codebase, it leverages compiler diagnostics; otherwise, it retrieves matched code segment pairs from the two codebases as mapping knowledge for mitigation. Experiments on two representative scenarios, cross-fork and cross-branch patch porting, show that MIP successfully resolves more than twice as many patches as the best-performing baseline in both settings. A user study with our industry partner further demonstrates its practical effectiveness.

cs.SE↗

arXiv2026

Multi-Agent End-to-End Vulnerability Management for Mitigating Recurring Vulnerabilities

Software vulnerability management has become increasingly critical as modern systems scale in size and complexity. However, existing automated approaches remain insufficient. Traditional static analysis methods struggle to precisely capture contextual dependencies, especially when vulnerabilities span multiple functions or modules. Large language models (LLMs) often lack the ability to retrieve and exploit sufficient contextual information, resulting in incomplete reasoning and unreliable outcomes. Meanwhile, recurring vulnerabilities emerge repeatedly due to code reuse and shared logic, making historical vulnerability knowledge an indispensable foundation for effective vulnerability detection and repair. Nevertheless, prior approaches such as clone-based detection and patch porting, have not fully leveraged this knowledge. To address these challenges, we present MAVM, a multi-agent framework for end-to-end recurring vulnerability management. MAVM integrates five components, including a vulnerability knowledge base, detection, confirmation, repair, and validation, into a unified multi-agent pipeline. We construct a knowledge base from publicly disclosed vulnerabilities, thereby addressing the underuse of historical knowledge in prior work and mitigating the lack of domain-specific expertise in LLMs. Furthermore, we design context-retrieval tools that allow agents to extract and reason over repository-level information, overcoming the contextual limitations of previous methods. Based on agents, MAVM effectively simulates real-world security workflows. To evaluate the performance of MAVM, we construct a dataset containing 78 real-world patch-porting cases (covering 114 function-level migrations). On this dataset, MAVM successfully detects and repairs 51 real vulnerabilities, outperforming baselines by 31.9%-45.2% in repair accuracy, which demonstrates its effectiveness.

cs.SE↗

arXiv2025

Unit Test Update through LLM-Driven Context Collection and Error-Type-Aware Refinement

Unit testing is critical for ensuring software quality and software system stability. The current practice of manually maintaining unit tests suffers from low efficiency and the risk of delayed or overlooked fixes. Therefore, an automated approach is required to instantly update unit tests, with the capability to both repair and enhance unit tests. However, existing automated test maintenance methods primarily focus on repairing broken tests, neglecting the scenario of enhancing existing tests to verify new functionality. Meanwhile, due to their reliance on rule-based context collection and the lack of verification mechanisms, existing approaches struggle to handle complex code changes and often produce test cases with low correctness. To address these challenges, we propose TESTUPDATER, a novel LLM based approach that enables automated just-in-time test updates in response to production code changes. TESTUPDATER first leverages the LLM to analyze code changes and identify relevant context, which it then extracts and filters. Then, through carefully designed prompts, TESTUPDATER guides the LLM step by step to handle various types of code changes and introduce new dependencies, enabling both test repair and enhancement. Finally, we introduce an error-type-aware iterative refinement mechanism that executes the LLM-updated tests and repairs failures, which significantly improves the overall correctness of test updates. Since existing test repair datasets lack scenarios of test enhancement, we further construct a new benchmark, UPDATES4J, with 195 real-world samples from 7 projects. Experimental results show that TESTUPDATER achieves a compilation pass rate of 94.4% and a test pass rate of 86.7%, outperforming the state-of-the-art method SYNTER by 15.9% and 20.0%, respectively. Furthermore, TESTUPDATER exhibits 12.9% higher branch coverage and 15.2% greater line coverage than SYNTER.

cs.SE↗

arXiv2024

Similar but Patched Code Considered Harmful -- The Impact of Similar but Patched Code on Recurring Vulnerability Detection and How to Remove Them

Identifying recurring vulnerabilities is crucial for ensuring software security. Clone-based techniques, while widely used, often generate many false alarms due to the existence of similar but patched (SBP) code, which is similar to vulnerable code but is not vulnerable due to having been patched. Although the SBP code poses a great challenge to the effectiveness of existing approaches, it has not yet been well explored. In this paper, we propose a programming language agnostic framework, Fixed Vulnerability Filter (FVF), to identify and filter such SBP instances in vulnerability detection. Different from existing studies that leverage function signatures, our approach analyzes code change histories to precisely pinpoint SBPs and consequently reduce false alarms. Evaluation under practical scenarios confirms the effectiveness and precision of our approach. Remarkably, FVF identifies and filters 65.1% of false alarms from four vulnerability detection tools (i.e., ReDeBug, VUDDY, MVP, and an elementary hash-based approach) without yielding false positives. We further apply FVF to 1,081 real-world software projects and construct a real-world SBP dataset containing 6,827 SBP functions. Due to the SBP nature, the dataset can act as a strict benchmark to test the sensitivity of the vulnerability detection approach in distinguishing real vulnerabilities and SBPs. Using this dataset, we demonstrate the ineffectiveness of four state-of-the-art deep learning-based vulnerability detection approaches. Our dataset can help developers make a more realistic evaluation of vulnerability detection approaches and also paves the way for further exploration of real-world SBP scenarios.

cs.SE↗

arXiv2024

Automating Zero-Shot Patch Porting for Hard Forks

Forking is a typical way of code reuse, which provides a simple way for developers to create a variant software (denoted as hard fork) by copying and modifying an existing codebase. Despite of the benefits, forking also leads to duplicate efforts in software maintenance. Developers need to port patches across the hard forks to address similar bugs or implement similar features. Due to the divergence between the source project and the hard fork, patch porting is complicated, which requires an adaption regarding different implementations of the same functionality. In this work, we take the first step to automate patch porting for hard forks under a zero-shot setting. We first conduct an empirical study of the patches ported from Vim to Neovim over the last ten years to investigate the necessities of patch porting and the potential flaws in the current practice. We then propose a large language model (LLM) based approach (namely PPatHF) to automatically port patches for hard forks on a function-wise basis. Specifically, PPatHF is composed of a reduction module and a porting module. Given the pre- and post-patch versions of a function from the reference project and the corresponding function from the target project, the reduction module first slims the input functions by removing code snippets less relevant to the patch. Then, the porting module leverages a LLM to apply the patch to the function from the target project. We evaluate PPatHF on 310 Neovim patches ported from Vim. The experimental results show that PPatHF outperforms the baselines significantly. Specifically, PPatHF can correctly port 131 (42.3%) patches and automate 57% of the manual edits required for the developer to port the patch.

cs.SE↗