arXiv subjects

Shaowei Wang

Publications and source records attributed to Shaowei Wang.

At least 37 records · Page 2Linked to original sources

arXiv2026

When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation

The rapid evolution of software libraries creates a significant challenge for Large Language Models (LLMs), whose static parametric knowledge often becomes stale post-training. While retrieval-augmented generation (RAG) is commonly used to provide up-to-date API specifications, "context-memory conflict" arises when external instructions contradict a model's internal parametric knowledge. This paper presents a systematic empirical study of LLM code generation under API evolution (e.g., API deprecation, API modification, and API addition), by constructing a benchmark of 270 real-world updates from eight Python libraries. We evaluate four LLM families of 11 models. Our results show that without comprehensive documentation, LLMs struggle to prioritize external context, averaging only 42.55% of generated code examples are executable in the target environment. While structured documentation and larger model scales improve LLMs' ability to update adoption, they do not fully resolve executability issues with a low 66.36% executable rate. In addition, reasoning-based strategies (e.g., Self-Reflection) significantly boost LLMs' performance with 11% improvement on executable rate. Our findings highlight the persistence of outdated patterns from LLMs, even when API update specifications are provided, and emphasize the need for evolution-aware benchmarks and techniques.

cs.SE↗

arXiv2026

CODEPROMPTZIP: Code-specific Prompt Compression for Retrieval-Augmented Generation in Coding Tasks with LMs

Retrieval-Augmented Generation (RAG) enhances coding tasks by incorporating retrieved code examples into prompts. However, lengthy prompts, often exceeding tens of thousands of tokens, introduce challenges related to limited context windows of language models (LMs) and high computational costs. Existing prompt compression techniques focus on natural language, lacking tailored solutions for code. To address the gap, we propose CodePromptZip, a framework that compresses code examples before integrating into RAG workflows. Our framework employs a type-aware, priority-driven strategy to construct training samples for training code compression model. By using program analysis, we identify token types (e.g., Identifier) and perform ablation analysis to rank their removal priorities based on their impact on task performance. We then train a small LM as the compressor on these samples, enabling flexible compression conditioned on specified ratios while minimizing performance degradation. Specially, the compressor is augmented with a copy mechanism, allowing tokens to be directly copied from the original code snippets. Evaluation results show that CodePromptZip surpasses SOTA entropy-based and distillation-based baselines, improving by 23.4%, 28.7%, and 8.7% over the best baseline for Assertion Generation, Bugs2Fix, and Code Suggestion, respectively.

cs.SE↗

arXiv2026

REAgent: Requirement-Driven LLM Agents for Software Issue Resolution

Issue resolution aims to automatically generate patches from given issue descriptions and has attracted significant attention with the rapid advancement of large language models (LLMs). However, due to the complexity of software issues and codebases, LLM-generated patches often fail to resolve corresponding issues. Although various advanced techniques have been proposed with carefully designed tools and workflows, they typically treat issue descriptions as direct inputs and largely overlook their quality (e.g., missing critical context or containing ambiguous information), which hinders LLMs from accurate understanding and resolution. To address this limitation, we draw on principles from software requirements engineering and propose REAgent, a requirement-driven LLM agent framework that introduces issue-oriented requirements as structured task specifications to better guide patch generation. Specifically, REAgent automatically constructs structured and information-rich issue-oriented requirements, identifies low-quality requirements, and iteratively refines them to improve patch correctness. We conduct comprehensive experiments on three widely used benchmarks using two advanced LLMs, comparing against five representative or state-of-the-art baselines. The results demonstrate that REAgent consistently outperforms all baselines, achieving an average improvement of 17.40% in terms of the number of successfully-resolved issues (% Resolved).

cs.SE↗

arXiv2026

Typify: A Lightweight Usage-driven Static Analyzer for Precise Python Type Inference

Python's dynamic type system, while offering significant flexibility and expressiveness, poses substantial challenges for static analysis and automated tooling, particularly in unannotated or partially annotated codebases. Existing type inference approaches often depend on existing type annotations or on deep learning models that require extensive training corpora and considerable computational resources, resulting in limited scalability and reduced interpretability. We introduce Typify, a lightweight, usage-driven static analysis engine designed to infer precise and contextually relevant type information without relying on statistical learning or large datasets. Typify integrates symbolic execution with iterative fixpoint analysis and a context-matching retrieval system to propagate and predict type information across entire projects. By constructing and traversing dependency graphs in an execution-aware manner, Typify accurately connects function calls to their definitions and infers usage-based type semantics, even in complex, interdependent modules. We evaluate Typify on a diverse corpus of real-world Python repositories, including the ManyTypes4Py and Typilus datasets, benchmarking its effectiveness in predicting types of variables, arguments, and return statements. Results from the evaluation show that Typify consistently matches or surpasses state-of-the-art deep learning-based systems such as Type4Py and HiTyper, as well as industry-standard static type inference tools like Pyre. Our findings demonstrate that usage-driven, retrieval-based inference can match or exceed the accuracy of data-driven methods, offering a practical, interpretable, and computationally efficient alternative for large and evolving Python codebases.

cs.SE↗

arXiv2026

A Survey of Code Review Benchmarks and Evaluation Practices in Pre-LLM and LLM Era

Code review is a critical practice in modern software engineering, helping developers detect defects early, improve code quality, and facilitate knowledge sharing. With the rapid advancement of large language models (LLMs), a growing body of work has explored automated support for code review. However, progress in this area is hindered by the lack of a systematic understanding of existing benchmarks and evaluation practices. Current code review datasets are scattered, vary widely in design, and provide limited insight into what review capabilities are actually being assessed. In this paper, we present a comprehensive survey of code review benchmarks spanning both the Pre-LLM and LLM eras (2015--2025). We analyze 99 research papers (58 Pre-LLM era and 41 LLM era) and extract key metadata, including datasets, evaluation metrics, data sources, and target tasks. Based on this analysis, we propose a multi-level taxonomy that organizes code review research into five domains and 18 fine-grained tasks. Our study reveals a clear shift toward end-to-end generative peer review, increasing multilingual coverage, and a decline in standalone change understanding tasks. We further identify limitations of current benchmarks and outline future directions, including broader task coverage, dynamic runtime evaluation, and taxonomy-guided fine-grained assessment. This survey provides a structured foundation for developing more realistic and comprehensive benchmarks for LLM-based code review.

cs.SE↗

arXiv2026

When Elo Lies: Hidden Biases in Codeforces-Based Evaluation of Large Language Models

As Large Language Models (LLMs) achieve breakthroughs in complex reasoning, Codeforces-based Elo ratings have emerged as a prominent metric for evaluating competitive programming capabilities. However, these ratings are often reported without critical experimental details, leading to significant discrepancies illustrated by recent reports where the score of the same model version fluctuated by nearly 500 points. This paper presents a systematic empirical study on the hidden factors biasing Elo evaluations: (1) the temporal ordering of submissions, (2) contest difficulty selection, and (3) run to run stochastic variability of LLMs. Utilizing a controlled benchmark of 37 recent Codeforces contests and 13,691 generated test cases, we demonstrate that Elo scores are highly sensitive to these parameters. Our findings reveal that varying submission orders can shift scores by 394 points, while contest selection can cause differences of up to 1,122 points for the same model. Run to run performance exhibits substantial instability, with a maximum difference of 349 points in mean scores observed when evaluating identical contests. We conclude that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.

cs.SE↗

arXiv2026

SWE-Lego: Pushing the Limits of Supervised Fine-tuning for Software Issue Resolving

We present SWE-Lego, a supervised fine-tuning (SFT) recipe designed to achieve state-ofthe-art performance in software engineering (SWE) issue resolving. In contrast to prevalent methods that rely on complex training paradigms (e.g., mid-training, SFT, reinforcement learning, and their combinations), we explore how to push the limits of a lightweight SFT-only approach for SWE tasks. SWE-Lego comprises three core building blocks, with key findings summarized as follows: 1) the SWE-Lego dataset, a collection of 32k highquality task instances and 18k validated trajectories, combining real and synthetic data to complement each other in both quality and quantity; 2) a refined SFT procedure with error masking and a difficulty-based curriculum, which demonstrably improves action quality and overall performance. Empirical results show that with these two building bricks alone,the SFT can push SWE-Lego models to state-of-the-art performance among open-source models of comparable size on SWE-bench Verified: SWE-Lego-Qwen3-8B reaches 42.2%, and SWE-Lego-Qwen3-32B attains 52.6%. 3) We further evaluate and improve test-time scaling (TTS) built upon the SFT foundation. Based on a well-trained verifier, SWE-Lego models can be significantly boosted--for example, 42.2% to 49.6% and 52.6% to 58.8% under TTS@16 for the 8B and 32B models, respectively.

cs.SE↗

arXiv2025

Understanding Private Learning From Feature Perspective

Differentially private Stochastic Gradient Descent (DP-SGD) has become integral to privacy-preserving machine learning, ensuring robust privacy guarantees in sensitive domains. Despite notable empirical advances leveraging features from non-private, pre-trained models to enhance DP-SGD training, a theoretical understanding of feature dynamics in private learning remains underexplored. This paper presents the first theoretical framework to analyze private training through a feature learning perspective. Building on the multi-patch data structure from prior work, our analysis distinguishes between label-dependent feature signals and label-independent noise, a critical aspect overlooked by existing analyses in the DP community. Employing a two-layer CNN with polynomial ReLU activation, we theoretically characterize both feature signal learning and data noise memorization in private training via noisy gradient descent. Our findings reveal that (1) Effective private signal learning requires a higher signal-to-noise ratio (SNR) compared to non-private training, and (2) When data noise memorization occurs in non-private learning, it will also occur in private learning, leading to poor generalization despite small training loss. Our findings highlight the challenges of private learning and prove the benefit of feature enhancement to improve SNR. Experiments on synthetic and real-world datasets also validate our theoretical findings.

cs.LG↗

arXiv2025

Studying the Impact of Early Test Termination Due to Assertion Failure on Code Coverage and Spectrum-based Fault Localization

An assertion is commonly used to validate the expected programs behavior (e.g., if the returned value of a method equals an expected value) in software testing. Although it is a recommended practice to use only one assertion in a single test to avoid code smells (e.g., Assertion Roulette), it is common to have multiple assertions in a single test. One issue with tests that have multiple assertions is that when the test fails at an early assertion (not the last one), the test will terminate at that point, and the remaining testing code will not be executed. This, in turn, can potentially reduce the code coverage and the performance of techniques that rely on code coverage information (e.g., spectrum-based fault localization). We refer to such a scenario as early test termination. Understanding the impact of early test termination on test coverage is important for software testing and debugging, particularly for the techniques that rely on coverage information obtained from the testing. We conducted the first empirical study on early test termination due to assertion failure (i.e., early test termination) by investigating 207 versions of 6 open-source projects. We found that a nonnegligible portion of the failed tests (19.1%) is early terminated due to assertion failure. Our findings indicate that early test termination harms both code coverage and the effectiveness of spectrum-based fault localization. For instance, after eliminating early test termination, the line/branch coverage is improved in 55% of the studied versions, and improves the performance of two popular SBFL techniques Ochiai and Tarantula by 15.1% and 10.7% compared to the original setting (without eliminating early test termination) in terms of MFR, respectively.

cs.SE↗

arXiv2025

Lithium Niobate Vertical Cavity Electro-Optic Modulator

Electro-optic modulators (EOMs) are vital for optical imaging and information processing, with free-space devices enabling LiDAR and beam control. Lithium niobate (LN), powered by the strong Pockels effect and scalable LN-on-insulator (LNOI) platform, has become a leading material for high-performance EOMs. Here we realize a vertical-cavity EOM in which an LN membrane is sandwiched between two photonic crystal (PhC) mirrors with integrated electrodes. The cavity supports sharp defect-mode resonances that shift efficiently under the Pockels effect, enabling strong modulation of transmission. Experiments show a depth of 43 % at 50 V and a bandwidth of 5 MHz. This architecture combines free-space compatibility with fabrication simplicity, opening new routes to compact electro-optic platforms for ranging, holography, and beam steering.

physics.optics↗

arXiv2025

LLM-Based Detection of Tangled Code Changes for Higher-Quality Method-Level Bug Datasets

Tangled code changes, commits that conflate unrelated modifications such as bug fixes, refactorings, and enhancements, introduce significant noise into bug datasets and adversely affect the performance of bug prediction models. Addressing this issue at a fine-grained, method-level granularity remains unexplored. This is critical to address, as recent bug prediction models, driven by practitioner demand, are increasingly focusing on finer granularity rather than traditional class- or file-level predictions. This study investigates the utility of Large Language Models (LLMs) for detecting tangled code changes by leveraging both commit messages and method-level code diffs. We formulate the problem as a binary classification task and evaluate multiple prompting strategies, including zero-shot, few-shot, and chain-of-thought prompting, using state-of-the-art proprietary LLMs such as GPT-5 and Gemini-2.0-Flash, and open-source models such as GPT-OSS-120B and CodeBERT. Our results demonstrate that combining commit messages with code diffs significantly enhances model performance, with the combined few-shot and chain-of-thought prompting achieving an F1-score of 0.883. Additionally, we explore machine learning models trained on LLM-generated embeddings, where a multi-layer perceptron classifier achieves superior performance (F1-score: 0.906, MCC: 0.807). Applying our approach to 49 open-source projects improves the distributional separability of code metrics between buggy and non-buggy methods, demonstrating the promise of LLMs for method-level commit untangling and potentially contributing to improving the accuracy of future bug prediction models.

cs.SE↗

arXiv2025

FPT-Noise: Dynamic Scene-Aware Counterattack for Test-Time Adversarial Defense in Vision-Language Models

Vision-Language Models (VLMs), such as CLIP, have demonstrated remarkable zero-shot generalizability across diverse downstream tasks. However, recent studies have revealed that VLMs, including CLIP, are highly vulnerable to adversarial attacks, particularly on their visual modality. Traditional methods for improving adversarial robustness, such as adversarial training, involve extensive retraining and can be computationally expensive. In this paper, we propose a new Test-Time defense: Feature Perception Threshold Counterattack Noise (FPT-Noise), which enhances the adversarial robustness of CLIP without costly fine-tuning. Our core contributions are threefold: First, we introduce a Dynamic Feature Modulator that dynamically generate an image-specific and attack-adaptive noise intensity parameter. Second, We reanalyzed the image features of CLIP. When images are exposed to different levels of noise, clean images and adversarial images exhibit distinct rates of feature change. We established a feature perception threshold to distinguish clean images from attacked ones. Finally, we integrate a Scene-Aware Regulation guided by a stability threshold and leverage Test-Time Transformation Ensembling (TTE) to further mitigate the impact of residual noise and enhance robustness.Extensive experimentation has demonstrated that FPT-Noise significantly outperforms existing Test-Time defense methods, boosting average robust accuracy from 0.07% to 56.86% under AutoAttack while maintaining high performance on clean images (-1.1%). The code will be made public following the publication of the study. The code will be made public following the publication of the study.

cs.CR↗

arXiv2025

Lingxi: Repository-Level Issue Resolution Framework Enhanced by Procedural Knowledge Guided Scaling

Driven by the advancements of Large Language Models (LLMs), LLM-powered agents are making significant improvements in software engineering tasks, yet struggle with complex, repository-level issue resolution. Existing agent-based methods have two key limitations. First, they lack of procedural knowledge (i.e., how an issue is fixed step-by-step and rationales behind it) to learn and leverage for issue resolution. Second, they rely on massive computational power to blindly explore the solution space. % To address those limitations, we propose Lingxi, an issue resolution framework that leverages procedural knowledge extracted from historical issue-fixing data to guide agents in solving repository-level issues. \ourTool first constructs this knowledge offline through a hierarchical abstraction mechanism, enabling agents to learn the how and why behind a fix, not just the final solution. During online application, it employs a knowledge-driven scaling method that leverages the procedural knowledge of similar issues to intelligently analyze the target issue from multiple perspectives, in sharp contrast to undirected, brute-force exploration. % Lingxi successfully resolves 74.6\% of bugs on the SWE-bench Verified benchmark in Past@1 setting, outperforming five state-of-the-art techniques by a significant margin (5.4\% to 14.9\%). Our comprehensive ablation study confirmed that the success of Lingxi comes directly from its use of procedural knowledge. Without it, the performance gains from scaling alone is negligible. Our qualitative study further shows that the ``design patterns $\&$ coding practices'' is the most critical knowledge aspect, and that the roles of different knowledge aspects switch across different stages (i.e., analysis, planning, and fixing).

cs.SE↗

arXiv2025

Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization

Large Language Models (LLMs) show great promise in software engineering tasks like Fault Localization (FL) and Automatic Program Repair (APR). This study investigates the impact of input order and context size on LLM performance in FL, a crucial step for many downstream software engineering tasks. We test different orders for methods using Kendall Tau distances, including "perfect" (where ground truths come first) and "worst" (where ground truths come last), using two benchmarks that consist of both Java and Python projects. Our results indicate a significant bias in order; Top-1 FL accuracy in Java projects drops from 57% to 20%, while in Python projects, it decreases from 38% to approximately 3% when we reverse the code order. Breaking down inputs into smaller contexts helps reduce this bias, narrowing the performance gap in FL from 22% to 6% and then to just 1% on both benchmarks. We then investigated whether the bias in order was caused by data leakage by renaming the method names with more meaningful alternatives. Our findings indicated that the trend remained consistent, suggesting that the bias was not due to data leakage. We also look at ordering methods based on traditional FL techniques and metrics. Ordering using DepGraph's ranking achieves 48% Top-1 accuracy, which is better than more straightforward ordering approaches like CallGraphDFS. These findings underscore the importance of how we structure inputs, manage contexts, and choose ordering methods to improve LLM performance in FL and other software engineering tasks.

cs.SE↗

arXiv2025

Consistent Estimation of Numerical Distributions under Local Differential Privacy by Wavelet Expansion

Distribution estimation under local differential privacy (LDP) is a fundamental and challenging task. Significant progresses have been made on categorical data. However, due to different evaluation metrics, these methods do not work well when transferred to numerical data. In particular, we need to prevent the probability mass from being misplaced far away. In this paper, we propose a new approach that express the sample distribution using wavelet expansions. The coefficients of wavelet series are estimated under LDP. Our method prioritizes the estimation of low-order coefficients, in order to ensure accurate estimation at macroscopic level. Therefore, the probability mass is prevented from being misplaced too far away from its ground truth. We establish theoretical guarantees for our methods. Experiments show that our wavelet expansion method significantly outperforms existing solutions under Wasserstein and KS distances.

cs.LG↗

arXiv2025

SLICET5: Static Program Slicing using Language Models with Copy Mechanism and Constrained Decoding

Static program slicing is a fundamental technique in software engineering. Traditional static slicing tools rely on parsing complete source code, which limits their applicability to real-world scenarios where code snippets are incomplete or unparsable. While recent research developed learning-based approaches to predict slices, they face critical challenges: (1) Inaccurate dependency identification, where models fail to precisely capture data and control dependencies between code elements; and (2) Unconstrained generation, where models produce slices with extraneous or hallucinated tokens not present in the input, violating the structural integrity of slices. To address these challenges, we propose \ourtool, a novel slicing framework that reformulates static program slicing as a sequence-to-sequence task using lightweight language models (e.g., CodeT5+). Our approach incorporates two key innovations. First, we introduce a copy mechanism that enables the model to more accurately capture inter-element dependencies and directly copy relevant tokens from the input, improving both dependency reasoning and generation constraint. Second, we design a constrained decoding process with (a) lexical constraint, restricting outputs to input tokens only, and (b) syntactic constraint, leveraging Tree Similarity of Edit Distance (TSED) monotonicity to detect structurally invalid outputs and discard them. We evaluate \ourtool on CodeNet and LeetCode datasets and show it consistently outperforms state-of-the-art baselines, improving ExactMatch scores by up to 27\%. Furthermore, \ourtool demonstrates strong performance on incomplete code, highlighting its robustness and practical utility in real-world development environments.

cs.SE↗

arXiv2025

Diagram-Driven Course Questions Generation

Visual Question Generation (VQG) research focuses predominantly on natural images while neglecting the diagram, which is a critical component in educational materials. To meet the needs of pedagogical assessment, we propose the Diagram-Driven Course Questions Generation (DDCQG) task and construct DiagramQG, a comprehensive dataset with 15,720 diagrams and 25,798 questions across 37 subjects and 371 courses. Our approach employs course and input text constraints to generate course-relevant questions about specific diagram elements. We reveal three challenges of DDCQG: domain-specific knowledge requirements across courses, long-tail distribution in course coverage, and high information density in diagrams. To address these, we propose the Hierarchical Knowledge Integration framework (HKI-DDCQG), which utilizes trainable CLIP for identifying relevant diagram patches, leverages frozen vision-language models for knowledge extraction, and generates questions with trainable T5. Experiments demonstrate that HKI-DDCQG outperforms existing models on DiagramQG while maintaining strong generalizability across natural image datasets, establishing a strong baseline for DDCQG.

cs.CV↗

arXiv2025

Model Performance-Guided Evaluation Data Selection for Effective Prompt Optimization

Optimizing Large Language Model (LLM) performance requires well-crafted prompts, but manual prompt engineering is labor-intensive and often ineffective. Automated prompt optimization techniques address this challenge but the majority of them rely on randomly selected evaluation subsets, which fail to represent the full dataset, leading to unreliable evaluations and suboptimal prompts. Existing coreset selection methods, designed for LLM benchmarking, are unsuitable for prompt optimization due to challenges in clustering similar samples, high data collection costs, and the unavailability of performance data for new or private datasets. To overcome these issues, we propose IPOMP, an Iterative evaluation data selection for effective Prompt Optimization using real-time Model Performance. IPOMP is a two-stage approach that selects representative and diverse samples using semantic clustering and boundary analysis, followed by iterative refinement with real-time model performance data to replace redundant samples. Evaluations on the BIG-bench dataset show that IPOMP improves effectiveness by 1.6% to 5.3% and stability by at least 57% compared with SOTA baselines, with minimal computational overhead below 1%. Furthermore, the results demonstrate that our real-time performance-guided refinement approach can be universally applied to enhance existing coreset selection methods.

cs.CL↗