arXiv subjects

Michael Pradel

Publications and source records attributed to Michael Pradel.

At least 19 recordsLinked to original sources

arXiv2026

IterTestQ: Assembly-Level, Cross-Platform Testing of Quantum Computing Platforms

Quantum computing platforms are susceptible to quantum-specific bugs, such as incorrectly ordering qubits or incorrectly implementing quantum abstractions. These bugs are difficult to detect and require specialized expertise. The field faces challenges due to a fragmented landscape of platforms and rapid development cycles that often prioritize new features over thorough testing, severely hindering the reliability of quantum software. To address these challenges, we present IterTestQ, a novel cross-platform testing approach for quantum computing platforms. The key technical contribution is our novel ITE process, which generates equivalent quantum programs by iteratively (I)mporting them into platform-specific representations, (T)ransforming the program via optimizations and gate conversions, and (E)xporting the program again. To transfer programs across platforms and test cross-platform consistency, IterTestQ leverages QASM, an assembly-level representation supported by most platforms. The approach uses a crash oracle to detect failures during cross-platform transformations and an equivalence oracle to validate the semantic consistency of the generated assembly programs, which are expected to be equivalent by construction. We evaluate IterTestQ on widely-used quantum computing platforms, including Qiskit, PennyLane, Pytket, BQSKit, and Cirq, revealing 23 bugs, 17 of which are already confirmed or fixed. Our results also demonstrate that IterTestQ complements existing quantum fuzzers (covering tens of thousands of otherwise uncovered lines), is efficient (with 0.00089 seconds per generated program), and that the ITE process is crucial for its effectiveness.

cs.SE↗

arXiv2026

Prompt Less, Infer More: Static Analysis-Guided LLM Type Annotation for Python

Type annotations improve the maintainability and reliability of Python programs, but many real-world projects remain under-annotated because manual annotation is time-consuming and error-prone. Recent neural and LLM-based type inference systems can recover missing annotations, yet they often trade accuracy for efficiency. Existing approaches often fail to exploit project-wide type evidence, or do so inefficiently through unnecessary model calls and noisy prompts. We present SALT, an LLM-based type inference system guided by static analysis. SALT integrates static analysis with LLM inference to resolve types statically, prioritize LLM queries, and construct concise type-aware context. To demonstrate the effectiveness of SALT, we evaluated it on 12 real-world Python projects. SALT achieves the best accuracy among the evaluated systems and reduces estimated LLM cost by up to 78.5% compared with a strong coding-assistant baseline. It also shows similar cost savings with the frontier model GPT-5.5, demonstrating that SALT remains effective with strong LLMs.

cs.SE↗

arXiv2026

BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications

Software-engineering research relies on benchmarks that evaluate new techniques on a set of problem instances. Unfortunately, manually constructing benchmarks is labor-intensive, as researchers must find suitable instances, prepare execution environments, establish expected outcomes, and validate correctness. This paper presents BenchMeUp, an approach for automatically creating software-engineering benchmarks from natural-language specifications. Given a specification defining the desired instances, data sources, and evaluation procedures, BenchMeUp generates search rules and a construction plan, invokes parallel LLM agents that identify candidate instances and construct executable instances, validates the results in fresh execution environments, and packages the instances behind a unified command-line interface. We evaluate the approach on four tasks: program repair, issue reproduction, compiler autotuning, and fixing static-analysis warnings. Targeting 50 instances per task (200 in total), BenchMeUp constructs 198 instances, of which 196 (99\%) are successfully packaged. Search and construction take 36 minutes and \$0.79 per instance, on average. Evaluating eight existing techniques on the generated benchmarks, we find that 189 of 196 packaged instances (96\%) support end-to-end evaluation, allowing users to study the effectiveness of the techniques. Checking every constructed instance against its source, the specification, and its expected outcome, we find that 180 of 198 (91\%) meet all criteria, compared with 10 of 200 instances delivered by Codex (general-purpose coding agent). Overall, we show that LLM-based agents can automate benchmark construction across diverse software-engineering tasks and produce executable instances that existing techniques can use, while validity beyond execution still needs to be checked.

cs.SE↗

arXiv2026

The Hidden Costs of Forgetting: A Study of Trajectory Deletion in Software Engineering Agents

Software engineering agents resend their entire trajectory to an LLM at every turn, motivating online trajectory reduction. Prior work shows that randomly deleting 75% of tokens in old turns may preserve agent effectiveness, but it remains unclear why, how agents compensate for missing content, and whether it actually reduces costs. This paper presents an empirical study of 42 trajectory-reduction configurations, focusing on effectiveness, agent behavior, and cost. We systematically explore deleting content from old turns by varying the scope, granularity, amount, and timing of deletion across 9,400 runs with two state-of-the-art agents on SWE-bench Verified. We find deleting large amounts of trajectory content (e.g., up to 100% of reasoning text) can preserve overall effectiveness. Importantly, such deletion must be applied with a delay, as immediate deletion tends to significantly degrade effectiveness. Agents often compensate for deleted content by re-fetching it, increasing both the number of turns taken and the input/output tokens. Whether trajectory reduction reduces costs depends on the pricing model: Under a flat pricing model, estimated costs fall in 26 of 42 evaluations. Surprisingly, under a caching-aware pricing model, zero evaluations yield lower costs, since reducing the trajectory breaks the cached prefix billed at a lower rate. Our findings have three implications: First, token counts alone are misleading; trajectory reduction techniques must consider overall costs under a realistic pricing model. Second, analyzing trajectories is crucial to understand agent behavior, as end-to-end effectiveness hides differences such as excessive re-fetching. Finally, these results highlight the need for future work on memory management to reduce costs without sacrificing effectiveness.

cs.SE↗

arXiv2026

LLM-Based Repair of Static Nullability Errors

Modern Java projects increasingly adopt static analysis tools that prevent null-pointer exceptions by treating nullness as a type property. However, integrating such tools into large, existing codebases remains a significant challenge. While annotation inference can eliminate many errors automatically, a subset of residual errors $-$ typically a mix of real bugs and false positives $-$ often persists and can only be resolved via code changes. Manually addressing these errors is tedious and error-prone. Large language models (LLMs) offer a promising path toward automating these repairs, but naively prompted LLMs often generate incorrect, contextually inappropriate edits. We present NullRepair, a system that integrates LLMs into a structured workflow for resolving the errors from a nullability checker. NullRepair's decision process follows a flowchart derived from manual analysis of 200 real-world errors. It leverages static analysis to identify safe and unsafe usage regions of symbols, using error-free usage examples to contextualize model prompts. Patches are generated through an iterative interaction with the LLM that incorporates project-wide context and decision logic. Our evaluation on 12 real-world Java projects shows that NullRepair resolves 63% of the 1,119 nullability errors that remain after applying a state-of-the-art annotation inference technique. Unlike two baselines (single-shot prompt and mini-SWE-agent), NullRepair also largely preserves program semantics, with all unit tests passing in 10/12 projects after applying every edit proposed by NullRepair, and 98% or more tests passing in the remaining two projects.

cs.SE↗

arXiv2026

Doc2Feat-Bench: Evaluating Documentation-Driven Feature Addition

Documentation changes in mature software projects often describe newly introduced or modified behavior. This makes them a natural basis for documentation-driven feature addition, where software engineering agents implement features from public-facing documentation updates. However, existing benchmarks for agentic software development primarily rely on issue reports written for maintainer coordination and often contain implementation-level details. Motivated by this observation, this work introduces Doc2Feat-bench, a benchmark for documentation-driven feature addition. Doc2Feat-bench consists of 634 tasks across 10 mature open-source software projects, involving about 114k code changes in total. Each task pairs a documentation change with the corresponding implementation and developer-written tests, enabling evaluation of whether a software engineering agent can implement the feature described by the documentation diff. The benchmark is constructed through a five-phase pipeline, starting from release notes, and offers broad coverage across libraries, developer tools, and frameworks while preserving real-world development settings. To facilitate lightweight and reliable evaluation under limited resources, we further curate a human-validated subset named Doc2Feat-bench Verified. It contains 114 high-quality instances whose task clarity and evaluation validity are manually verified. We use Doc2Feat-bench to assess a range of state-of-the-art software engineering agents with two widely used scaffolds. Experimental results show that despite significant token consumption, the best task success rate remains as low as 37.72% (OpenHands with Qwen3-Coder-480B). Our analysis reveals that software engineering agents face key challenges in performing cross-file edits, understanding existing code modules, and correctly interpreting documentation semantics.

cs.SE↗

arXiv2026

Automated Table Reproduction via Code Generation

Reproducibility is an important goal in computer science research, e.g., for artifact evaluation and to build upon experimental results of prior work. Recently, LLM agents are being used to automatically reproduce research results, but they fail to provide executable evidence of reproduction and do not consider the method of reproduction, which limits their usefulness. We present Artisan, an LLM agent that reproduces tables of numeric results, given a paper and its artifact. The approach is enabled by two key contributions: First, we frame the reproduction problem as a code generation task, enabling users to audit and re-run the resulting reproduction script independently of the agent. Second, we design automated judging mechanisms that steer the agent toward correct results without exposing them, while preventing shortcuts like copying precomputed results. To evaluate Artisan, we introduce Artisan-Bench, the first benchmark assessing the ability to generate code that reproduces research results. Artisan-Bench comprises 60 tasks derived from 23 software engineering papers. Our experiments show that Artisan is effective and efficient, with the added benefit of aiding the discovery of 20 new errors in either the paper or artifact.

cs.SE↗

arXiv2026

Evaluating LLM Agents on Automated Software Analysis Tasks

Numerous software analysis tools exist today, yet applying them to diverse open-source projects remains challenging due to environment setup, dependency resolution, and analysis tool configuration. LLM-based agents offer a potential solution, yet their effectiveness on the specific task of automated software analysis has not been systematically studied. Unlike issue solving or general environment setup, this task requires installing and configuring a separate analysis tool alongside the target project, generating tool-specific prerequisites, and validating that the analysis tool produces meaningful analysis outputs rather than merely declaring successful termination without evidence. We introduce AnalysisBench, a benchmark of 35 tool-project pairs spanning seven analysis tools and ten diverse C/C++ and Java projects, each with a manually constructed reference setup. Using AnalysisBench, we evaluate four agent architectures across four LLM backends. Our custom agent, AnalysisAgent, achieves manually verified success rates of 94% (Gemini-3-Flash, 33/35 tasks), compared to 77% for the best baseline (ExecutionAgent). Beyond quantitative results, we identify key limitations in existing agents, including stage mixing, poor error localization, and premature termination, and show that agentic architecture plays a critical role beyond LLM capability alone. We further find that whole-program analyses and Java-specific tools are the most difficult tasks, that Java toolchains pose greater challenges than C/C++, and that self-validated success consistently overstates manually verified success. Extended runs with AnalysisAgent-produced setups also surface two previously unknown defects in masscan and radare2, confirming that the produced setups support real downstream analysis.

cs.SE↗

arXiv2026

PatchGuru: Patch Oracle Inference from Natural Language Artifacts

As software systems evolve, patches may unintentionally alter program behavior. Validating patches against their intended semantics is difficult due to incomplete regression tests and informal, non-executable natural language (NL) descriptions of patch intent. We present PatchGuru, the first automated technique that infers executable patch specifications from real-world pull requests (PRs). Given a PR, PatchGuru uses large language models (LLMs) to extract developer intent from NL artifacts and synthesizes patch oracles: under-approximate yet practical specifications expressed as runtime assertions in comparison programs that integrate pre- and post-patch versions. Patch oracles focus on patch-relevant behaviors, enable automated validation, and support cross-version properties. PatchGuru iteratively refines inferred oracles by comparing pre- and post-patch behaviors, identifies violations, filters inconsistencies via self-review, and generates bug reports. We evaluate PatchGuru on 400 recent PRs from four widely used open-source Python projects. PatchGuru reports 39 warnings with a precision of 0.62, yielding 24 confirmed true positives, including 12 previously unknown bugs, 11 of which were subsequently fixed by developers. Compared to the state-of-the-art technique Testora, PatchGuru detects 17 more bugs (24 vs. 7) while improving precision from 0.32 to 0.62. PatchGuru incurs an average cost of 8.9 minutes and USD 0.07 per PR. These results suggest that PatchGuru complements code review and regression testing by providing executable documentation and automated validation of patch intent.

cs.SE↗

arXiv2026

Prefactory: Automated Discovery and Application of Library-Adoption Refactorings

Replacing hand-written code with library API calls is a common refactoring that can reduce code size, make code more idiomatic, and reuse well-tested implementations. Yet many library-adoption opportunities are hard to find automatically: the original code often does not mention the target library and may resemble the library API only in behavior, with little syntactic overlap. Existing tools, such as linters and static modernizers, cover only a small set of manually specified patterns. LLMs and LLM-based agents, on the other hand, can generalize to more patterns, but they are costly, difficult to reproduce and to apply systematically at scale. This paper introduces Prefactory, an automated approach for library-adoption refactoring in Python. The key idea is to use an LLM to synthesize executable search heuristics rather than relying on repeated LLM prompting over a codebase. Given a target project and a target library name, Prefactory collects library metadata and project vocabulary, then generates lexical and structural detectors. Prefactory executes the detectors during a scan phase to find candidate functions. It then heuristically ranks the candidate functions, generates refactorings for the highest-ranked ones using an LLM, and validates the results with project tests and newly generated differential tests. We evaluate Prefactory on PrefactoryBench, a benchmark of 100 real-world library-adoption refactorings from 61 open-source Python projects and 18 libraries. Prefactory detects 75 instances at the file level and 56 at the function level, compared with 35 and 32 for the strongest baseline (Codex CLI). From the 56 detected functions, Prefactory produces 40 test-validated refactorings.

cs.SE↗

arXiv2026

READU: Inconsistency-Driven Just-in-Time Detection and Repair of README Bugs

Repository-level documentation, such as READMEs, is often the first point of contact between users and a repository. When this documentation is incorrect, users may encounter runtime errors or waste their time debugging. We call such mistakes in repository-level documentation README bugs. Addressing README bugs is challenging because documentation mixes prose with code, its connection to the source of truth is loose, and finding a bug still leaves developers to craft a repair. This paper presents READU, an inconsistency-driven technique for just-in-time detection and repair of README bugs. The key insight behind READU is that README bugs often manifest as inconsistencies between documentation and another source of truth: either repository-internal facts, such as source code, or repository-external facts, such as external dependencies. READU applies a high-recall commit filter, runs internal and external consistency checkers in parallel, uses an alert judge to remove false positives, and automatically synthesizes documentation patches. On 6,000 recent commits from six popular repositories including Linux and Spring Boot, READU detects 244 true positives with 75% precision, while consuming less than $0.01 and less than one minute per commit, on average. Of these true positives, READU correctly repairs 217. We report 66 found README bugs, of which (so far) 44 are confirmed and 26 are fixed.

cs.SE↗

arXiv2026

DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences

As software evolves over time, it is important to ensure that any behavioral changes occur as intended by developers. A promising approach for this goal is to generate tests that expose behavioral differences between the old and new versions of a program. However, current approaches fail to trigger behavioral differences for many code changes. This paper presents~DiffTestGen, a novel change-directed, LLM-based differential testing approach specifically designed to expose behavioral differences introduced by a code change. The approach is enabled by two key contributions: First, DiffTestGen leverages static call graph analysis and project documentation to identify valid entry points for test generation and to guide the LLM toward reaching the changed code. Second, DiffTestGen iteratively improves our newly introduced union coverage metric, which combines coverage of modified code in the old and the new version, by providing targeted coverage feedback to the LLM. We evaluate DiffTestGen on two datasets comprising a total of 463 PRs. DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%. Compared with the baselines, DiffTestGen exposes 99 more PRs overall and increases code coverage by 12.5% and 15.6% percentage points, respectively. By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.

cs.SE↗

arXiv2026

PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packages

Security vulnerabilities in software packages are a significant concern for developers and users alike. Patching these vulnerabilities in a timely manner is crucial to restoring the integrity and security of software systems. However, previous work has shown that vulnerability reports often lack proof-of-concept (PoC) exploits, which are essential for fixing the vulnerability, testing patches, and avoiding regressions. Creating a PoC exploit is challenging because vulnerability reports are informal and often incomplete, and because it requires a detailed understanding of how inputs passed to potentially vulnerable APIs may reach security-relevant sinks. In this paper, we present PoCGen, a novel approach to autonomously generate and validate PoC exploits for vulnerabilities in npm packages. The approach is the first to address this task by combining the complementary strengths of large language models (LLMs), e.g., to understand informal vulnerability reports, with static analysis, e.g., to identify taint paths, and dynamic analysis, e.g., to validate generated exploits. PoCGen successfully generates exploits for 77% of the vulnerabilities in the SecBench$.$js dataset. This success rate significantly outperforms a recent baseline (by 45 absolute percentage points), while imposing an average cost of only $0.02 per generated exploit. Moreover, PoCGen generates six successful exploits for recent real-world vulnerabilities, five of which are now included in their respective vulnerability reports.

cs.CR↗

arXiv2026

Natural Language-Focused Software Engineering via Code-Documentation Equivalence

Source code documentation is an integral part of software development and maintenance, as it helps in understanding the code and facilitates communication among developers. However, existing documentation is often incomplete, outdated, or inaccurate, which can lead to misunderstandings and errors. In the era of large language models (LLMs), which are being extensively used for software engineering tasks, the quality of documentation becomes even more critical, as documentation provides important context for the models. In this paper, we introduce the notion of documentation-to-code equivalence, a novel property that captures whether documentation accurately and completely describes the code it documents. We present a novel approach, called Documentary, to automatically generate equivalent documentation for a given code snippet. Our evaluation shows that Documentary can generate equivalent documentation for 53.4% of the evaluated function-level code snippets. To show the benefits of documentation-to-code equivalence, we describe and evaluate two software engineering tasks: code understanding and code editing. Our results show that documentation-to-code equivalence allows an LLM to predict the output of a function with 12.8--24.5% higher accuracy, when compared to human-written documentation and documentation generated by a baseline approach. Furthermore, human developers consider documentation generated by Documentary to be more useful for understanding and editing code than the original human-written documentation.

cs.SE↗

arXiv2026

Names Are All You Need: Effective and Safe Regression Test Selection for Python

Regression test selection reduces the cost of regression testing by executing only those tests affected by a code change. Despite extensive study of RTS in statically typed languages, achieving effective and safe RTS in Python is challenging. Python's dynamic typing makes precise call-graph construction difficult, which can cause call-graph-based RTS to miss affected tests. Python's eager importing mechanism, in contrast, renders file-level dependency analysis overly conservative. This paper presents NameRTS, the first Python RTS approach based on fine-grained dependency analysis. NameRTS models a Python program as a bipartite graph of code element nodes and name nodes, with edges capturing definitions and references. RTS is formulated as a reachability problem on this graph: a test is selected if any modified code element is reachable from the names used in that test. This design avoids call-graph construction, enabling a conservative analysis amenable to safety. To control dependency cascades introduced by coarse name matching, NameRTS applies two pruning strategies that leverage prior test executions and context information to refine name matching. To evaluate NameRTS, we construct the first Python RTS dataset with a ground truth indicating which test files are affected by each commit. We compare NameRTS with the best-performing baseline, BabelRTS, an RTS technique based on coarse file-level dependencies. On this benchmark, NameRTS skips 69.90% of test files on average, outperforming BabelRTS by 146.5%. It also reduces end-to-end testing time by 45.59%, yielding a 107.7% improvement over BabelRTS. In terms of safety, NameRTS selects all affected tests for 99.6% of commits, with only rare misses in exceptional cases. In contrast, BabelRTS is safe for 76.6% of commits. These results demonstrate the effectiveness of NameRTS, paving the way for more efficient regression testing in Python.

cs.SE↗

arXiv2026

FlyCatcher: Neural Inference of Runtime Checkers from Tests

Complex software systems often suffer from silent failures, i.e., violations of the intended semantics that do not cause explicit errors. A promising approach to detect such errors is to use system-specific runtime checkers that monitor the execution of a system and check for violations of the intended semantics. However, writing such checkers for a given software system is challenging and time-consuming, and hence, rarely done in practice. This work presents FlyCatcher, an automated approach to derive runtime checkers from existing tests, i.e., from a resource available for most software systems. The critical challenge of such an approach is to generalize the behavioral properties encoded in a test case to arbitrary executions of a system. FlyCatcher addresses this challenge through a combination of LLM-based synthesis, static analysis, and dynamic validation, which infers a checker that monitors specific method calls and asserts properties that should hold when they are called. The inferred checkers are stateful, i.e., they reason about the system's behavior by maintaining a shadow state that abstracts the actual system state as needed by the checker. Our evaluation applies FlyCatcher to 400 tests from four widely used, complex software systems. The approach infers 334 checkers, out of which 300 are found to be correct via cross-validation. Compared with a state-of-the-art approach, our approach infers 2.6x more correct checkers, which enables it to detect 5.2x more errors. By contributing to the automated inference of runtime checkers from tests, this work enables the broader adoption of runtime checking as a practical approach to detect silent failures in complex software systems.

cs.SE↗

arXiv2026

CodeCureAgent: Automatic Classification and Repair of Static Analysis Warnings

Static analysis tools are widely used to detect bugs, vulnerabilities, and code smells. Traditionally, developers must resolve these warnings manually. Because this process is tedious, developers sometimes ignore warnings, leading to an accumulation of warnings and a degradation of code quality. This paper presents CodeCureAgent, an approach that harnesses LLM-based agents to automatically analyze, classify, and repair static analysis warnings. Unlike previous work, our method does not follow a predetermined algorithm. Instead, we adopt an agentic framework that iteratively invokes tools to gather additional information from the codebase (e.g., via code search) and edit the codebase to resolve the warning. CodeCureAgent detects and suppresses false positives, while fixing true positives when identified. We equip CodeCureAgent with a three-step heuristic to approve patches: (1) build the project, (2) verify that the warning disappears without introducing new warnings, and (3) run the test suite. We evaluate CodeCureAgent on a dataset of 1,000 SonarQube warnings found in 106 Java projects and covering 291 distinct rules. Our approach produces plausible fixes for 96.8% of the warnings, outperforming state-of-the-art baseline approaches by 29.2%-34.0% in plausible-fix rate. Manual inspection of 291 cases reveals a correct-fix rate of 86.3%, showing that CodeCureAgent can reliably repair static analysis warnings. The approach incurs LLM costs of about 2.9 cents (USD) and an end-to-end processing time of about four minutes per warning. We envision CodeCureAgent helping to clean existing codebases and being integrated into CI/CD pipelines to prevent the accumulation of static analysis warnings.

cs.SE↗

arXiv2026

RippleGUItester: Change-Aware Exploratory Testing

Software systems evolve continuously through frequent code changes, yet such changes often introduce unintended bugs despite extensive testing and code review. Existing testing approaches are largely constrained to predefined execution paths or rely on unguided exploration, leaving many change-induced issues undetected. To address this challenge, we present RippleGUItester, a change-driven testing system that treats a code change as the epicenter of a ripple effect and explores its broader, user-visible impacts via the GUI. Given a code change, RippleGUItester performs LLM-based change-impact analysis to generate and enrich realistic test scenarios, executes these scenarios on both pre-change and post-change versions of the system, and applies differential analysis to identify behavioral differences. Crucially, RippleGUItester employs multimodal bug detection, comparing visual GUI changes and interpreting them in the context of natural-language change intents to distinguish unintended bugs from intended behavioral updates. We evaluate our approach on hundreds of real-world code changes across four widely used software systems: Firefox, Zettlr, JabRef, and Godot. Our results show that the proposed approach uncovers bugs introduced by code changes that were missed by existing test suites, CI pipelines, and code review. In total, we identify 26 previously unknown bugs that still exist in the latest versions of the evaluated systems. After reporting, 16 bugs have been fixed, 2 have been confirmed, 6 are still under discussion, and 2 were marked as intended. We envision RippleGUItester being applied before or shortly after a code change is merged, enabling earlier detection of regressions.

cs.SE↗