arXiv subjects

Khashayar Etemadi

Publications and source records attributed to Khashayar Etemadi.

14 recordsLinked to original sources

arXiv2026

JETO-Bench: A Reproducible Benchmark for Execution Time Improvement Patches in Java

Automated fixing of performance issues is gaining attention, but existing benchmarks of execution time improvement patches (ETIPs) target Python, C++, or .NET and are fixed datasets that cannot be extended under user-defined configurations. We present JETO-Mine, the first configurable and reusable tool for automatically creating reproducible benchmarks of ETIPs in real-world Java projects. Java is challenging because just-in-time compilation and garbage collection make execution measurements volatile. JETO-Mine employs a three-phase pipeline: static analysis identifies ETIPs from GitHub repositories using user-defined filters and an LLM-based issue classifier, dynamic analysis wraps identified ETIPs in Docker images and performs statistical testing for evidence of execution time improvement, and an evaluation harness supports quantitative assessment of both patch and test generation tools. Using JETO-Mine, we build JETO-Bench, a benchmark of 660 identified and 91 manually verified executable ETIPs from 174 Java repositories, mined from nearly 1.8 million commits across 11 years. Running OpenHands with GPT-5-mini fixes 14.3% (13/91) of issues, aligning with prior work on other languages. Our results also reveal that open-source Java projects largely lack tests demonstrating execution time improvements, presenting an opportunity for future research in test generation. JETO-Mine and JETO-Bench are publicly available at https://github.com/khesoem/JETO-Bench.

cs.SE↗

arXiv2026

CppPerf: An Automated Pipeline and Dataset for Performance-Improving C++ Commits

Recent progress in automated repair of performance bugs demands realistic, executable benchmarks. However, existing C++ performance benchmarks are largely built from competitive programming submissions, and recent real-world benchmarks predominantly target Python and .NET. To fill this gap, we present CppPerf-Mine, a configurable pipeline that mines execution-time-improving patches from open-source C++ repositories on GitHub by combining structural commit filtering, an LLM-based commit classifier, and a containerized build & test stage that produces fully reproducible Docker images for each patch. Using CppPerf-Mine, we build CppPerf-DB, a benchmark comprising 347 manually verified patches from 42 mature C++ repositories, 39% of which are multi-file, enabling the evaluation of repository-level repair tools. In our preliminary study, OpenHands correctly fixes only 13.5% of the patches in CppPerf-DB, confirming that real-world C++ performance repair remains an open challenge. CppPerf-Mine and CppPerf-DB are open-source and publicly available at: https://doi.org/10.5281/zenodo.20097425. In addition, a demonstration video is available at: https://www.youtube.com/watch?v=nixlupIgSdM.

cs.SE↗

arXiv2025

Mokav: Execution-driven Differential Testing with LLMs

It is essential to detect functional differences between programs in various software engineering tasks, such as automated program repair, mutation testing, and code refactoring. The problem of detecting functional differences between two programs can be reduced to searching for a difference exposing test (DET): a test input that results in different outputs on the subject programs. In this paper, we propose Mokav, a novel execution-driven tool that leverages LLMs to generate DETs. Mokav takes two versions of a program (P and Q) and an example test input. When successful, Mokav generates a valid DET, a test input that leads to provably different outputs on P and Q. Mokav iteratively prompts an LLM with a specialized prompt to generate new test inputs. At each iteration, Mokav provides execution-based feedback from previously generated tests until the LLM produces a DET. We evaluate Mokav on 1535 pairs of Python programs collected from the Codeforces competition platform and 32 pairs of programs from the QuixBugs dataset. Our experiments show that Mokav outperforms the state-of-the-art, Pynguin and Differential Prompting, by a large margin. Mokav can generate DETs for 81.7% (1,255/1535) of the program pairs in our benchmark (versus 4.9% for Pynguin and 37.3% for Differential Prompting). We demonstrate that the iterative and execution-driven feedback components of the system contribute to its high effectiveness.

cs.SE↗

arXiv2025

LLM-based Property-based Test Generation for Guardrailing Cyber-Physical Systems

Cyber-physical systems (CPSs) are complex systems that integrate physical, computational, and communication subsystems. The heterogeneous nature of these systems makes their safety assurance challenging. In this paper, we propose a novel automated approach for guardrailing cyber-physical systems using property-based tests (PBTs) generated by Large Language Models (LLMs). Our approach employs an LLM to extract properties from the code and documentation of CPSs. Next, we use the LLM to generate PBTs that verify the extracted properties on the CPS. The generated PBTs have two uses. First, they are used to test the CPS before it is deployed, i.e., at design time. Secondly, these PBTs can be used after deployment, i.e., at run time, to monitor the behavior of the system and guardrail it against unsafe states. We implement our approach in ChekProp and conduct preliminary experiments to evaluate the generated PBTs in terms of their relevance (how well they match manually crafted properties), executability (how many run with minimal manual modification), and effectiveness (coverage of the input space partitions). The results of our experiments and evaluation demonstrate a promising path forward for creating guardrails for CPSs using LLM-generated property-based tests.

cs.SE↗

arXiv2025

VeCoGen: Automating Generation of Formally Verified C Code with Large Language Models

Large language models have demonstrated impressive capabilities in generating code, yet they often produce programs with flaws or deviations from intended behavior, limiting their suitability for safety-critical applications. To address this limitation, this paper introduces VECOGEN, a novel tool that combines large language models with formal verification to automate the generation of formally verified C programs. VECOGEN takes a formal specification in ANSI/ISO C Specification Language, a natural language specification, and a set of test cases to attempt to generate a verified program. This program-generation process consists of two steps. First, VECOGEN generates an initial set of candidate programs. Secondly, the tool iteratively improves on previously generated candidates. If a candidate program meets the formal specification, then we are sure the program is correct. We evaluate VECOGEN on 15 problems presented in Codeforces competitions. On these problems, VECOGEN solves 13 problems. This work shows the potential of combining large language models with formal verification to automate program generation.

cs.SE↗

arXiv2024

Generative AI to Generate Test Data Generators

Generating fake data is an essential dimension of modern software testing, as demonstrated by the number and significance of data faking libraries. Yet, developers of faking libraries cannot keep up with the wide range of data to be generated for different natural languages and domains. In this paper, we assess the ability of generative AI for generating test data in different domains. We design three types of prompts for Large Language Models (LLMs), which perform test data generation tasks at different levels of integrability: 1) raw test data generation, 2) synthesizing programs in a specific language that generate useful test data, and 3) producing programs that use state-of-the-art faker libraries. We evaluate our approach by prompting LLMs to generate test data for 11 domains. The results show that LLMs can successfully generate realistic test data generators in a wide range of domains at all three levels of integrability.

cs.SE↗

arXiv2024

CigaR: Cost-efficient Program Repair with LLMs

Large language models (LLM) have proven to be effective at automated program repair (APR). However, using LLMs can be costly, with companies invoicing users by the number of tokens. In this paper, we propose CigaR, the first LLM-based APR tool that focuses on minimizing the repair cost. CigaR works in two major steps: generating a first plausible patch and multiplying plausible patches. CigaR optimizes the prompts and the prompt setting to maximize the information given to LLMs using the smallest possible number of tokens. Our experiments on 429 bugs from the widely used Defects4J and HumanEval-Java datasets shows that CigaR reduces the token cost by 73%. On average, CigaR spends 127k tokens per bug while the baseline uses 467k tokens per bug. On the subset of bugs that are fixed by both, CigaR spends 20k per bug while the baseline uses 608k tokens, a cost saving of 96%. Our extensive experiments show that CigaR is a cost-effective LLM-based program repair tool that uses a low number of tokens to automatically generate patches.

cs.SE↗

arXiv2023

Augmenting Diffs With Runtime Information

Source code diffs are used on a daily basis as part of code review, inspection, and auditing. To facilitate understanding, they are typically accompanied by explanations that describe the essence of what is changed in the program. As manually crafting high-quality explanations is a cumbersome task, researchers have proposed automatic techniques to generate code diff explanations. Existing explanation generation methods solely focus on static analysis, i.e., they do not take advantage of runtime information to explain code changes. In this paper, we propose Collector-Sahab, a novel tool that augments code diffs with runtime difference information. Collector-Sahab compares the program states of the original (old) and patched (new) versions of a program to find unique variable values. Then, Collector-Sahab adds this novel runtime information to the source code diff as shown, for instance, in code reviewing systems. As an evaluation, we run Collector-Sahab on 584 code diffs for Defects4J bugs and find it successfully augments the code diff for 95% (555/584) of them. We also perform a user study and ask eight participants to score the augmented code diffs generated by Collector-Sahab. Per this user study, we conclude that developers find the idea of adding runtime data to code diffs promising and useful. Overall, our experiments show the effectiveness and usefulness of Collector-Sahab in augmenting code diffs with runtime difference information. Publicly-available repository: https://github.com/ASSERT-KTH/collector-sahab.

cs.SE↗

arXiv2022

Estimating the Potential of Program Repair Search Spaces with Commit Analysis

The most natural method for evaluating program repair systems is to run them on bug datasets, such as Defects4J. Yet, using this evaluation technique on arbitrary real-world programs requires heavy configuration. In this paper, we propose a purely static method to evaluate the potential of the search space of repair approaches. This new method enables researchers and practitioners to encode the search spaces of repair approaches and select potentially useful ones without struggling with tool configuration and execution. We encode the search spaces by specifying the repair strategies they employ. Next, we use the specifications to check whether past commits lie in repair search spaces. For a repair approach, including many human-written past commits in its search space indicates its potential to generate useful patches. We implement our evaluation method in LighteR. LighteR gets a Git repository and outputs a list of commits whose source code changes lie in repair search spaces. We run LighteR on 55,309 commits from the history of 72 Github repositories with and show that LighteR's precision and recall are 77% and 92%, respectively. Overall, our experiments show that our novel method is both lightweight and effective to study the search space of program repair approaches.

cs.SE↗

arXiv2022

Sorald: Automatic Patch Suggestions for SonarQube Static Analysis Violations

Previous work has shown that early resolution of issues detected by static code analyzers can prevent major costs later on. However, developers often ignore such issues for two main reasons. First, many issues should be interpreted to determine if they correspond to actual flaws in the program. Second, static analyzers often do not present the issues in a way that is actionable. To address these problems, we present Sorald: a novel system that devise metaprogramming templates to transform the abstract syntax trees of programs and suggest fixes for static analysis warnings. Thus, the burden on the developer is reduced from interpreting and fixing static issues, to inspecting and approving full fledged solutions. Sorald fixes violations of 10 rules from SonarJava, one of the most widely used static analyzers for Java. We evaluate Sorald on a dataset of 161 popular repositories on Github. Our analysis shows the effectiveness of Sorald as it fixes 65% (852/1,307) of the violations that meets the repair preconditions. Overall, our experiments show it is possible to automatically fix notable violations of the static analysis rules produced by the state-of-the-art static analyzer SonarJava.

cs.SE↗

arXiv2021

A Software-Repair Robot based on Continual Learning

Software bugs are common and correcting them accounts for a significant part of costs in the software development and maintenance process. This calls for automatic techniques to deal with them. One promising direction towards this goal is gaining repair knowledge from historical bug fixing examples. Retrieving insights from software development history is particularly appealing with the constant progress of machine learning paradigms and skyrocketing `big' bug fixing data generated through Continuous Integration (CI). In this paper, we present R-Hero, a novel software repair bot that applies continual learning to acquire bug fixing strategies from continuous streams of source code changes, implemented for the single development platform Github/Travis CI. We describe R-Hero, our novel system for learning how to fix bugs based on continual training, and we uncover initial successes as well as novel research challenges for the community.

cs.SE↗

arXiv2020

On the Relevance of Cross-project Learning with Nearest Neighbours for Commit Message Generation

Commit messages play an important role in software maintenance and evolution. Nonetheless, developers often do not produce high-quality messages. A number of commit message generation methods have been proposed in recent years to address this problem. Some of these methods are based on neural machine translation (NMT) techniques. Studies show that the nearest neighbor algorithm (NNGen) outperforms existing NMT-based methods, although NNGen is simpler and faster than NMT. In this paper, we show that NNGen does not take advantage of cross-project learning in the majority of the cases. We also show that there is an even simpler and faster variation of the existing NNGen method which outperforms it in terms of the BLEU_4 score without using cross-project learning.

cs.SE↗

arXiv2020

Incorporating fault-proneness estimations into coverage-based test case prioritization methods

Context: During the development process of a software program, regression testing is used to ensure that the correct behavior of the software is retained after updates to the source code. This regression testing becomes costly over time as the number of test cases increases and it makes sense to prioritize test cases in order to execute fault-detecting test cases as soon as possible. There are many coverage-based test case prioritization (TCP) methods that only use the code coverage data to prioritize test cases. By incorporating the fault-proneness estimations of code units into the coverage-based TCP methods, we can improve such techniques. Objective: In this paper, we aim to propose an approach which improves coverage-based TCP methods by considering the fault-proneness distribution over code units. Further, we present the results of an empirical study that shows using our proposed approach significantly improves the additional strategy, which is a widely used coverage-based TCP method. Method: The approach presented in this study uses the bug history of the software in order to introduce a defect prediction method to learn a neural network model. This model is then used to estimate fault-proneness of each area of the source code and then the estimations are incorporated into coverage-based TCP methods. Our proposed approach is a general idea that can be applied to many coverage-based methods, such as the additional and total TCP methods. Results: The proposed methods are evaluated on datasets collected from the development history of five real-world projects including 357 versions in total. The experiments show that using an appropriate bug history can improve coverage-based TCP methods.

cs.SE↗

arXiv2015

Even and Odd Cycles Passing a Given Edge or a Vertex

In this paper we provide some sufficient conditions for the existence of an odd or even cycle that passing a given vertex or an edge in $2$-connected or $2$-edge connected graphs. We provide some similar conditions for the existence of an odd or even circuit that passing a given vertex or an edge in 2-edge connected graphs. We show that if $G$ is a $2$-connected $k$-regular graph, $k \geq 3$, then every edge of $G$ is contained in an even cycle. We also prove that in a $2$-edge connected graph, if a vertex has odd degree, then there is an even cycle containing this vertex.

cs.DM↗