arXiv subjects

Eduardo Almeida

Publications and source records attributed to Eduardo Almeida.

6 recordsLinked to original sources

arXiv2026

An Empirical Study of Foundation Models for Variability-Induced Compilation Errors in Configurable C Code

In configurable systems, conditional compilation can hide compilation errors under untested feature combinations. We investigate foundation models for detecting such errors and, in a controlled setting, restoring compilability in configurable C code. Study I evaluates GPT-OSS-20B on 5,000 synthetic snippets generated by ChatGPT-5.2 from 30 curated seeds and exhaustively compiled under all Boolean feature assignments; it also compares TypeChef and evaluates Gemini 3.6 Flash on a stratified sample. GPT-OSS-20B achieved 84.7% micro-precision and 52.1% micro-recall for affected configurations. Coverage depended on reporting style: presence conditions covered 99.4% of failing configurations, whereas explicit enumerations covered 29.5% under a prompt requesting only a minimal justifiable set. GPT-OSS-20B restored compilability for 1,930 of 2,665 faulty snippets (72.4%), while Gemini 3.6 Flash did so for 182 of 190 sampled faulty snippets (95.8%). A paired counterfactual audit found no evidence that an identified label-correlated #define property materially influenced GPT-OSS-20B's predictions. Study II evaluates Codex-GPT5.5 on 100 faulty file-level subjects from five mature configurable systems and reports target-fault-aligned problems in 94 subjects, including four of five historical bugs. Overall, foundation models can support localized detection, explanation, and triage, but should complement compiler-based and variability-aware analyses; compiler acceptance does not establish semantic correctness

cs.SE↗

arXiv2025

Agentic LMs: Hunting Down Test Smells

Test smells reduce test suite reliability and complicate maintenance. While many methods detect test smells, few support automated removal, and most rely on static analysis or machine learning. This study evaluates models with relatively small parameter counts - Llama-3.2-3B, Gemma-2-9B, DeepSeek-R1-14B, and Phi-4-14B - for their ability to detect and refactor test smells using agent-based workflows. We assess workflows with one, two, and four agents over 150 instances of 5 common smells from real-world Java projects. Our approach generalizes to Python, Golang, and JavaScript. All models detected nearly all instances, with Phi-4-14B achieving the best refactoring accuracy (pass@5 of 75.3%). Phi-4-14B with four-agents performed within 5% of proprietary LLMs (single-agent). Multi-agent setups outperformed single-agent ones in three of five smell types, though for Assertion Roulette, one agent sufficed. We submitted pull requests with Phi-4-14B-generated code to open-source projects and six were merged.

cs.SE↗

arXiv2025

Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks

A growing variety of prompt engineering techniques has been proposed for Large Language Models (LLMs), yet systematic evaluation of each technique on individual software engineering (SE) tasks remains underexplored. In this study, we present a systematic evaluation of 14 established prompt techniques across 10 SE tasks using four LLM models. As identified in the prior literature, the selected prompting techniques span six core dimensions (Zero-Shot, Few-Shot, Thought Generation, Ensembling, Self-Criticism, and Decomposition). They are evaluated on tasks such as code generation, bug fixing, and code-oriented question answering, to name a few. Our results show which prompting techniques are most effective for SE tasks requiring complex logic and intensive reasoning versus those that rely more on contextual understanding and example-driven scenarios. We also analyze correlations between the linguistic characteristics of prompts and the factors that contribute to the effectiveness of prompting techniques in enhancing performance on SE tasks. Additionally, we report the time and token consumption for each prompting technique when applied to a specific task and model, offering guidance for practitioners in selecting the optimal prompting technique for their use cases.

cs.SE↗

arXiv2025

Prompting in the Wild: An Empirical Study of Prompt Evolution in Software Repositories

The adoption of Large Language Models (LLMs) is reshaping software development as developers integrate these LLMs into their applications. In such applications, prompts serve as the primary means of interacting with LLMs. Despite the widespread use of LLM-integrated applications, there is limited understanding of how developers manage and evolve prompts. This study presents the first empirical analysis of prompt evolution in LLM-integrated software development. We analyzed 1,262 prompt changes across 243 GitHub repositories to investigate the patterns and frequencies of prompt changes, their relationship with code changes, documentation practices, and their impact on system behavior. Our findings show that developers primarily evolve prompts through additions and modifications, with most changes occurring during feature development. We identified key challenges in prompt engineering: only 21.9% of prompt changes are documented in commit messages, changes can introduce logical inconsistencies, and misalignment often occurs between prompt changes and LLM responses. These insights emphasize the need for specialized testing frameworks, automated validation tools, and improved documentation practices to enhance the reliability of LLM-integrated applications.

cs.SE↗

arXiv2025

Test Smell: A Parasitic Energy Consumer in Software Testing

Traditionally, energy efficiency research has focused on reducing energy consumption at the hardware level and, more recently, in the design and coding phases of the software development life cycle. However, software testing's impact on energy consumption did not receive attention from the research community. Specifically, how test code design quality and test smell (e.g., sub-optimal design and bad practices in test code) impact energy consumption has not been investigated yet. This study examined 12 Apache projects to analyze the association between test smell and its effects on energy consumption in software testing. We conducted a mixed-method empirical analysis from two dimensions; software (data mining in Apache projects) and developers' views (a survey of 62 software practitioners). Our findings show that: 1) test smell is associated with energy consumption in software testing. Specifically smelly part of a test case consumes 10.92\% more energy compared to the non-smelly part. 2) certain test smells are more energy-hungry than others, 3) refactored test cases tend to consume less energy than their smelly counterparts, and 4) most developers lack knowledge about test smells' impact on energy consumption. We conclude the paper with several observations that can direct future research and developments.

cs.SE↗

arXiv2020

Examining user reviews of conversational systems: a case study of Alexa skills

Conversational systems use spoken language to interact with their users. Although conversational systems, such as Amazon Alexa, are becoming common and afford interesting functionalities, there is little known about the issues users of these systems face. In this paper, we study user reviews of more than 2,800 Alexa skills to understand the characteristics of the reviews and issues that are raised in them. Our results suggest that most skills receive less than 50 reviews. Our qualitative study of user reviews using open coding resulted in identifying 16 types of issues in the user reviews. Issues related to the content, integration with online services and devices, error, and regression are top issues raised by the users. Our results also indicate differences in volume and types of complaints by users when compared with more traditional mobile applications. We discuss the implication of our results for practitioners and researchers.

cs.SE↗