arXiv subjects

Diego Elias Costa

Publications and source records attributed to Diego Elias Costa.

At least 19 recordsLinked to original sources

arXiv2026

Evaluating Change Point Detection Methods for Software Performance Regression Analysis

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

cs.SE↗

arXiv2026

BreakGuard: Towards Detecting Dependency Breaking Changes with LLM-Generated Tests

Open-source libraries play an important role in software development by providing reusable features that expedite the development process. As libraries evolve, they release new versions that add features, fix bugs, or apply security patches. In this process, they may break the contract established with their clients by introducing breaking changes (BCs) that alter the runtime behavior and break client applications. Client-side test suites often fail to detect these BCs because of limited library coverage that does not exercise all library methods used in the client's codebase. We propose BreakGuard, an approach that generates a test suite to detect breaking changes in clients. BreakGuard statically extracts every client method (focal method) that invokes the target library method (call site), then generates tests per focal method. A test detects a BC if it passes on the pre-breaking version and fails on the breaking version. We evaluate our approach on 89 real-world breaking changes from the BUMP dataset, using 3 LLMs (GPT4o, Qwen3-coder-480B, GPT-OSS-120B) and three context levels: minimal, method, and class. Using the best-performing configuration, BreakGuard detects 30.3% of breaking changes (27 of 89) at a mean cost of roughly $0.90 USD per detected breaking change. We successfully detected BCs from different library categories (e.g., JSON libraries, logging, parsing), but we find LLM-generated tests to be more reliable for detecting crash-type breaking changes as opposed to behavioural BCs.

cs.SE↗

arXiv2026

Rethinking Training Data for Generating Code Review Comments

Generating code review comments has become a prominent research direction in automated code review, commonly formulated as a text generation task over diff-comment pairs. Despite advances in learning-based approaches, generated review comments are often generic, weakly grounded, or non-actionable. Recent studies have also shown that review comment datasets contain noisy or unsuitable training instances, motivating LLM-based dataset cleaning approaches. In this paper, we argue that problematic training instances are not homogeneous and that some limitations stem from deeper issues in the task formulation itself. Through an empirical inspection of a widely used review comment dataset, we identify misaligned training pairs: instances where the relationship between the code change and the review comment does not provide a reliable learning signal for generating actionable review feedback from localized inputs. We derive a taxonomy of misalignment capturing three recurring sources: semantic ambiguity, lack of actionability, and context dependence. We further explore whether incorporating this taxonomy into LLM-based filtering improves the identification of problematic training instances, observing that detecting misaligned training pairs remains challenging. Based on these observations, we argue that improving review comment generation requires more than dataset cleaning alone, motivating explicit validity criteria, richer contextual inputs, and evaluation practices aligned with review intent and actionability.

cs.SE↗

arXiv2026

What's Inside a GitHub Repository? An Empirical Study on the Contents of 10K Projects

GitHub is the largest code hosting platform, with millions of repositories spanning multiple technologies. Despite this, little is known about the actual contents of GitHub's repositories in the wild. This paper presents an initial empirical analysis to better understand the contents of real-world GitHub repositories. We analyze the files, directories, and extensions present in 10,000 GitHub repositories, as well as their evolution over ten years. Our results show major changes in GitHub over the last decade: (1) the consolidation of README.md, .gitignore, and LICENSE as standard artifacts; (2) the rise of GitHub Actions as the dominant CI/CD platform; (3) the growth of configuration formats such as TOML, YAML, and JSON, alongside a decline in XML; (4) new trends, such as the growth of Dockerfile; and (5) emerging content related to LLMs and generative AI (e.g., AGENTS.md and CLAUDE.md). Based on our findings, we discuss implications, including that open source is not only evolving organically but also increasingly guided by GitHub's standards, the rise and fall of technologies, and the potential support for mining software repository studies.

cs.SE↗

arXiv2026

Retrieval-Oriented Code Representations in Agentic Bug Localization

LLM-based agents are increasingly being used to support software development, yet their performance in repository-level tasks depends on retrieving the right code context. Existing studies have explored file-level localization using traditional information retrieval over file paths and raw source code. However, the role of textual code representations in retrieval and localization remains underexplored. We study file-level bug localization as a representation-driven retrieval problem. Across the Long Code Arena (LCA) and SWE-bench Verified (SWE) datasets, we compare five code representations: file paths, raw source code, and three LLM-generated textual representations. Our experiments include lexical, semantic, and LLM-based retrieval, followed by LLM-based post-retrieval ranking. We quantify the cost incurred by a representation through the representation footprint. We find that the choice of code representation affects both localization effectiveness and cost. Role-aware summaries outperform file-path representations by up to 40% Hit@5 while requiring a representation footprint 10.4 to 20.9x smaller than raw source code. Combining complementary representation results and ranking retrieved candidates with an LLM provides further gains of up to 31.9% and 42.0%, respectively. Overall, role-aware summaries provide the best cost-effectiveness trade-off, while raw source code offers effectiveness in some settings at a significantly higher cost. A case study with Agentless reveals the utility of our techniques within a well-known pipeline, reaching 94% Hit@6 on file localization (+4.7% against the baseline). Our findings suggest that code representation should be treated as a first-class design choice in agentic localization pipelines, guided by pipeline stage and cost-accuracy requirements.

cs.SE↗

arXiv2026

Beyond Compliance: A Large Scale Study on the Completeness and Consistency of the GitHub SBOMs

Modern software development relies heavily on open-source components. Reusing components accelerates innovation but increases exposure to supply-chain attacks exploiting known vulnerabilities. Software Bills of Materials (SBOMs) improve software supply chain transparency by enumerating components, their versions, and their provenance. GitHub, the largest open-source development hosting platform, now automatically generates SBOMs for repositories, providing valuable metadata for risk assessment. Yet, it is unclear whether GitHub SBOMs can serve as a reliable source for vulnerability and license analysis, and how incomplete or inconsistent metadata may affect different programming ecosystems. To address this, we conduct a large-scale analysis of 10,000 GitHub repositories across ten programming language ecosystems, evaluating GitHub SBOMs against three other popular SBOM generators: Syft, Trivy, and the Microsoft SBOM Tool. Our study finds a lack of NTIA compliance in GitHub SBOMs, though core metadata is consistently present. We also find that component version and license information availability is highly dependent on the programming ecosystem. Compared with the other three tools, GitHub yields results similar to the Microsoft SBOM Tool and often outperforms Syft and Trivy in providing version and license information. Finally, we discuss potential shortcomings of the GitHub SBOM Tool, directly related to how each ecosystem manages its dependencies.

cs.SE↗

arXiv2026

Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla

Software performance regressions can have significant business consequences, making automated detection a critical component of modern continuous integration pipelines. At Mozilla, performance anomaly detection is handled by Perfherder, Mozilla's performance engineering management system that relies on a Student's T-test-based approach to flag regressions across hundreds of daily code changes. However, our preliminary analysis of one year of Mozilla performance data reveals that 12.5% of generated alert groups are false positives, while approximately 6.8% of them contain regressions missed by the automated system. This paper presents an empirical study evaluating 25 change-point detection (CPD) methods and 15 ensemble approaches as alternatives to Mozilla's current method. We construct a ground-truth dataset of 174 performance time series manually annotated by eleven Mozilla performance engineers, representing one of the first practitioner-annotated CPD benchmarks for performance engineering. Our results show that while offline and hybrid CPD methods improve recall over Mozilla's method, they do so at a high cost to precision. Ensemble voting strategies alleviate this trade-off and offer more consistent performance, resulting in 11% improvement in the F1-score. We validate the experimental results through a practitioner survey and report on lessons learned from integrating the best methods into Mozilla's performance engineering system.

cs.SE↗

arXiv2026

LLM Based Web Accessibility Repair: An Empirical Study of Detection, Remediation, and Cost

Ensuring web accessibility at scale remains challenging because rule-based tools provide limited coverage while manual remediation is costly and error-prone. This paper evaluates large language model based agents, specifically Kimi K2.5, for automated accessibility detection and repair compared with rule-based approaches. For detection, the LLM achieves performance comparable to rule-based tools, with F1 around 0.65, strong semantic understanding with F1 of 0.83, but lower reliability for syntactic and layout-related violations. For remediation, LLM-generated fixes are syntactically valid in over 99.7 percent of cases and improve accessibility compliance in 80.2 percent of instances, reducing violations from 3.98 to 1.7 per file. However, fewer than 26 percent of cases are fully resolved, and about 30 percent of patches introduce structural changes. We also find that iterative agent-based refinement increases computational cost by 52 percent and API usage by 1.64 times without improving remediation outcomes. These findings indicate that while LLMs are effective for partial accessibility repair, they are insufficient for complete and reliable remediation. Scalable accessibility solutions require hybrid approaches that combine LLM capabilities with rule-based validation and constraint-aware correction mechanisms.

cs.SE↗

arXiv2025

Towards Supporting Open Source Library Maintainers with Community-Based Analytics

Open-source software (OSS) is a pillar of modern software development. Its success depends on the dedication of maintainers who work constantly to keep their libraries stable, adapt to changing needs, and support a growing community. Yet, they receive little to no continuous feedback on how the projects that rely on their libraries actually use their APIs. We believe that gaining these insights can help maintainers make better decisions, such as refining testing strategies, understanding the impact of changes, and guiding the evolution of their libraries more effectively. We propose the use of community-based analytics to analyze how an OSS library is used across its dependent ecosystem. We conduct an empirical study of 10 popular Java libraries and each with their respective dependent ecosystem of 50 projects. Our results reveal that while library developers offer a wide range of API methods, only 16% on average are actively used by their dependent ecosystem. Moreover, only 74% of the used API methods are partially or fully covered by their library test suite. We propose two metrics to help developers evaluate their test suite according to the APIs used by their community, and we conduct a survey on open-source practitioners to assess the practical value of these insights in guiding maintenance decisions.

cs.SE↗

arXiv2025

ADPerf: Investigating and Testing Performance in Autonomous Driving Systems

Obstacle detection is crucial to the operation of autonomous driving systems, which rely on multiple sensors, such as cameras and LiDARs, combined with code logic and deep learning models to detect obstacles for time-sensitive decisions. Consequently, obstacle detection latency is critical to the safety and effectiveness of autonomous driving systems. However, the latency of the obstacle detection module and its resilience to various changes in the LiDAR point cloud data are not yet fully understood. In this work, we present the first comprehensive investigation on measuring and modeling the performance of the obstacle detection modules in two industry-grade autonomous driving systems, i.e., Apollo and Autoware. Learning from this investigation, we introduce ADPerf, a tool that aims to generate realistic point cloud data test cases that can expose increased detection latency. Increasing latency decreases the availability of the detected obstacles and stresses the capabilities of subsequent modules in autonomous driving systems, i.e., the modules may be negatively impacted by the increased latency in obstacle detection. We applied ADPerf to stress-test the performance of widely used 3D obstacle detection modules in autonomous driving systems, as well as the propagation of such tests on trajectory prediction modules. Our evaluation highlights the need to conduct performance testing of obstacle detection components, especially 3D obstacle detection, as they can be a major bottleneck to increased latency of the autonomous driving system. Such an adverse outcome will also further propagate to other modules, reducing the overall reliability of autonomous driving systems.

cs.SE↗

arXiv2025

Beyond More Context: How Granularity and Order Drive Code Completion Quality

Context plays an important role in the quality of code completion, as Large Language Models (LLMs) require sufficient and relevant information to assist developers in code generation tasks. However, composing a relevant context for code completion poses challenges in large repositories: First, the limited context length of LLMs makes it impractical to include all repository files. Second, the quality of generated code is highly sensitive to noisy or irrelevant context. In this paper, we present our approach for the ASE 2025 Context Collection Challenge. The challenge entails outperforming JetBrains baselines by designing effective retrieval and context collection strategies. We develop and evaluate a series of experiments that involve retrieval strategies at both the file and chunk levels. We focus our initial experiments on examining the impact of context size and file ordering on LLM performance. Our results show that the amount and order of context can significantly influence the performance of the models. We introduce chunk-based retrieval using static analysis, achieving a 6% improvement over our best file-retrieval strategy and a 16% improvement over the no-context baseline for Python in the initial phase of the competition. Our results highlight the importance of retrieval granularity, ordering and hybrid strategies in developing effective context collection pipelines for real-world development scenarios.

cs.SE↗

arXiv2025

Exploring the Lifecycle and Maintenance Practices of Pre-Trained Models in Open-Source Software Repositories

Pre-trained models (PTMs) are becoming a common component in open-source software (OSS) development, yet their roles, maintenance practices, and lifecycle challenges remain underexplored. This report presents a plan for an exploratory study to investigate how PTMs are utilized, maintained, and tested in OSS projects, focusing on models hosted on platforms like Hugging Face and PyTorch Hub. We plan to explore how PTMs are used in open-source software projects and their related maintenance practices by mining software repositories that use PTMs and analyzing their code-base, historical data, and reported issues. This study aims to provide actionable insights into improving the use and sustainability of PTM in open-source projects and a step towards a foundation for advancing software engineering practices in the context of model dependencies.

cs.SE↗

arXiv2025

Towards Assessing Deep Learning Test Input Generators

Deep Learning (DL) systems are increasingly deployed in safety-critical applications, yet they remain vulnerable to robustness issues that can lead to significant failures. While numerous Test Input Generators (TIGs) have been developed to evaluate DL robustness, a comprehensive assessment of their effectiveness across different dimensions is still lacking. This paper presents a comprehensive assessment of four state-of-the-art TIGs--DeepHunter, DeepFault, AdvGAN, and SinVAD--across multiple critical aspects: fault-revealing capability, naturalness, diversity, and efficiency. Our empirical study leverages three pre-trained models (LeNet-5, VGG16, and EfficientNetB3) on datasets of varying complexity (MNIST, CIFAR-10, and ImageNet-1K) to evaluate TIG performance. Our findings reveal important trade-offs in robustness revealing capability, variation in test case generation, and computational efficiency across TIGs. The results also show that TIG performance varies significantly with dataset complexity, as tools that perform well on simpler datasets may struggle with more complex ones. In contrast, others maintain steadier performance or better scalability. This paper offers practical guidance for selecting appropriate TIGs aligned with specific objectives and dataset characteristics. Nonetheless, more work is needed to address TIG limitations and advance TIGs for real-world, safety-critical systems.

cs.LG↗

arXiv2025

A Dataset of Performance Measurements and Alerts from Mozilla (Data Artifact)

Performance regressions in software systems can lead to significant financial losses and degraded user satisfaction, making their early detection and mitigation critical. Despite the importance of practices that capture performance regressions early, there is a lack of publicly available datasets that comprehensively capture real-world performance measurements, expert-validated alerts, and associated metadata such as bugs and testing conditions. To address this gap, we introduce a unique dataset to support various research studies in performance engineering, anomaly detection, and machine learning. This dataset was collected from Mozilla Firefox's performance testing infrastructure and comprises 5,655 performance time series, 17,989 performance alerts, and detailed annotations of resulting bugs collected from May 2023 to May 2024. By publishing this dataset, we provide researchers with an invaluable resource for studying performance trends, developing novel change point detection methods, and advancing performance regression analysis across diverse platforms and testing environments. The dataset is available at https://doi.org/10.5281/zenodo.14642238

cs.PF↗

arXiv2024

Opportunities and Security Risks of Technical Leverage: A Replication Study on the NPM Ecosystem

To comply with high productivity demands, software developers reuse free open-source software (FOSS) code to avoid reinventing the wheel when incorporating software features. The reliance on FOSS reuse has been shown to improve productivity and the quality of delivered software; however, reusing FOSS comes at the risk of exposing software projects to public vulnerabilities. Massacci and Pashchenko have explored this trade-off in the Java ecosystem through the lens of technical leverage: the ratio of code borrowed from FOSS over the code developed by project maintainers. In this paper, we replicate the work of Massacci and Pashchenko and we expand the analysis to include level-1 transitive dependencies to study technical leverage in the fastest-growing NPM ecosystem. We investigated 14,042 NPM library releases and found that both opportunities and risks of technical leverage are magnified in the NPM ecosystem. Small-medium libraries leverage 2.5x more code from FOSS than their code, while large libraries leverage only 3\% of FOSS code in their projects. Our models indicate that technical leverage shortens the release cycle for small-medium libraries. However, the risk of vulnerability exposure is 4-7x higher for libraries with high technical leverage. We also expanded our replication study to include the first level of transitive dependencies, and show that the results still hold, albeit with significant changes in the magnitude of both opportunities and risks of technical leverage. Our results indicate the extremes of opportunities and risks in NPM, where high technical leverage enables fast releases but comes at the cost of security risks.

cs.SE↗

arXiv2024

A Machine Learning-Based Approach For Detecting Malicious PyPI Packages

Background. In modern software development, the use of external libraries and packages is increasingly prevalent, streamlining the software development process and enabling developers to deploy feature-rich systems with little coding. While this reliance on reusing code offers substantial benefits, it also introduces serious risks for deployed software in the form of malicious packages - harmful and vulnerable code disguised as useful libraries. Aims. Popular ecosystems, such PyPI, receive thousands of new package contributions every week, and distinguishing safe contributions from harmful ones presents a significant challenge. There is a dire need for reliable methods to detect and address the presence of malicious packages in these environments. Method. To address these challenges, we propose a data-driven approach that uses machine learning and static analysis to examine the package's metadata, code, files, and textual characteristics to identify malicious packages. Results. In evaluations conducted within the PyPI ecosystem, we achieved an F1-measure of 0.94 for identifying malicious packages using a stacking ensemble classifier. Conclusions. This tool can be seamlessly integrated into package vetting pipelines and has the capability to flag entire packages, not just malicious function calls. This enhancement strengthens security measures and reduces the manual workload for developers and registry maintainers, thereby contributing to the overall integrity of the ecosystem.

cs.SE↗

arXiv2024

Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study

Code reviews are an integral part of software development and have been recognized as a crucial practice for minimizing bugs and favouring higher code quality. They serve as an important checkpoint before committing code and play an essential role in knowledge transfer between developers. However, code reviews can be time-consuming and can stale the development of large software projects. In a recent study, Guo et al. assessed how ChatGPT3.5 can help the code review process. They evaluated the effectiveness of ChatGPT in automating the code refinement tasks, where developers recommend small changes in the submitted code. While Guo et al. 's study showed promising results, proprietary models like ChatGPT pose risks to data privacy and incur extra costs for software projects. In this study, we explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2 (7B parameters). Our results show that, if properly tuned, the Llama models, particularly CodeLlama, can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. However, not all code refinement tasks are equally successful: tasks that require changing existing code (e.g., refactoring) are more manageable for models to automate than tasks that demand new code. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.

cs.SE↗

arXiv2024

Early Detection of Performance Regressions by Bridging Local Performance Data and Architectural Models

During software development, developers often make numerous modifications to the software to address existing issues or implement new features. However, certain changes may inadvertently have a detrimental impact on the overall system performance. To ensure that the performance of new software releases does not degrade, existing practices rely on system-level performance testing, such as load testing, or component-level performance testing to detect performance regressions. However, performance testing for the entire system is often expensive and time-consuming, posing challenges to adapting to the rapid release cycles common in modern DevOps practices. System-level performance testing cannot be conducted until the system is fully built and deployed. On the other hand, component-level testing focuses on isolated components, neglecting overall system performance and the impact of system workloads. In this paper, we propose a novel approach to early detection of performance regressions by bridging the local performance data generated by component-level testing and the system-level architectural models. Our approach uses local performance data to identify deviations at the component level, and then propagate these deviations to the architectural model. We then use the architectural model to predict regressions in the performance of the overall system. We evaluate our approach on two open-source benchmark systems and show that it can effectively detect end-to-end system performance regressions from local performance deviations with different intensities and under various system workloads. More importantly, our approach can detect regressions as early as in the development phase, in contrast to existing approaches that require the system to be fully built and deployed. Our approach is lightweight and can complement traditional system performance testing when testing resources are scarce.

cs.SE↗