arXiv subjects

Leonardo Mariani

Publications and source records attributed to Leonardo Mariani.

At least 37 records · Page 2Linked to original sources

arXiv2025

Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT

Leveraging LLMs for code generation is becoming increasingly common, as tools like ChatGPT can suggest method implementations with minimal input, such as a method signature and brief description. Empirical studies further highlight the effectiveness of LLMs in handling such tasks, demonstrating notable performance in code generation scenarios. However, LLMs are inherently non-deterministic, with their output influenced by parameters such as temperature, which regulates the model's level of creativity, and top-p, which controls the choice of the tokens that shall appear in the output. Despite their significance, the role of these parameters is often overlooked. This paper systematically studies the impact of these parameters, as well as the number of prompt repetitions required to account for non-determinism, in the context of 548 Java methods. We observe significantly different performances across different configurations of ChatGPT, with temperature having a marginal impact compared to the more prominent influence of the top-p parameter. Additionally, we show how creativity can enhance code generation tasks. Finally, we provide concrete recommendations for addressing the non-determinism of the model.

cs.SE↗

arXiv2024

Testing in the Evolving World of DL Systems:Insights from Python GitHub Projects

In the ever-evolving field of Deep Learning (DL), ensuring project quality and reliability remains a crucial challenge. This research investigates testing practices within DL projects in GitHub. It quantifies the adoption of testing methodologies, focusing on aspects like test automation, the types of tests (e.g., unit, integration, and system), test suite growth rate, and evolution of testing practices across different project versions. We analyze a subset of 300 carefully selected repositories based on quantitative and qualitative criteria. This study reports insights on the prevalence of testing practices in DL projects within the open-source community.

cs.SE↗

arXiv2024

ReProbe: An Architecture for Reconfigurable and Adaptive Probes

Modern distributed systems are highly dynamic and scalable, requiring monitoring solutions that can adapt to rapid changes. Monitoring systems that rely on external probes can only achieve adaptation through expensive operations such as deployment, undeployment, and reconfiguration. This poster paper introduces ReProbes, a class of adaptive monitoring probes that can handle rapid changes in data collection strategies. ReProbe offers controllable and configurable self-adaptive capabilities for data transmission, collection, and analysis methods. The resulting architecture can effectively enhance probe adaptability when qualitatively compared to state-of-the-art monitoring solutions.

cs.SE↗

arXiv2024

Towards Model-Driven Dashboard Generation for Systems-of-Systems

Configuring and evolving dashboards in complex and large-scale Systems-of-Systems (SoS) can be an expensive and cumbersome task due to the many Key Performance Indicators (KPIs) that are usually collected and have to be arranged in a number of visualizations. Unfortunately, setting up dashboards is still a largely manual and error-prone task requiring extensive human intervention. This short paper describes emerging results about the definition of a model-driven technology-agnostic approach that can automatically transform a simple list of KPIs into a dashboard model, and then translate the model into an actual dashboard for a target dashboard technology. Dashboard customization can be efficiently obtained by solely modifying the abstract model representation, freeing operators from expensive interactions with actual dashboards.

cs.SE↗

arXiv2024

Assessing AI-Based Code Assistants in Method Generation Tasks

AI-based code assistants are increasingly popular as a means to enhance productivity and improve code quality. This study compares four AI-based code assistants, GitHub Copilot, Tabnine, ChatGPT, and Google Bard, in method generation tasks, assessing their ability to produce accurate, correct, and efficient code. Results show that code assistants are useful, with complementary capabilities, although they rarely generate ready-to-use correct code.

cs.SE↗

arXiv2024

Generating Java Methods: An Empirical Assessment of Four AI-Based Code Assistants

AI-based code assistants are promising tools that can facilitate and speed up code development. They exploit machine learning algorithms and natural language processing to interact with developers, suggesting code snippets (e.g., method implementations) that can be incorporated into projects. Recent studies empirically investigated the effectiveness of code assistants using simple exemplary problems (e.g., the re-implementation of well-known algorithms), which fail to capture the spectrum and nature of the tasks actually faced by developers. In this paper, we expand the knowledge in the area by comparatively assessing four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard, with a dataset of 100 methods that we constructed from real-life open-source Java projects, considering a variety of cases for complexity and dependency from contextual elements. Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches. Interestingly, the effectiveness of these solutions dramatically decreases when dealing with dependencies outside the boundaries of single classes.

cs.SE↗

arXiv2024

Analyzing Prompt Influence on Automated Method Generation: An Empirical Study with Copilot

Generative AI is changing the way developers interact with software systems, providing services that can produce and deliver new content, crafted to satisfy the actual needs of developers. For instance, developers can ask for new code directly from within their IDEs by writing natural language prompts, and integrated services based on generative AI, such as Copilot, immediately respond to prompts by providing ready-to-use code snippets. Formulating the prompt appropriately, and incorporating the useful information while avoiding any information overload, can be an important factor in obtaining the right piece of code. The task of designing good prompts is known as prompt engineering. In this paper, we systematically investigate the influence of eight prompt features on the style and the content of prompts, on the level of correctness, complexity, size, and similarity to the developers' code of the generated code. We specifically consider the task of using Copilot with 124,800 prompts obtained by systematically combining the eight considered prompt features to generate the implementation of 200 Java methods. Results show how some prompt features, such as the presence of examples and the summary of the purpose of the method, can significantly influence the quality of the result.

cs.SE↗

arXiv2024

Anonymizing Test Data in Android: Does It Hurt?

Failure data collected from the field (e.g., failure traces, bug reports, and memory dumps) represent an invaluable source of information for developers who need to reproduce and analyze failures. Unfortunately, field data may include sensitive information and thus cannot be collected indiscriminately. Privacy-preserving techniques can address this problem anonymizing data and reducing the risk of disclosing personal information. However, collecting anonymized information may harm reproducibility, that is, the anonymized data may not allow the reproduction of a failure observed in the field. In this paper, we present an empirical investigation about the impact of privacy-preserving techniques on the reproducibility of failures. In particular, we study how five privacy-preserving techniques may impact reproducibilty for 19 bugs in 17 Android applications. Results provide insights on how to select and configure privacy-preserving techniques.

cs.SE↗

arXiv2024

MutaBot: A Mutation Testing Approach for Chatbots

Mutation testing is a technique aimed at assessing the effectiveness of test suites by seeding artificial faults into programs. Although available for many platforms and languages, no mutation testing tool is currently available for conversational chatbots, which represent an increasingly popular solution to design systems that can interact with users through a natural language interface. Note that since conversations must be explicitly engineered by the developers of conversational chatbots, these systems are exposed to specific types of faults not supported by existing mutation testing tools. In this paper, we present MutaBot, a mutation testing tool for conversational chatbots. MutaBot addresses mutations at multiple levels, including conversational flows, intents, and contexts. We designed the tool to potentially target multiple platforms, while we implemented initial support for Google Dialogflow chatbots. We assessed the tool with three Dialogflow chatbots and test cases generated with Botium, revealing weaknesses in the test suites.

cs.SE↗

arXiv2023

Automated Probe Life-Cycle Management for Monitoring-as-a-Service

Cloud services must be continuously monitored to guarantee that misbehaviors can be timely revealed, compensated, and fixed. While simple applications can be easily monitored and controlled, monitoring non-trivial cloud systems with dynamic behavior requires the operators to be able to rapidly adapt the set of collected indicators. Although the currently available monitoring frameworks are equipped with a rich set of probes to virtually collect any indicator, they do not provide the automation capabilities required to quickly and easily change (i.e., deploy and undeploy) the probes used to monitor a target system. Indeed, changing the collected indicators beyond standard platform-level indicators can be an error-prone and expensive process, which often requires manual intervention. This paper presents a Monitoring-as-a-Service framework that provides the capability to automatically deploy and undeploy arbitrary probes based on a user-provided set of indicators to be collected. The life-cycle of the probes is fully governed by the framework, including the detection and resolution of the erroneous states at deployment time. The framework can be used jointly with existing monitoring technologies, without requiring the adoption of a specific probing technology. We experimented our framework with cloud systems based on containers and virtual machines, obtaining evidence of the efficiency and effectiveness of the proposed solution.

cs.DC↗

arXiv2023

Supporting Early-Safety Analysis of IoT Systems by Exploiting Testing Techniques

IoT systems complexity and susceptibility to failures pose significant challenges in ensuring their reliable operation Failures can be internally generated or caused by external factors impacting both the systems correctness and its surrounding environment To investigate these complexities various modeling approaches have been proposed to raise the level of abstraction facilitating automation and analysis FailureLogic Analysis FLA is a technique that helps predict potential failure scenarios by defining how a components failure logic behaves and spreads throughout the system However manually specifying FLA rules can be arduous and errorprone leading to incomplete or inaccurate specifications In this paper we propose adopting testing methodologies to improve the completeness and correctness of these rules How failures may propagate within an IoT system can be observed by systematically injecting failures while running test cases to collect evidence useful to add complete and refine FLA rules

cs.SE↗

arXiv2023

An Energy-Aware Approach to Design Self-Adaptive AI-based Applications on the Edge

The advent of edge devices dedicated to machine learning tasks enabled the execution of AI-based applications that efficiently process and classify the data acquired by the resource-constrained devices populating the Internet of Things. The proliferation of such applications (e.g., critical monitoring in smart cities) demands new strategies to make these systems also sustainable from an energetic point of view. In this paper, we present an energy-aware approach for the design and deployment of self-adaptive AI-based applications that can balance application objectives (e.g., accuracy in object detection and frames processing rate) with energy consumption. We address the problem of determining the set of configurations that can be used to self-adapt the system with a meta-heuristic search procedure that only needs a small number of empirical samples. The final set of configurations are selected using weighted gray relational analysis, and mapped to the operation modes of the self-adaptive application. We validate our approach on an AI-based application for pedestrian detection. Results show that our self-adaptive application can outperform non-adaptive baseline configurations by saving up to 81\% of energy while loosing only between 2% and 6% in accuracy.

cs.SE↗

arXiv2023

Measuring Software Testability via Automatically Generated Test Cases

Estimating software testability can crucially assist software managers to optimize test budgets and software quality. In this paper, we propose a new approach that radically differs from the traditional approach of pursuing testability measurements based on software metrics, e.g., the size of the code or the complexity of the designs. Our approach exploits automatic test generation and mutation analysis to quantify the evidence about the relative hardness of developing effective test cases. In the paper, we elaborate on the intuitions and the methodological choices that underlie our proposal for estimating testability, introduce a technique and a prototype that allows for concretely estimating testability accordingly, and discuss our findings out of a set of experiments in which we compare the performance of our estimations both against and in combination with traditional software metrics. The results show that our testability estimates capture a complementary dimension of testability that can be synergistically combined with approaches based on software metrics to improve the accuracy of predictions.

cs.SE↗

arXiv2023

Property-Based Mutation Testing

Mutation testing is an established software quality assurance technique for the assessment of test suites. While it is well-suited to estimate the general fault-revealing capability of a test suite, it is not practical and informative when the software under test must be validated against specific requirements. This is often the case for embedded software, where the software is typically validated against rigorously-specified safety properties. In such a scenario (i) a mutant is relevant only if it can impact the satisfaction of the tested properties, and (ii) a mutant is meaningfully-killed with respect to a property only if it causes the violation of that property. To address these limitations of mutation testing, we introduce property-based mutation testing, a method for assessing the capability of a test suite to exercise the software with respect to a given property. We evaluate our property-based mutation testing framework on Simulink models of safety-critical Cyber-Physical Systems (CPS) from the automotive and avionic domains and demonstrate how property-based mutation testing is more informative than regular mutation testing. These results open new perspectives in both mutation testing and test case generation of CPS.

cs.SE↗

arXiv2022

Non-Functional Testing of Runtime Enforcers in Android

Runtime enforcers can be used to ensure that running applications satisfy desired correctness properties. Although runtime enforcers that are correct-by-construction with respect to abstract behavioral models are relatively easy to specify, the concrete software enforcers generated from these specifications may easily introduce issues in the target application. Indeed developers can generate test suites to verify the functional behavior of the enforcers, for instance exploiting the same models used to specify them. However, it remains challenging and tedious to verify the behavior of enforcers in terms of non-functional performance characteristics. This paper describes a practical approach to reveal runtime enforcers that may introduce inefficiencies in the target application. The approach relies on a combination of automatic test generation and runtime monitoring of multiple key performance indicators. We designed our approach to reveal issues in four indicators for mobile systems: responsiveness, launch time, memory, and energy consumption. Experimental results show that our approach can detect performance issues that might be introduced by automatically generated enforcers.

cs.SE↗

arXiv2022

Towards Self-Adaptive Peer-to-Peer Monitoring for Fog Environments

Monitoring is a critical component in fog environments: it promptly provides insights about the behavior of systems, reveals Service Level Agreements (SLAs) violations, enables the autonomous orchestration of services and platforms, calls for the intervention of operators, and triggers self-healing actions. In such environments, monitoring solutions have to cope with the heterogeneity of the devices and platforms present in the Fog, the limited resources available at the edge of the network, and the high dynamism of the whole Cloud-to-Thing continuum. This paper addresses the challenge of accurately and efficiently monitoring the Fog with a self-adaptive peer-to-peer (P2P) monitoring solution that can opportunistically adjust its behavior according to the collected data exploiting a lightweight rule-based expert system. Empirical results show that adaptation can improve monitoring accuracy, while reducing network and power consumption at the cost of higher memory consumption.

eess.SY↗

arXiv2022

Proactive Libraries: Enforcing Correct Behaviors in Android Apps

The Android framework provides a rich set of APIs that can be exploited by developers to build their apps. However, the rapid evolution of these APIs jointly with the specific characteristics of the lifecycle of the Android components challenge developers, who may release apps that use APIs incorrectly. In this demo, we present Proactive Libraries, a tool that can be used to decorate regular libraries with the capability of proactively detecting and healing API misuses at runtime. Proactive Libraries blend libraries with multiple proactive modules that collect data, check the compliance of API usages with correctness policies, and heal executions as soon as the possible violation of a policy is detected. The results of our evaluation with 27 possible API misuses show the effectiveness of Proactive Libraries in correcting API misuses with negligible runtime overhead.

cs.SE↗

arXiv2022

Exception-Driven Fault Localization for Automated Program Repair

Automated Program Repair (APR) techniques typically exploit spectrum-based fault localization (SBFL) to identify the program locations that should be patched, making the effectiveness of APR techniques dependent on the effectiveness of fault localization. Indeed, results show that SBFL often does not localize faults accurately, hindering the effectiveness of APR. In this paper, we propose EXCEPT, a technique that addresses the localization problem by focusing on the semantics of failures rather than on the correlation between the executed statements and the failed tests, as SBFL does. We focus on failures due to exceptions and we exploit their type and source to localize and guess the faults. Experiments with 43 exception-raising faults from the Defects4J benchmark show that EXCEPT can perform better than Ochiai and ssFix.

cs.SE↗