arXiv subjects

Alessio Ferrari

Publications and source records attributed to Alessio Ferrari.

At least 19 recordsLinked to original sources

arXiv2026

Exploring the Role of Security Experience and ChatGPT Usage Strategies on Secure Software Engineering Education

The rapid adoption of Large Language Models (LLMs) is reshaping software engineering education, but their role in secure software engineering education remains underexplored. We report an exploratory empirical study of how 26 graduate students in a part-time MSc Cybersecurity programme used ChatGPT during a vulnerability-fixing assignment. To characterise ChatGPT use, we analysed students' ChatGPT interaction logs using a structured double-coding procedure and examined whether usage patterns and prior cybersecurity expertise were associated with assignment performance. The results show that students with varying levels of cybersecurity expertise used broadly similar ChatGPT strategies. Individual usage patterns showed descriptive differences by grade, but none remained statistically significant after correcting for multiple comparisons. In contrast, diversity of ChatGPT usage, i.e., the number of distinct usage patterns adopted, was positively associated with performance, even after controlling for cybersecurity expertise. These exploratory findings suggest that the way students engage with ChatGPT may be more informative than whether they use it, and motivate future controlled studies to guide students toward effective LLM use in secure software engineering education.

cs.SE↗

arXiv2026

When Stakeholder-centric Requirements Engineering is Not Enough: An Action Research Study on Legacy System Modernisation

Legacy system modernisation is a major challenge in digital transformation, especially when organisations depend on long-lived, business-critical systems that are only partly understood. In such context, organisations must define future needs while determining what current systems actually do and which functions to retain, adapt, or replace. Modernisation is therefore not only a technical challenge but also a requirements engineering (RE) problem, shaped by stakeholder perspectives. This study examines how far stakeholder-centric RE can support gap analysis between the system-as-is and the system-to-be in a legacy modernisation context. We conducted an action research study in a multinational energy company engaged in system modernisation. In the study, we applied stakeholder-centric RE practices, including stakeholder identification, semi-structured elicitation interviews, agreement-building through the Delphi method, and prioritisation with the MoSCoW method. The results show that this process was effective in producing requirements stakeholders generally viewed as understandable and correct, but less effective in achieving agreement on how elicited requirements mapped to legacy system functionality. The findings suggest that stakeholder-centric RE is necessary but not sufficient, pointing to the need for uncertainty-aware, iterative, and evidence-based modernisation practices that combine stakeholder perspectives with manual and tool-assisted analysis of legacy systems.

cs.SE↗

arXiv2026

ReqEvolve: User-Oriented Software Self-Evolution through Automatic Requirement Interpretation

The paradigm of software self-evolution enables systems to autonomously extend and reconfigure their own capabilities during execution in response to technical specifications. Yet requests for new functionality often originate from end users and are rarely expressed in technical terms. As a result, developers must translate user needs into technical specifications before the system can evolve, delaying early validation of the requested functionality by preventing users from immediately observing the resulting behaviour. To address this gap, we present ReqEvolve, a runtime code generation system that enables user-driven self-evolution by accepting high-level user requests. The system integrates automatic requirements engineering (RE) and test-driven development (TDD) to transform these requests into executable functionality through clarification, specification decomposition, test generation, and runtime integration. We evaluate ReqEvolve on 72 software evolution cases across 18 projects against two baselines: SpecFix, an RE-focused code generation approach, and an ablation variant of our system. ReqEvolve achieves 89.2% Pass@1, outperforming SpecFix by 18.8% (p < 0.01, r = 0.79, large effect) and the ablation baseline by 32.6% (p < 0.001, r = 0.88, large effect). These results provide initial evidence that user-driven self-evolution is a viable paradigm for autonomously extending software capabilities from user requests, thereby accelerating requirements validation prior to developer verification.

cs.SE↗

arXiv2026

Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering

Background. LLM outputs are highly sensitive to prompt formulation: small wording changes can substantially affect output quality. This matters in software engineering, where prompts guide requirements analysis, code generation, and artefact synthesis. Poor formulations yield unreliable artefacts, yet practitioners lack principled ways to assess a prompt before inference, making selection depend on costly LLM calls and trial-and-error refinement. Aims. We investigate whether measurable linguistic properties of prompts can predict LLM performance before inference, enabling low-cost prompt selection and refinement, validated on binary requirements classification targeting F1, F2, precision, and recall. Method. We generate 9,000 linguistically controlled prompt variants from 100 initial prompts by varying 30 linguistic metrics, evaluated with five open-source LLMs on 625 annotated requirements. Regression predictors are trained via stratified 10-fold cross-validation with permutation-based significance testing; feature importance analysis identifies cross-LLM and model-specific predictors. Results. Linguistic features significantly predict prompt performance across all targets (R2 in [0.38,0.42], q<0.05). Syntactic and morphosyntactic features drive most predictive signal; cross-LLM predictors include compound dependency distribution, conjunction density, and word/sentence length, reflecting sensitivity to domain vocabulary and complex structures. Conclusions. Results suggest practical implications for prompt engineering, including overlap between linguistic patterns that reduce LLM performance and those that increase human comprehension difficulty, and the irrelevance of lexical variety as a quality dimension. More broadly, linguistic profiling combined with standard regression provides an effective, interpretable, low-cost prior before costly optimisation pipelines.

cs.SE↗

arXiv2026

Fairness Hazard Analysis for Socio-Technical Processes: A Multiple-Case Study in Bias-sensitive Organisational Settings

Fairness is increasingly recognised as a first-class requirement in socio-technical processes, where interactions among human actors, software systems, and AI technologies may lead to unfair outcomes in decision-making workflows. If left unaddressed, fairness hazards may accumulate and reinforce systemic bias, highlighting the need to engineer fairness proactively. Despite growing interest in fairness-aware systems, systematic methods for identifying fairness hazards in socio-technical processes and deriving requirements-level mitigations remain limited. To support fairness-by-design during requirements engineering (RE), Fairness Hazard Analysis (FHA) is introduced as a methodology for systematically identifying, analysing, and mitigating fairness hazards. FHA is first assessed through a proof-of-concept validation conducted via two focus groups. Then, a qualitative multiple-case study involving two organisations examines its applicability in real-world settings. The proof-of-concept validation highlighted the benefits derived from the structured nature of the method, and suggested the need to include iterative, dialogic reflection with domain experts. In the multiple case-study where FHA was applied, the practitioners involved were positively impressed by the results and confirmed the relevance of the identified fairness hazards (spanning up to 27% of the process elements), as well as the appropriateness of most of the proposed mitigations, while noting that contextual factors might hinder their implementation. The evaluation also highlighted mitigation patterns, such as independent review and collective decision-making, which can be transferred to different organisations. This paper contributes a structured and empirically validated methodology for integrating fairness considerations in RE and preventing systemic bias in socio-technical processes.

cs.SE↗

arXiv2026

Human-AI Collaboration in Requirements Engineering: Evidence of the Negative Effect of LLMs on Requirements Inspection

Background. Requirements inspection (RI) is a well-established practice for detecting potential defects in requirements artifacts early in the software lifecycle. Recent advances in large language models (LLMs) have stimulated interest in their potential to support requirements engineering (RE) tasks. However, empirical evidence on the effects of LLMs when used as collaborative assistants in human-performed RI remains scarce. Aims. We aim to investigate the impact of LLM support on human-performed RI, considering inspection effectiveness in terms of smell identification and severity classification (i.e., nocuous vs innocuous), as well as inspection duration. Method. We conducted a controlled crossover design experiment with 34 participants, who inspected textual specifications with and without LLM support, identifying and classifying requirements smells while recording inspection time. We analyzed the data using one Bayesian regression model per outcome variable, accounting for validity threats induced by the crossover design as well as covariates and mediators. Results. Results show that LLM support negatively affects smell detection accuracy but has no significant effect on smell classification or task duration. A learning effect is present across experimental periods, but reduced when RI is first performed with LLM support. Conclusions. Our findings provide empirical evidence that LLM support does not necessarily improve performance and may, instead, hinder it for novice inspectors. Moreover, the results suggest that learning RI with LLM-support from the beginning may slow down the skill acquisition process, implying threats for LLM-supported learning.

cs.SE↗

arXiv2026

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

cs.SE↗

arXiv2026

BT-APE: A Computationally Light Backtracking Approach to Automatic Prompt Engineering for Requirements Classification

Large language models (LLMs) are increasingly applied to requirements engineering (RE) tasks, yet the prompts guiding them are typically designed manually through trial and error, yielding inconsistent and suboptimal results. Automated prompt construction remains largely unexplored in RE, leaving its effectiveness unclear. To address this, we propose a lightweight Automatic Prompt Engineering approach, Backtracking APE (BT-APE), and apply it to requirements classification. We frame prompt design as an optimization problem, iteratively refining prompts via LLM-generated candidates, backtracking search, and dynamic example selection. Evaluating BT-APE on three benchmark datasets with five instruction-tuned LLMs, we compare it against four classical prompting baselines (zero-shot, few-shot, chain-of-thought, CoT+few-shot) and a state-of-the-art but resource-intensive APE baseline (PE2). BT-APE and PE2 achieve nearly identical accuracy, both substantially outperforming the classical baselines with large effect sizes; however, BT-APE imposes a far lighter computational footprint, consuming roughly 72% fewer input tokens and 66% less wall-clock time at equivalent accuracy, making it better suited to resource-constrained deployment. Our contributions are threefold: (i) a lightweight APE framework with an open interactive tool and replication package; (ii) the first systematic comparison of APE against classical prompting for requirements classification; and (iii) insights into how class definitions and prompt evolution affect performance.

cs.SE↗

arXiv2026

Class Model Generation from Requirements using Large Language Models

The emergence of Large Language Models (LLMs) has opened new opportunities to automate software engineering activities that traditionally require substantial manual effort. Among these, class diagram generation represents a critical yet resource-intensive phase in software design. This paper investigates the capabilities of state-of-the-art LLMs, including GPT-5, Claude Sonnet 4.0, Gemini 2.5 Flash Thinking, and Llama-3.1-8B-Instruct, to generate UML class diagrams from natural language requirements automatically. To evaluate the effectiveness and reliability of LLM-based model generation, we propose a comprehensive dual-validation framework that integrates an LLM-as-a-Judge methodology with human-in-the-loop assessment. Using eight heterogeneous datasets, we apply chain-of-thought prompting to extract domain entities, attributes, and associations, generating corresponding PlantUML representations. The resulting models are evaluated across five quality dimensions: completeness, correctness, conformance to standards, comprehensibility, and terminological alignment. Two independent LLM judges (Grok and Mistral) perform structured pairwise comparisons, and their judgments are further validated against expert evaluations. Our results demonstrate that LLMs can generate structurally coherent and semantically meaningful UML diagrams, achieving substantial alignment with human evaluators. The consistency observed between LLM-based and human-based assessments highlights the potential of LLMs not only as modeling assistants but also as reliable evaluators in automated requirements engineering workflows, offering practical insights into the capabilities and limitations of LLM-driven UML class diagram automation.

cs.SE↗

arXiv2026

Software Self-Extension with SelfEvolve: an Agentic Architecture for Runtime Code Generation

Traditional self-adaptive systems automatically reconfigure existing components in response to changing requirements, but provide limited support for the generation of novel functionalities. The software generation capabilities of large language models (LLMs) open the possibility to create entirely new modules at runtime, enabling a form of self-evolution beyond traditional self-adaptation. We present SelfEvolve, an orchestrated agentic pipeline architecture enabling runtime self-extension--the autonomous addition of new capabilities during execution--as a preliminary form of self-evolution. Self-extension focuses on the autonomous generation and integration of new functions, based on user requests, without requiring a system restart or developer intervention. Evaluation of our architecture across 11 self-extension tasks demonstrates an average Pass@1 of 92.7% (51/55), outperforming developer-focused code generation baselines like AutoGen, MetaGPT, and AgentCoder. SelfEvolve achieves 61.8% improvement over the best baseline, i.e. Autogen, with statistical significance. This work demonstrates the feasibility of runtime capability extension through autonomous code generation. This provides preliminary evidence for a paradigm in which systems autonomously evolve to satisfy user needs, paving the way towards individualised, self-improving systems.

cs.SE↗

arXiv2026

Comprehension vs. Adoption: Evaluating a Language Workbench Through a Family of Experiments

Language workbenches are tools that enable the definition, reuse, and composition of programming languages and their ecosystems, aiming to streamline language development. To facilitate their adoption by language designers, the comprehensibility of the language used to define other languages is an important aspect to evaluate. Moreover, considering that language workbenches are relatively new tools, user acceptance emerges as a crucial factor to be accounted for during their assessment. Current literature often neglects user-centred aspects like comprehensibility and acceptance in the assessment of this breed of tools. This paper addresses this gap through a family of experiments assessing Neverlang, a modular language workbench. The study adopts a tailored version of the Method Evaluation Model (MEM) to evaluate the comprehensibility of Neverlang's meta-language and programs, as well as user acceptance in terms of perceived ease of use, perceived usefulness, and intention to use. It also investigates the relationships among these dimensions. The experiments were conducted in three iterations involving participants from academia. The results reveal that users demonstrate sufficient comprehension of Neverlang's meta-language, particularly concerning its syntax, express a favourable perception of its usefulness, and indicate their intention to use it. However, the results also indicate that Neverlang's ease of use remains a challenge. Additionally, variations in the perceived ease of use and perceived usefulness, whether low or high, influence the users' intention to use the tool. Surprisingly, no significant correlation is found between comprehensibility and user acceptance. Notably, higher comprehensibility of the meta-language does not necessarily translate into greater acceptance, underscoring the complex interplay between comprehension and adoption.

cs.SE↗

arXiv2026

The Role of Cognitive Abilities in Requirements Inspection: Comparing UML and Textual Representations

The representation of requirements plays a critical role in the accuracy of requirements inspection. While visual representations, such as UML diagrams, are widely used alongside text-based requirements, their effectiveness in supporting inspection is still debated. Cognitive abilities, such as working memory and mental rotation skills, may also influence inspection accuracy. This study aims to evaluate whether the use of UML sequence diagrams alongside text-based requirements improves the accuracy of requirements inspection compared to text-based requirements alone and to explore whether cognitive abilities are associated with differences in performance across the two treatments (text vs text with UML support). We conducted a crossover experiment with 38 participants to assess the accuracy of requirements inspection under the two treatments in terms of issues found and justifications provided. Linear mixed-effects and generalized linear models were used to analyse the effects of treatment, period, sequence, and cognitive abilities. The results indicate a significant three-way interaction between representation type, working memory capacity, and mental rotation ability. This finding suggests that the effectiveness of UML support is not uniform across individuals: participants with high scores in both cognitive abilities experienced reduced performance when using UML for violation detection. Conversely, the same cognitive profile was associated with improved justification accuracy under UML-aided inspection, indicating that higher cognitive abilities may support deeper reasoning processes when dealing with multi-modal information, i.e., diagrams and text.

cs.SE↗

arXiv2026

RITA: A Tool for Automated Requirements Classification and Specification from Online User Feedback

Context and motivation. Online user feedback is a valuable resource for requirements engineering, but its volume and noise make analysis difficult. Existing tools support individual feedback analysis tasks, but their capabilities are rarely integrated into end-to-end support. Problem. The lack of end-to-end integration limits the practical adoption of existing RE tools and makes it difficult to assess their real-world usefulness. Solution. To address this challenge, we present RITA, a tool that integrates lightweight open-source large language models into a unified workflow for feedback-driven RE. RITA supports automated request classification, non-functional requirement identification, and natural-language requirements specification generation from online feedback via a user-friendly interface, and integrates with Jira for seamless transfer of requirements specifications to development tools. Results and conclusions. RITA exploits previously evaluated LLM-based RE techniques to efficiently transform raw user feedback into requirements artefacts, helping bridge the gap between research and practice. A demonstration is available at: https://youtu.be/8meCLpwQWV8.

cs.SE↗

arXiv2025

From Online User Feedback to Requirements: Evaluating Large Language Models for Classification and Specification Tasks

[Context and Motivation] Online user feedback provides valuable information to support requirements engineering (RE). However, analyzing online user feedback is challenging due to its large volume and noise. Large language models (LLMs) show strong potential to automate this process and outperform previous techniques. They can also enable new tasks, such as generating requirements specifications. [Question-Problem] Despite their potential, the use of LLMs to analyze user feedback for RE remains underexplored. Existing studies offer limited empirical evidence, lack thorough evaluation, and rarely provide replication packages, undermining validity and reproducibility. [Principal Idea-Results] We evaluate five lightweight open-source LLMs on three RE tasks: user request classification, NFR classification, and requirements specification generation. Classification performance was measured on two feedback datasets, and specification quality via human evaluation. LLMs achieved moderate-to-high classification accuracy (F1 ~ 0.47-0.68) and moderately high specification quality (mean ~ 3/5). [Contributions] We newly explore lightweight LLMs for feedback-driven requirements development. Our contributions are: (i) an empirical evaluation of lightweight LLMs on three RE tasks, (ii) a replication package, and (iii) insights into their capabilities and limitations for RE.

cs.SE↗

arXiv2025

Does Model Size Matter? A Comparison of Small and Large Language Models for Requirements Classification

[Context and motivation] Large language models (LLMs) show notable results in natural language processing (NLP) tasks for requirements engineering (RE). However, their use is compromised by high computational cost, data sharing risks, and dependence on external services. In contrast, small language models (SLMs) offer a lightweight, locally deployable alternative. [Question/problem] It remains unclear how well SLMs perform compared to LLMs in RE tasks in terms of accuracy. [Results] Our preliminary study compares eight models, including three LLMs and five SLMs, on requirements classification tasks using the PROMISE, PROMISE Reclass, and SecReq datasets. Our results show that although LLMs achieve an average F1 score of 2% higher than SLMs, this difference is not statistically significant. SLMs almost reach LLMs performance across all datasets and even outperform them in recall on the PROMISE Reclass dataset, despite being up to 300 times smaller. We also found that dataset characteristics play a more significant role in performance than model size. [Contribution] Our study contributes with evidence that SLMs are a valid alternative to LLMs for requirements classification, offering advantages in privacy, cost, and local deployability.

cs.SE↗

arXiv2025

Green Prompt Engineering: Investigating the Energy Impact of Prompt Design in Software Engineering

Language Models are increasingly applied in software engineering, yet their inference raises growing environmental concerns. Prior work has examined hardware choices and prompt length, but little attention has been paid to linguistic complexity as a sustainability factor. This paper introduces Green Prompt Engineering, framing linguistic complexity as a design dimension that can influence energy consumption and performance. We conduct an empirical study on requirement classification using open-source Small Language Models, varying the readability of prompts. Our results reveal that readability affects environmental sustainability and performance, exposing trade-offs between them. For practitioners, simpler prompts can reduce energy costs without a significant F1-score loss; for researchers, it opens a path toward guidelines and studies on sustainable prompt design within the Green AI agenda.

cs.SE↗

arXiv2025

Large Language Models (LLMs) for Requirements Engineering (RE): A Systematic Literature Review

Large Language Models (LLMs) are finding applications in numerous domains, and Requirements Engineering (RE) is increasingly benefiting from their capabilities to assist with complex, language-intensive tasks. This paper presents a systematic literature review of 74 primary studies published between 2023 and 2024, examining how LLMs are being applied in RE. The study categorizes the literature according to several dimensions, including publication trends, RE activities, prompting strategies, and evaluation methods. Our findings indicate notable patterns, among which we observe substantial differences compared to previous works leveraging standard Natural Language Processing (NLP) techniques. Most of the studies focus on using LLMs for requirements elicitation and validation, rather than defect detection and classification, which were dominant in the past. Researchers have also broadened their focus and addressed novel tasks, e.g., test generation, exploring the integration of RE with other software engineering (SE) disciplines. Although requirements specifications remain the primary focus, other artifacts are increasingly considered, including issues from issue tracking systems, regulations, and technical manuals. The studies mostly rely on GPT-based models, and often use Zero-shot or Few-shot prompting. They are usually evaluated in controlled environments, with limited use in industry settings and limited integration in complex workflows. Our study outlines important future directions, such as leveraging the potential to expand the influence of RE in SE, exploring less-studied tasks, improving prompting methods, and testing in real-world environments. Our contribution also helps researchers and practitioners use LLMs more effectively in RE, by providing a list of identified tools leveraging LLMs for RE, as well as datasets.

cs.SE↗

arXiv2025

How Effective are Generative Large Language Models in Performing Requirements Classification?

In recent years, transformer-based large language models (LLMs) have revolutionised natural language processing (NLP), with generative models opening new possibilities for tasks that require context-aware text generation. Requirements engineering (RE) has also seen a surge in the experimentation of LLMs for different tasks, including trace-link detection, regulatory compliance, and others. Requirements classification is a common task in RE. While non-generative LLMs like BERT have been successfully applied to this task, there has been limited exploration of generative LLMs. This gap raises an important question: how well can generative LLMs, which produce context-aware outputs, perform in requirements classification? In this study, we explore the effectiveness of three generative LLMs-Bloom, Gemma, and Llama-in performing both binary and multi-class requirements classification. We design an extensive experimental study involving over 400 experiments across three widely used datasets (PROMISE NFR, Functional-Quality, and SecReq). Our study concludes that while factors like prompt design and LLM architecture are universally important, others-such as dataset variations-have a more situational impact, depending on the complexity of the classification task. This insight can guide future model development and deployment strategies, focusing on optimising prompt structures and aligning model architectures with task-specific needs for improved performance.

cs.CL↗