arXiv subjects

Jennifer Horkoff

Publications and source records attributed to Jennifer Horkoff.

At least 19 recordsLinked to original sources

arXiv2026

From Prompting to Engineering: A Research Agenda for Prompt Engineering in Software Engineering

Prompt engineering is increasingly used across Software Engineering (SE) activities, including requirements analysis, coding, testing, documentation, repository analysis, and planning. Yet prompts and related instruction artifacts are often created and evolved through task-specific and informal practices, with limited support for their systematic evaluation, management, traceability, and governance. To examine how SE can contribute to the maturation of these practices, we organized a structured community discussion at the First International Workshop on Empirical Prompt Engineering for Software Engineering (PROMPT-SE), co-located with EASE 2026. Participants discussed current prompting practices, challenges to their adoption and evaluation, and future directions for integrating prompt engineering into software development. We synthesized these discussions into five areas: prompt artifacts and standardization; evaluation and benchmarking; lifecycle integration; human-AI collaboration and skills; and governance, privacy, and technical debt. Based on these areas, we outline a research agenda to move prompt engineering from predominantly ad hoc interactions toward more systematic, maintainable, evaluable, traceable, and governable SE practices.

cs.SE↗

arXiv2026

Recommendations for Efficient and Responsible LLM Adoption within Industrial Software Development

Context: Large language models (LLMs) are observed to have a significant positive impact on various software engineering (SE) activities. With improved accessibility, the adoption of powerful LLMs in industry has surged recently. However, there is a lack of actionable best practices for the efficient and responsible adoption of LLMs within industrial software settings. Objectives: We developed seven actionable recommendations to address this research gap. Methods: We conducted a multi-case study with three organisations that use LLMs within their SE activities and synthesised seven recommendations through qualitative thematic analysis. We conducted a complementary online survey with software practitioners from various industries to evaluate the perceived relevance of our recommendations. Results: Our results and recommendations focus on (i) users' preference to use LLMs as AI assistants, (ii) the importance of relevant stakeholders' satisfaction in the LLM-output evaluation, (iii) scoping the applicability of LLMs within SE tasks, (iv) the effect of LLMs on SE workflows, (v) the necessity and directions for developing human oversight mechanisms, and (vi) the necessary skills for practitioners for leveraging LLMs within SE. The online survey indicates a high level of agreement from the participants regarding the perceived relevance of the recommendations. Conclusion: We outline future research directions, including mapping the seven recommendations to the principles of the EU AI Act (AIA) in order to examine how they relate to the current regulatory compliance frameworks.

cs.SE↗

arXiv2025

A Data Annotation Requirements Representation and Specification (DARS)

With the rise of AI-enabled cyber-physical systems, data annotation has become a critical yet often overlooked process in the development of these intelligent information systems. Existing work in requirements engineering (RE) has explored how requirements for AI systems and their data can be represented. However, related interviews with industry professionals show that data annotations and their related requirements introduce distinct challenges, indicating a need for annotation-specific requirement representations. We propose the Data Annotation Requirements Representation and Specification (DARS), including an Annotation Negotiation Card to align stakeholders on objectives and constraints, and a Scenario-Based Annotation Specification to express atomic and verifiable data annotation requirements. We evaluate DARS with an automotive perception case related to an ongoing project, and a mapping against 18 real-world data annotation error types. The results suggest that DARS mitigates root causes of completeness, accuracy, and consistency annotation errors. By integrating DARS into RE, this work improves the reliability of safety-critical systems using data annotations and demonstrates how engineering frameworks must evolve for data-dependent components of today's intelligent information systems.

cs.SE↗

arXiv2025

From Machine Learning Documentation to Requirements: Bridging Processes with Requirements Languages

In software engineering processes for machine learning (ML)-enabled systems, integrating and verifying ML components is a major challenge. A prerequisite is the specification of ML component requirements, including models and data, an area where traditional requirements engineering (RE) processes face new obstacles. An underexplored source of RE-relevant information in this context is ML documentation such as ModelCards and DataSheets. However, it is uncertain to what extent RE-relevant information can be extracted from these documents. This study first investigates the amount and nature of RE-relevant information in 20 publicly available ModelCards and DataSheets. We show that these documents contain a significant amount of potentially RE-relevant information. Next, we evaluate how effectively three established RE representations (EARS, Rupp's template, and Volere) can structure this knowledge into requirements. Our results demonstrate that there is a pathway to transform ML-specific knowledge into structured requirements, incorporating ML documentation in software engineering processes for ML systems.

cs.SE↗

arXiv2025

RE for AI in Practice: Managing Data Annotation Requirements for AI Autonomous Driving Systems

High-quality data annotation requirements are crucial for the development of safe and reliable AI-enabled perception systems (AIePS) in autonomous driving. Although these requirements play a vital role in reducing bias and enhancing performance, their formulation and management remain underexplored, leading to inconsistencies, safety risks, and regulatory concerns. Our study investigates how annotation requirements are defined and used in practice, the challenges in ensuring their quality, practitioner-recommended improvements, and their impact on AIePS development and performance. We conducted $19$ semi-structured interviews with participants from six international companies and four research organisations. Our thematic analysis reveals five main key challenges: ambiguity, edge case complexity, evolving requirements, inconsistencies, and resource constraints and three main categories of best practices, including ensuring compliance with ethical standards, improving data annotation requirements guidelines, and embedded quality assurance for data annotation requirements. We also uncover critical interrelationships between annotation requirements, annotation practices, annotated data quality, and AIePS performance and development, showing how requirement flaws propagate through the AIePS development pipeline. To the best of our knowledge, this study is the first to offer empirically grounded guidance on improving annotation requirements, offering actionable insights to enhance annotation quality, regulatory compliance, and system reliability. It also contributes to the emerging fields of Software Engineering (SE for AI) and Requirements Engineering (RE for AI) by bridging the gap between RE and AI in a timely and much-needed manner.

cs.SE↗

arXiv2025

An AUTOSAR-Aligned Architectural Study of Vulnerabilities in Automotive SoC Software

Cooperative, Connected and Automated Mobility (CCAM) are complex cyber-physical systems (CPS) that integrate computation, communication, and control in safety-critical environments. At their core, System-on-Chip (SoC) platforms consolidate processing units, communication interfaces, AI accelerators, and security modules into a single chip. AUTOSAR (AUTomotive Open System ARchitecture) standard was developed in the automotive domain to better manage this complexity, defining layered software structures and interfaces to facilitate reuse of HW/SW components. However, in practice, this integrated SoC software architecture still poses security challenges, particularly in real-time, safety-critical environments. Recent reports highlight a surge in SoC-related vulnerabilities, yet systematic analysis of their root causes and impact within AUTOSAR-aligned architectures is lacking. This study fills that gap by analyzing 180 publicly reported automotive SoC vulnerabilities, mapped to a representative SoC software architecture model that is aligned with AUTOSAR principles for layered abstraction and service orientation. We identify 16 root causes and 56 affected software modules, and examine mitigation delays across Common Weakness Enumeration (CWE) categories and architectural layers. We uncover dominant vulnerability patterns and critical modules with prolonged patch delays, and provide actionable insights for securing automotive CPS platforms, including guides for improved detection, prioritization, and localization strategies for SoC software architectures in SoC-based vehicle platforms.

cs.SE↗

arXiv2025

Extending Behavior Trees for Robotic Missions with Quality Requirements

Context and motivation: In recent years, behavior trees have gained growing interest within the robotics community as a specification and control switching mechanism for the different tasks that form a robotics mission. Problem: Given the rising complexity and prevalence of robotic systems, it is increasingly challenging and important for practitioners to design high-quality missions that meet certain qualities, for instance, to consider potential failures or mitigate safety risks. In software requirements engineering, quality or non-functional requirements have long been recognized as a key factor in system success. Currently, qualities are not represented in behavior tree models, which capture a robotic mission, making it difficult to assess the extent to which different mission components comply with those qualities. Principal ideas: In this paper, we propose an extension for behavior trees to have qualities and quality requirements explicitly represented in robotics missions. We provide a meta-model for the extension, develop a domain-specific language (DSL), and describe how we integrated our DSL in one of the most used languages in robotics for developing behavior trees, BehaviorTree.CPP. A preliminary evaluation of the implemented DSL shows promising results for the feasibility of our approach and the need for similar DSLs. Contribution: Our approach paves the way for incorporating qualities into the behavior model of robotics missions. This promotes early expression of qualities in robotics missions, and a better overview of missions components and their contribution to the satisfaction of quality concerns.

cs.SE↗

arXiv2024

RE-centric Recommendations for the Development of Trustworthy(er) Autonomous Systems

Complying with the EU AI Act (AIA) guidelines while developing and implementing AI systems will soon be mandatory within the EU. However, practitioners lack actionable instructions to operationalise ethics during AI systems development. A literature review of different ethical guidelines revealed inconsistencies in the principles addressed and the terminology used to describe them. Furthermore, requirements engineering (RE), which is identified to foster trustworthiness in the AI development process from the early stages was observed to be absent in a lot of frameworks that support the development of ethical and trustworthy AI. This incongruous phrasing combined with a lack of concrete development practices makes trustworthy AI development harder. To address this concern, we formulated a comparison table for the terminology used and the coverage of the ethical AI principles in major ethical AI guidelines. We then examined the applicability of ethical AI development frameworks for performing effective RE during the development of trustworthy AI systems. A tertiary review and meta-analysis of literature discussing ethical AI frameworks revealed their limitations when developing trustworthy AI. Based on our findings, we propose recommendations to address such limitations during the development of trustworthy AI.

cs.CY↗

arXiv2023

Requirements Engineering using Generative AI: Prompts and Prompting Patterns

[Context]: Companies are increasingly recognizing the importance of automating Requirements Engineering (RE) tasks due to their resource-intensive nature. The advent of GenAI has made these tasks more amenable to automation, thanks to its ability to understand and interpret context effectively. [Problem]: However, in the context of GenAI, prompt engineering is a critical factor for success. Despite this, we currently lack tools and methods to systematically assess and determine the most effective prompt patterns to employ for a particular RE task. [Method]: Two tasks related to requirements, specifically requirement classification and tracing, were automated using the GPT-3.5 turbo API. The performance evaluation involved assessing various prompts created using 5 prompt patterns and implemented programmatically to perform the selected RE tasks, focusing on metrics such as precision, recall, accuracy, and F-Score. [Results]: This paper evaluates the effectiveness of the 5 prompt patterns' ability to make GPT-3.5 turbo perform the selected RE tasks and offers recommendations on which prompt pattern to use for a specific RE task. Additionally, it also provides an evaluation framework as a reference for researchers and practitioners who want to evaluate different prompt patterns for different RE tasks.

cs.SE↗

arXiv2023

Investigating ChatGPT's Potential to Assist in Requirements Elicitation Processes

Natural Language Processing (NLP) for Requirements Engineering (RE) (NLP4RE) seeks to apply NLP tools, techniques, and resources to the RE process to increase the quality of the requirements. There is little research involving the utilization of Generative AI-based NLP tools and techniques for requirements elicitation. In recent times, Large Language Models (LLM) like ChatGPT have gained significant recognition due to their notably improved performance in NLP tasks. To explore the potential of ChatGPT to assist in requirements elicitation processes, we formulated six questions to elicit requirements using ChatGPT. Using the same six questions, we conducted interview-based surveys with five RE experts from academia and industry and collected 30 responses containing requirements. The quality of these 36 responses (human-formulated + ChatGPT-generated) was evaluated over seven different requirements quality attributes by another five RE experts through a second round of interview-based surveys. In comparing the quality of requirements generated by ChatGPT with those formulated by human experts, we found that ChatGPT-generated requirements are highly Abstract, Atomic, Consistent, Correct, and Understandable. Based on these results, we present the most pressing issues related to LLMs and what future research should focus on to leverage the emergent behaviour of LLMs more effectively in natural language-based RE activities.

cs.SE↗

arXiv2023

Automotive Perception Software Development: An Empirical Investigation into Data, Annotation, and Ecosystem Challenges

Software that contains machine learning algorithms is an integral part of automotive perception, for example, in driving automation systems. The development of such software, specifically the training and validation of the machine learning components, require large annotated datasets. An industry of data and annotation services has emerged to serve the development of such data-intensive automotive software components. Wide-spread difficulties to specify data and annotation needs challenge collaborations between OEMs (Original Equipment Manufacturers) and their suppliers of software components, data, and annotations. This paper investigates the reasons for these difficulties for practitioners in the Swedish automotive industry to arrive at clear specifications for data and annotations. The results from an interview study show that a lack of effective metrics for data quality aspects, ambiguities in the way of working, unclear definitions of annotation quality, and deficits in the business ecosystems are causes for the difficulty in deriving the specifications. We provide a list of recommendations that can mitigate challenges when deriving specifications and we propose future research opportunities to overcome these challenges. Our work contributes towards the on-going research on accountability of machine learning as applied to complex software systems, especially for high-stake applications such as automated driving.

cs.SE↗

arXiv2023

Requirements Engineering for Automotive Perception Systems: an Interview Study

Background: Driving automation systems (DAS), including autonomous driving and advanced driver assistance, are an important safety-critical domain. DAS often incorporate perceptions systems that use machine learning (ML) to analyze the vehicle environment. Aims: We explore new or differing requirements engineering (RE) topics and challenges that practitioners experience in this domain. Method: We have conducted an interview study with 19 participants across five companies and performed thematic analysis. Results: Practitioners have difficulty specifying upfront requirements, and often rely on scenarios and operational design domains (ODDs) as RE artifacts. Challenges relate to ODD detection and ODD exit detection, realistic scenarios, edge case specification, breaking down requirements, traceability, creating specifications for data and annotations, and quantifying quality requirements. Conclusions: Our findings contribute to understanding how RE is practiced for DAS perception systems and the collected challenges can drive future research for DAS and other ML-enabled systems.

cs.SE↗

arXiv2022

Non-Functional Requirements for Machine Learning: An Exploration of System Scope and Interest

Systems that rely on Machine Learning (ML systems) have differing demands on system quality compared to traditional systems. Such quality demands, known as non-functional requirements (NFRs), may differ in their definition, scope, and importance from NFRs for traditional systems. Despite the importance of NFRs for ML systems, our understanding of their definitions and scope -- and of the extent of existing research in each NFR -- is lacking compared to our understanding in traditional domains. Building on an investigation into importance and treatment of ML system NFRs in industry, we make three contributions towards narrowing this gap: (1) we present clusters of ML system NFRs based on shared characteristics, (2) we use Scopus search results -- as well as inter-coder reliability on a sample of NFRs -- to estimate the number of relevant studies on a subset of the NFRs, and (3), we use our initial reading of titles and abstracts in each sample to define the scope of NFRs over parts of the system (e.g., training data, ML model, or other system elements). These initial findings form the groundwork for future research in this emerging domain.

cs.SE↗

arXiv2021

Non-functional Requirements for Machine Learning: Understanding Current Use and Challenges in Industry

Machine Learning (ML) is an application of Artificial Intelligence (AI) that uses big data to produce complex predictions and decision-making systems, which would be challenging to obtain otherwise. To ensure the success of ML-enabled systems, it is essential to be aware of certain qualities of ML solutions (performance, transparency, fairness), known from a Requirement Engineering (RE) perspective as non-functional requirements (NFRs). However, when systems involve ML, NFRs for traditional software may not apply in the same ways; some NFRs may become more prominent or less important; NFRs may be defined over the ML model, data, or the entire system; and NFRs for ML may be measured differently. In this work, we aim to understand the state-of-the-art and challenges of dealing with NFRs for ML in industry. We interviewed ten engineering practitioners working with NFRs and ML. We find examples of (1) the identification and measurement of NFRs for ML, (2) identification of more and less important NFRs for ML, and (3) the challenges associated with NFRs and ML in the industry. This knowledge paints a picture of how ML-related NFRs are treated in practice and helps to guide future RE for ML efforts.

cs.SE↗

arXiv2021

Managing Traceability Information Models: Not such a simple task after all?

Practitioners are poorly supported by the scientific literature when managing traceability information models (TIMs), which capture the structure and semantics of trace links. In practice, companies manage their TIMs in very different ways, even in cases where companies share many similarities. We present our findings from an in-depth focus group about TIM management with three different systems engineering companies. We find that the concrete needs of the companies as well as challenges such as scale and workflow integration are not considered by existing scientific work. We thus issue a call-to-arms for the requirements engineering and software and systems traceability communities, the two main communities for traceability research, to refocus their work on these practical problems.

cs.SE↗

arXiv2020

Modeling and Analysis of Boundary Objects and Methodological Islands in Large-Scale Systems Development

Large-scale companies commonly face the challenge of managing relevant knowledge between different organizational groups, particularly in increasingly agile contexts. In previous studies, we found the importance of analyzing methodological islands (i.e., groups using different development methods than the surrounding organization) and boundary objects between them. In this paper, we propose a metamodel to better capture and analyze coordination and knowledge management in practice. Such a metamodel can allow practitioners to describe current practices, analyze issues, and design better-suited coordination mechanisms. We evaluated the conceptual model together with four large-scale companies developing complex systems. In particular, we derived an initial list of bad smells that can be leveraged to detect issues and devise suitable improvement strategies for inter-team coordination in large-scale development. We present the model, smells, and our evaluation results.

cs.SE↗

arXiv2020

Charting Coordination Needs in Large-Scale Agile Organisationswith Boundary Objects and Methodological Islands

Large-scale system development companies are increasingly adopting agile methods. While this adoption may improve lead-times, such companies need to balance two trade-offs: (i) the need to have a uniform, consistent development method on system level with the need for specialised methods for teams in different disciplines(e.g., hardware, software, mechanics, sales, support); (ii) the need for comprehensive documentation on system level with the need to have lightweight documentation enabling iterative and agile work. With specialised methods for teams, isolated teams work within larger ecosystems of plan-driven culture, i.e., teams become agile "islands". At the boundaries, these teams share knowledge which needs to be managed well for a correct system to be developed. While it is useful to support diverse and specialised methods, it is important to understand which islands are repeatedly encountered, the reasons or factors triggering their existence, and how best to handle coordination between them. Based on a multiple case study, this work presents a catalogue of islands and the boundary objects between them. We believe this work will be beneficial to practitioners aiming to understand their ecosystems and researchers addressing communication and coordination challenges in large-scale development.

cs.SE↗

arXiv2019

Challenges of Scaled Agile for Safety-Critical Systems

Automotive companies increasingly adopt scaled agile methods to allow them to deal with their organisational and product complexity. Suitable methods are needed to ensure safety when developing automotive systems. On a small scale, R-Scrum and SafeScrum are two concrete suggestions for how to develop safety-critical systems using agile methods. However, for large-scale environments, existing frameworks like SAFe or LeSS do not support the development of safety-critical systems out of the box. We, therefore, aim to understand which challenges exist when developing safety-critical systems within large-scale agile industrial settings, in particular in the automotive domain. Based on an analysis of R-Scrum and SafeScrum, we conducted a focus group with three experts from industry to collect challenges in their daily work. We found challenges in the areas of living traceability, continuous compliance, and organisational flexibility. Among others, organisations struggle with defining a suitable traceability strategy, performing incremental safety analysis, and with integrating safety practices into their scaled way of working. Our results indicate a need to provide practical approaches to integrate safety work into large-scale agile development and point towards possible solutions, e.g., modular safety cases. Keywords: Scaled Agile, Safety-Critical Systems, Software Processes, R-Scrum, SafeScrum

cs.SE↗