arXiv subjects

Tor Sporsem

Publications and source records attributed to Tor Sporsem.

10 recordsLinked to original sources

arXiv2025

Towards Requirements Engineering for RAG Systems

This short paper explores how a maritime company develops and integrates large-language models (LLM). Specifically by looking at the requirements engineering for Retrieval Augmented Generation (RAG) systems in expert settings. Through a case study at a maritime service provider, we demonstrate how data scientists face a fundamental tension between user expectations of AI perfection and the correctness of the generated outputs. Our findings reveal that data scientists must identify context-specific "retrieval requirements" through iterative experimentation together with users because they are the ones who can determine correctness. We present an empirical process model describing how data scientists practically elicited these "retrieval requirements" and managed system limitations. This work advances software engineering knowledge by providing insights into the specialized requirements engineering processes for implementing RAG systems in complex domain-specific applications.

cs.SE↗

arXiv2025

What Does Explainable AI Mean in Practice? Evaluative Requirements from a Longitudinal Clinical Case Study

This paper reports a case study on how explainability requirements were elicited during the development of an AI system for predicting cerebral palsy (CP) risk in infants. Over 18 months, we followed a development team and hospital clinicians as they sought to design explanations that would make the AI system trustworthy. Contrary to the assumption that users need detailed explanations of the inner workings of AI systems, our findings show that clinicians trusted it when it enabled them to evaluate predictions against their own assessments. Our findings show how a simple prediction graph proved effective by supporting clinicians' existing decision-making practices. Drawing on concepts from both Requirements Engineering and Explainable AI, we use the theoretical lens of Evaluative AI to introduce the notion of Evaluative Requirements: system requirements that allow users to scrutinize AI outputs and compare them with their own assessments. Our study demonstrates that such requirements are best discovered through the well-known methods of iterative prototyping and observation, making them essential for building trustworthy AI systems in expert domains.

cs.SE↗

arXiv2024

Using LLM-Generated Draft Replies to Support Human Experts in Responding to Stakeholder Inquiries in Maritime Industry: A Real-World Case Study of Industrial AI

The maritime industry requires effective communication among diverse stakeholders to address complex, safety-critical challenges. Industrial AI, including Large Language Models (LLMs), has the potential to augment human experts' workflows in this specialized domain. Our case study investigated the utility of LLMs in drafting replies to stakeholder inquiries and supporting case handlers. We conducted a preliminary study (observations and interviews), a survey, and a text similarity analysis (LLM-as-a-judge and Semantic Embedding Similarity). We discover that while LLM drafts can streamline workflows, they often require significant modifications to meet the specific demands of maritime communications. Though LLMs are not yet mature enough for safety-critical applications without human oversight, they can serve as valuable augmentative tools. Final decision-making thus must remain with human experts. However, by leveraging the strengths of both humans and LLMs, fostering human-AI collaboration, industries can increase efficiency while maintaining high standards of quality and precision tailored to each case.

cs.HC↗

arXiv2024

User Feedback in Continuous Software Engineering: Revealing the State-of-Practice

Context: Organizations opt for continuous delivery of incremental updates to deal with uncertainty and minimize waste. However, applying continuous engineering (CSE) practices requires a continuous feedback loop with input from customers and end-users. Challenges: It becomes increasingly challenging to apply traditional requirements elicitation and validation techniques with ever-shrinking software delivery cycles. At the same time, frequent deliveries generate an abundance of usage data and telemetry informing engineering teams of end-user behavior. The literature describing how practitioners work with user feedback in CSE, is limited. Objectives: We aim to explore the state of practice related to utilization of user feedback in CSE. Specifically, what practices are used, how, and the shortcomings of these practices. Method: We conduct a qualitative survey and report analysis from 21 interviews in 13 product development companies. We apply thematic and cross-case analysis to interpret the data. Results: Based on our earlier work we suggest a conceptual model of how user feedback is utilized in CSE. We further report the identified challenges with the continuous collection and analysis of user feedback and identify implications for practice. Conclusions: Companies use a combination of qualitative and quantitative methods to infer end-user preferences. At the same time, continuous collection, analysis, interpretation, and use of data in decisions are problematic. The challenges pertain to selecting the right metrics and analysis techniques, resource allocation, and difficulties in accessing vaguely defined user groups. Our advice to practitioners in CSE is to ensure sufficient resources and effort for interpretation of the feedback, which can be facilitated by telemetry dashboards.

cs.SE↗

arXiv2023

Knowns and Unknowns: An Experience Report on Discovering Tacit Knowledge of Maritime Surveyors

Context: Requirements elicitation is an essential activity to ensure that systems provide the necessary functionality to users, and that they are fit for purpose. In addition to traditional `reductionist' techniques, the use of observations and ethnography-style techniques have been proposed to identify requirements. Research Problem: One frequently heard issue with observational techniques is that they are costly to use, as developers would lose considerable time to partake, and also depend on luck in identifying requirements. Very few experience reports exist to evaluate observational techniques in practice. Results: In this experience report, we draw on several data sources, covering insights from both developers and users. The data were collected through 9 interviews with users and developers, and over 80 hours of observation of prospective users in the maritime domain. We capture `knowns' and `unknowns' from both developers and users, and highlight the importance of observational studies. Contribution: While observational techniques are costly to use, we conclude that essential information is uncovered, which is key for developers to understand system users and their concerns.

cs.SE↗

arXiv2022

Coordination Strategies When Working from Anywhere: A Case Study of Two Agile Teams

Effective coordination is the key to successful agile teams. They rely on frequent interactions and mutual adjustment to manage dependencies between activities, which traditionally has been solved by co-locating the team. As the world is adjusting to post-covid work-life, companies are moving towards a work-from-anywhere approach where workers can choose to what degree they want to work from home or office. However, little is known about coordination in such a context. We report findings on developers' emerging strategies when working-from-anywhere, from an exploratory case study in Norway, including eight interviews. Our study shows that new strategies for mutual adjustment emerged as teams experimented with different tools and approaches: developers chose tasks according to location, tasks with vague requirements are performed collocated while individual tasks requiring focus are best performed at home; large meetings are virtual, preserving co-located time for collaborative tasks; using virtual rooms to maintain unscheduled meetings as they communicate mental presence to teammates, lowering the threshold for intra-team unscheduled talks. The strategies can help organizations create a productive and effective environment for developers.

cs.SE↗

arXiv2021

Invisible Data Curation Practices: A Case Study from Facility Management

Facility management, which concerns the administration, operations, and mainte-nance of buildings, is a sector undergoing significant changes while becoming digitalized and data driven. In facility management sector, companies seek to ex-tract value from data about their buildings. As a consequence, craftsmen, such as janitors, are becoming involved in data curation. Data curation refers to activities related to cleaning, assembling, setting up, and stewarding data to make them fit existing templates. Craftsmen in facility management, despite holding a pivotal role for successful data curation in the domain, are understudied and disregarded. To remedy this, our holistic case study investigates how janitors' data curation practices shape the data being produced in three facility management organiza-tions. Our findings illustrate the unfortunate that janitors are treated more like a sensor than a human data curator. This treatment makes them less engaged in data curation, and hence do not engage in a much necessary correction of essential fa-cility data. We apply the conceptual lens of invisible work - work that blends into the background and is taken for granted - to explain why this happens and how data comes to be. The findings also confirm the usefulness of a previously pro-posed analytical framework by using it to interpret data curation practices within facility management. The paper contributes to practitioners by proposing training and education in data curation.

cs.CY↗

arXiv2021

Using Guilds to Foster Internal Startups in Large Organizations: A case study

Software product innovation in large organizations is fundamentally chal-lenging because of restrained freedom and flexibility to conduct experi-ments. As a response, large agile companies form internal startups to initiate employ-driven innovation, inspired by Lean startup. This case study investi-gates how communities of practice support five internal startups in develop-ing new software products within a large organization. We observed six communities of practice meetings, two workshops and conducted ten semi-structured interviews over the course of a year. Our findings show that a community of practice, called the Innovation guild, allowed internal startups to help each other by collectively solving problems, creating shared practic-es, and sharing knowledge. This study confirms that benefits documented in earlier research into CoPs also hold true in the context of software product innovation in large organizations. Henceforth, we suggest that similar innova-tion guilds, as described in this paper, can support large companies in the in-novation race for new software products.

cs.SE↗

arXiv2021

Employee-Driven Innovation to Fuel Internal Software Startups: Preliminary Findings

To keep up with the pace of innovation, established companies are increasingly relying on internal software startups. However, succeeding with such startups is a challenging task because internal startups need to find a balance between the interests of the company and the interest of the innovator. One approach that is argued to strengthen innovation in existing companies is employee-driven innovation (EDI). This study explores this argument by examining two internal software startups in companies aligned with the principles of EDI and with a strong focus on innovation. The preliminary findings indicate that startups with EDI are characterized by commitment towards innovation, cooperative orientation, and autonomy. The findings suggest that internal software startups may be strengthened when the parent companies practice EDI.

cs.SE↗

arXiv2021

Understanding Barriers to Internal Startups in Large Organizations: Evidence from a Globally Distributed Company

Large global companies need to speed up their innovation activities to increase competitive advantage. However, such companies' organizational structures impede their ability to capture trends they are well aware of due to bureaucracy, slow decision-making, distributed departments, and distributed processes. One way to strengthen the innovation capability is through fostering internal startups. We report findings from an embedded multiple-case study of five internal startups in a globally distributed company to identify barriers for software product innovation: late involvement of software developers, executive sponsor is missing or not clarified, yearly budgeting and planning, unclear decision-making authority, lack of digital infrastructure for experimentation and access to data from external actors. Drawing on the framework of continuous software engineering proposed by Fitzgerald and Stol, we discuss the role of BizDev in software product innovation. We suggest that lack of continuity, rather than the lack of speed, is an ultimate challenge for internal startups in large global companies.

cs.SE↗