arXiv subjects

Dietmar Pfahl

Publications and source records attributed to Dietmar Pfahl.

At least 19 recordsLinked to original sources

arXiv2026

In the Driver's Seat: A Multi-Company Study on the Reality of Autonomous Driving System Testing

Autonomous driving systems (ADS) are rapidly advancing and increasingly deployed in real-world applications. This creates growing demands for effective testing to ensure system functionality and safety. However, ADS testing remains complex and lacks well-established standards for scenario selection, performance evaluation, and acceptance criteria. To better understand current ADS testing practices and challenges, we conducted an interview study with experts working on ADS development and testing in nine companies from six different countries. Through thematic analysis, we synthesized industrial testing practices, challenges, potential solutions, future trends, and proposed an evidence-centered closed-loop testing framework for ADS testing. Our findings show that current practices primarily focus on scenario-based and X-in-the-loop testing approaches, supported by diverse tools, metrics, benchmarks, and testing strategies. The participants highlighted major challenges related to scenario realism, scenario coverage, simulation fidelity, and acceptance criteria, while also discussing potential solutions such as the use of AI, world models, and end-to-end approaches. Furthermore, participants envisioned future ADS testing to become more automated, data-driven, and transparent across the industry. Overall, this study provides a comprehensive industry-grounded overview of ADS testing, proposes an evidence-centered closed-loop testing framework to provide actionable guidance for ADS testing, and outlines important directions for future research and practice.

cs.SE↗

arXiv2026

The challenge of generating and evolving real-life like synthetic test data without accessing real-world raw data -- a Systematic Review

Background: High-level system testing of applications that use data from e-Government services as input requires test data that is real-life-like but where the privacy of personal information is guaranteed. Applications with such strong requirement include information exchange between countries, medicine, banking, etc. This review aims to synthesize the current state-of-the-practice in this domain. Objectives: The objective of this Systematic Review is to identify existing approaches for creating and evolving synthetic test data without using real-life raw data. Methods: We followed well-known methodologies for conducting systematic literature reviews, including the ones from Kitchenham as well as guidelines for analysing the limitations of our review and its threats to validity. Results: A variety of methods and tools exist for creating privacy-preserving test data. Our search found 1,013 publications in IEEE Xplore, ACM Digital Library, and SCOPUS. We extracted data from 75 of those publications and identified 37 approaches that answer our research question partly. A common prerequisite for using these methods and tools is direct access to real-life data for data anonymization or synthetic test data generation. Nine existing synthetic test data generation approaches were identified that were closest to answering our research question. Nevertheless, further work would be needed to add the ability to evolve synthetic test data to the existing approaches. Conclusions: None of the publications really covered our requirements completely, only partially. Synthetic test data evolution is a field that has not received much attention from researchers but needs to be explored in Digital Government Solutions, especially since new legal regulations are being placed in force in many countries.

cs.LG↗

arXiv2025

Evaluating the impact of code smell refactoring on the energy consumption of Android applications

Energy consumption of mobile apps is a domain that is receiving a lot of attention from researchers. Recent studies indicate that the energy consumption of mobile devices could be improved by improving the quality of mobile apps. Frequent refactoring is one way of achieving this goal. In this paper, we explore the performance and energy impact of several common code refactorings in Android apps. Experimental results indicate that some code smell refactorings positively impact the energy consumption of Android apps. Refactoring of the code smells "Duplicated code" and "Type checking" reduce energy consumption by up to 10.8%. Significant reduction in energy consumption, however, does not seem to be directly related to the increase or decrease of execution time. In addition, the energy impact over permutations of code smell refactorings in the selected Android apps was small. When analyzing the order in which refactorings were made across code smell types, it turned out that some permutations resulted in a reduction and some in an increase of energy consumption for the analyzed apps. More research needs to be done to investigate how factors like size and age of software apps, experience, and number of contributors to app development correlate with (a) the number and type of code smells found and (b) the impact of energy consumption and performance after refactoring.

cs.SE↗

arXiv2025

How Effectively Do LLMs Extract Feature-Sentiment Pairs from App Reviews?

Automatic analysis of user reviews to understand user sentiments toward app functionality (i.e. app features) helps align development efforts with user expectations and needs. Recent advances in Large Language Models (LLMs) such as ChatGPT have shown impressive performance on several new tasks without updating the model's parameters i.e. using zero or a few labeled examples, but the capabilities of LLMs are yet unexplored for feature-specific sentiment analysis. The goal of our study is to explore the capabilities of LLMs to perform feature-specific sentiment analysis of user reviews. This study compares the performance of state-of-the-art LLMs, including GPT-4, ChatGPT, and different variants of Llama-2 chat, against previous approaches for extracting app features and associated sentiments in zero-shot, 1-shot, and 5-shot scenarios. The results indicate that GPT-4 outperforms the rule-based SAFE by 17% in f1-score for extracting app features in the zero-shot scenario, with 5-shot further improving it by 6%. However, the fine-tuned RE-BERT exceeds GPT-4 by 6% in f1-score. For predicting positive and neutral sentiments, GPT-4 achieves f1-scores of 76% and 45% in the zero-shot setting, which improve by 7% and 23% in the 5-shot setting, respectively. Our study conducts a thorough evaluation of both proprietary and open-source LLMs to provide an objective assessment of their performance in extracting feature-sentiment pairs.

cs.CL↗

arXiv2025

Teaching Simulation as a Research Method in Empirical Software Engineering

The chapter supports educators and postgraduate students in understanding the role of simulation in software engineering research based on the authors' experience. This way, it includes a background positioning simulation-based studies in software engineering research, the proposition of learning objectives for teaching simulation as a research method, and presents our experience when teaching simulation concepts and practice. For educators, it further provides learning objectives when teaching simulation, considering the current state of the art in software engineering research and the necessary guidance and recommended learning activities to achieve these objectives. For students, it drives the learning path for those interested in learning this method but had no opportunity to engage in an entire course on simulation in the context of empirical research.

cs.SE↗

arXiv2025

An LSTM-based Test Selection Method for Self-Driving Cars

Self-driving cars require extensive testing, which can be costly in terms of time. To optimize this process, simple and straightforward tests should be excluded, focusing on challenging tests instead. This study addresses the test selection problem for lane-keeping systems for self-driving cars. Road segment features, such as angles and lengths, were extracted and treated as sequences, enabling classification of the test cases as "safe" or "unsafe" using a long short-term memory (LSTM) model. The proposed model is compared against machine learning-based test selectors. Results demonstrated that the LSTM-based method outperformed machine learning-based methods in accuracy and precision metrics while exhibiting comparable performance in recall and F1 scores. This work introduces a novel deep learning-based approach to the road classification problem, providing an effective solution for self-driving car test selection using a simulation environment.

cs.RO↗

arXiv2023

Towards a Complete Metamorphic Testing Pipeline

Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT). These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. However, achieving full automation in generating, selecting, and understanding MR violations poses challenges. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes. In the MR generation phase, we explore automated techniques that utilise a domain-specific language to generate and describe MRs. The MR constraint definition focuses on capturing the nuances of MR applicability by defining constraints. These constraints help identify the specific conditions under which MRs are expected to hold. The evaluation and validation involve conducting empirical studies to assess the effectiveness of the developed methods and validate their applicability in real-world regression testing scenarios. Through this research, we aim to advance the automation of MR generation, enhance the understanding of MR violations, and facilitate their effective application in regression testing.

cs.SE↗

arXiv2023

Towards Automatic Generation of Amplified Regression Test Oracles

Regression testing is crucial in ensuring that pure code refactoring does not adversely affect existing software functionality, but it can be expensive, accounting for half the cost of software maintenance. Automated test case generation reduces effort but may generate weak test suites. Test amplification is a promising solution that enhances tests by generating additional or improving existing ones, increasing test coverage, but it faces the test oracle problem. To address this, we propose a test oracle derivation approach that uses object state data produced during System Under Test (SUT) test execution to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT's intended behaviour. Our preliminary evaluation shows that the proposed approach can enhance the detection of behaviour changes substantially, providing initial evidence of its effectiveness.

cs.SE↗

arXiv2023

Exploring a Test Data-Driven Method for Selecting and Constraining Metamorphic Relations

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

cs.SE↗

arXiv2023

Analysis of Library Dependency Networks of Package Managers Used in iOS Development

Reusing existing solutions in the form of third-party libraries is common practice when writing software. Package managers are used to manage dependencies to third-party libraries by automating the process of installing and updating the libraries. Library dependencies themselves can have dependencies to other libraries creating a dependency network with several levels of indirections. The library dependency network in the Swift ecosystem encompasses libraries from CocoaPods, Carthage and Swift Package Manager (PM). These package managers are used when developing, for example, iOS or Mac OS applications in Swift and Objective-C. We provide the first analysis of the library dependency network evolution in the Swift ecosystem. Although CocoaPods is the package manager with the biggest set of libraries, the difference to other package managers is not as big as expected. The youngest package manager and official package manager for Swift, Swift PM, is becoming more and more popular, resulting in a gradual slow-down of the growth of the other two package managers. When analyzing direct and transitive dependencies, we found that the mean total number of dependencies is lower in the Swift ecosystem compared to many other ecosystems. Still, the total number of dependencies shows a clear growing trend over the last five years.

cs.SE↗

arXiv2023

Vulnerability Propagation in Package Managers Used in iOS Development

Although using third-party libraries is common practice when writing software, vulnerabilities may be found even in well-known libraries. Detected vulnerabilities are often fixed quickly in the library code. The easiest way to include these fixes in a dependent software application, is to update the used library version. Package managers provide automated solutions for updating library dependencies. However, library dependencies can have dependencies to other libraries resulting in a dependency network with several levels of indirections. Assessing vulnerability risks induced by dependency networks is a non-trivial task for software developers. The library dependency network in the Swift ecosystem encompasses libraries from CocoaPods, Carthage and Swift Package Manager. We analysed how vulnerabilities propagate in the library dependency network of the Swift ecosystem, how vulnerable dependencies could be fixed via dependency upgrades, and if third party vulnerability analysis could be made more precise given public information on these vulnerabilities. We found that only 5.9% of connected libraries had a direct or transitive dependency to a vulnerable library. Although we found that most libraries with publicly reported vulnerabilities are written in C, the highest impact of publicly reported vulnerabilities originated from libraries written in native iOS languages. We found that around 30% of vulnerable dependencies could have been fixed via upgrading the library dependency. In case of critical vulnerabilities and latest library versions, over 70% of vulnerable dependencies would have been fixed via a dependency upgrade. Lastly, we checked whether the analysis of vulnerable dependency use could be refined using publicly available information on the code location (method or class) of a reported vulnerability. We found that such information is not available most of the time.

cs.SE↗

arXiv2023

An Exploratory Study on the Evidence of Hackathons' Role in Solving OSS Newcomers' Challenges

Background: OSS projects face various challenges. One major challenge is to onboard and integrate newcomers to the project. Aim: We aim to understand and discuss the challenges newcomers face when joining an OSS project and present evidence on how hackathons can mitigate those challenges. Method: We conducted two searches on digital libraries to (1) explore challenges faced by newcomers to join OSS projects, and (2) collect evidence on how hackathons were used to address them. We defined four evidence categories (positive, inconclusive, and no evidence) to classify evidence how hackathons address challenges. In addition, we investigated whether a hackathon event was related to an OSS project or not. Result: We identified a range of newcomer challenges that were successfully addressed using hackathons. However, not all of the solutions we identified were applied in the context of OSS. Conclusion: There seems to be potential in using hackathons to overcome newcomers' challenges in OSS projects and allow them to integrate faster into the project.

cs.SE↗

arXiv2023

Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations

Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

cs.SE↗

arXiv2022

Dataset: Dependency Networks of Open Source Libraries Available Through CocoaPods, Carthage and Swift PM

Third party libraries are used to integrate existing solutions for common problems and help speed up development. The use of third party libraries, however, can carry risks, for example through vulnerabilities in these libraries. Studying the dependency networks of package managers lets us better understand and mitigate these risks. So far, the dependency networks of the three most important package managers of the Apple ecosystem, CocoaPods, Carthage and Swift PM, have not been studied. We analysed the dependencies for all publicly available open source libraries up to December 2021 and compiled a dataset containing the dependency networks of all three package managers. The dependency networks can be used to analyse how vulnerabilities are propagated through transitive dependencies. In order to ease the tracing of vulnerable libraries we also queried the NVD database and included publicly reported vulnerabilities for these libraries in the dataset.

cs.SE↗

arXiv2022

A Replication Study on Predicting Metamorphic Relations at Unit Testing Level

Metamorphic Testing (MT) addresses the test oracle problem by examining the relations between inputs and outputs of test executions. Such relations are known as Metamorphic Relations (MRs). In current practice, identifying and selecting suitable MRs is usually a challenging manual task, requiring a thorough grasp of the SUT and its application domain. Thus, Kanewala et al. proposed the Predicting Metamorphic Relations (PMR) approach to automatically suggest MRs from a list of six pre-defined MRs for testing newly developed methods. PMR is based on a classification model trained on features extracted from the control-flow graph (CFG) of 100 Java methods. In our replication study, we explore the generalizability of PMR. First, we rebuild the entire preprocessing and training pipeline and repeat the original study in a close replication to verify the reported results and establish the basis for further experiments. Second, we perform a conceptual replication to explore the reusability of the PMR model trained on CFGs from Java methods in the first step for functionally identical methods implemented in Python and C++. Finally, we retrain the model on the CFGs from the Python and C++ methods to investigate the dependence on programming language and implementation details. We were able to successfully replicate the original study achieving comparable results for the Java methods set. However, the prediction performance of the Java-based classifiers significantly decreases when applied to functionally equivalent Python and C++ methods despite using only CFG features to abstract from language details. Since the performance improved again when the classifiers were retrained on the CFGs of the methods written in Python and C++, we conclude that the PMR approach can be generalized, but only when classifiers are developed starting from code artefacts in the used programming language.

cs.SE↗

arXiv2022

Using Source Code Metrics for Predicting Metamorphic Relations at Method Level

Metamorphic testing (TM) examines the relations between inputs and outputs of test runs. These relations are known as metamorphic relations (MR). Currently, MRs are handpicked and require in-depth knowledge of the System Under Test (SUT), as well as its problem domain. As a result, the identification and selection of high-quality MRs is a challenge. \citeauthor{PMR1} suggested the Predicting Metamorphic Relations (PMR) approach for automatic prediction of applicable MRs picked from a predefined list. PMR is based on a Support Vector Machine (SVM) model using features derived from the Control Flow Graphs (CFGs) of 100 Java methods. The original study of \citeauthor{PMR1} showed encouraging results, but developing classification models from CFG-related features is costly. In this paper, we aim at developing a PMR approach that is less costly without losing performance. We complement the original PMR approach by considering other than CFG-related features. We define 21 features that can be directly extracted from source code and build several classifiers, including SVM models. Our results indicate that using the original CFG-based method-level features, in particular for a SVM with random walk kernel (RWK), achieve better predictions in terms of AUC-ROC for most of the candidate MRs than our models. However, for one of the candidate MRs, using source code features achieved the best AUC-ROC result (greater than 0.8).

cs.SE↗

arXiv2021

What Makes Agile Software Development Agile?

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders' collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15%). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

cs.SE↗

arXiv2021

Towards the statistical construction of hybrid development methods

Hardly any software development process is used as prescribed by authors or standards. Regardless of company size or industry sector, a majority of project teams and companies use hybrid development methods (short: hybrid methods) that combine different development methods and practices. Even though such hybrid methods are highly individualized, a common understanding of how to systematically construct synergetic practices is missing. In this article, we make a first step towards a statistical construction procedure for hybrid methods. Grounded in 1467 data points from a large-scale practitioner survey, we study the question: What are hybrid methods made of and how can they be systematically constructed? Our findings show that only eight methods and few practices build the core of modern software development. Using an 85% agreement level in the participants' selections, we provide examples illustrating how hybrid methods can be characterized by the practices they are made of. Furthermore, using this characterization, we develop an initial construction procedure, which allows for defining a method frame and enriching it incrementally to devise a hybrid method using ranked sets of practice.

cs.SE↗