Searcharxiv⌕ Search

arXiv subjects

André van Hoorn

Publications and source records attributed to André van Hoorn.

8 recordsLinked to original sources
arXiv2026

Evaluating Change Point Detection Methods for Software Performance Regression Analysis

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

cs.SE↗
arXiv2024

Introducing Interactions in Multi-Objective Optimization of Software Architectures

Software architecture optimization aims to enhance non-functional attributes like performance and reliability while meeting functional requirements. Multi-objective optimization employs metaheuristic search techniques, such as genetic algorithms, to explore feasible architectural changes and propose alternatives to designers. However, this resource-intensive process may not always align with practical constraints. This study investigates the impact of designer interactions on multi-objective software architecture optimization. Designers can intervene at intermediate points in the fully automated optimization process, making choices that guide exploration towards more desirable solutions. Through several controlled experiments as well as an initial user study (14 subjects), we compare this interactive approach with a fully automated optimization process, which serves as a baseline. The findings demonstrate that designer interactions lead to a more focused solution space, resulting in improved architectural quality. By directing the search towards regions of interest, the interaction uncovers architectures that remain unexplored in the fully automated process. In the user study, participants found that our interactive approach provides a better trade-off between sufficient exploration of the solution space and the required computation time.

cs.SE↗
arXiv2024

Overhead Measurement Noise in Different Runtime Environments

In order to detect performance changes, measurements are performed with the same execution environment. In cloud environments, the noise from different processes running on the same cluster nodes might change measurement results and thereby make performance changes hard to measure. The benchmark MooBench determines the overhead of different observability tools and is executed continuously. In this study, we compare the suitability of different execution environments to benchmark the observability overhead using MooBench. To do so, we compare the execution times and standard deviation of MooBench in a cloud execution environment to three bare-metal execution environments. We find that bare metal servers have lower runtime and standard deviation for multi-threaded MooBench execution. Nevertheless, we see that performance changes up to 4.41% are detectable by GitHub actions, as long as only sequential workloads are examined.

cs.PF↗
arXiv2023

Learning From Each Other: How Are Architectural Mistakes Communicated in Industry?

Context. Own experiences and faulty decisions can be an important source of information for software architects. The experiences and mistakes of other architects can also be valuable information sources. Goal. Under the assumption that the knowledge about faulty decisions, i.e., mistakes, regarding software architecture is not shared adequately in practice, this work qualitatively investigates the handling and particularly communication of those mistakes by software architects. Method. We conducted a grounded-theory study in which we interviewed ten German software architects from various domains. Results. We identified software architects' definitions of architectural mistakes, their handling of these mistakes, and their preferred communication strategies regarding these mistakes. We found that architects communicate mistakes mainly within their project teams and seldom within or across companies. Conclusions. We derived strategies to make learning and prevention of mistakes more effective. To share experiences and knowledge beyond architects' peer groups, companies should invest more effort in discussing mistakes more consciously and create an environment where mistakes can be discussed openly.

cs.SE↗
arXiv2021

A Case Study on the Stability of Performance Tests for Serverless Applications

Context. While in serverless computing, application resource management and operational concerns are generally delegated to the cloud provider, ensuring that serverless applications meet their performance requirements is still a responsibility of the developers. Performance testing is a commonly used performance assessment practice; however, it traditionally requires visibility of the resource environment. Objective. In this study, we investigate whether performance tests of serverless applications are stable, that is, if their results are reproducible, and what implications the serverless paradigm has for performance tests. Method. We conduct a case study where we collect two datasets of performance test results: (a) repetitions of performance tests for varying memory size and load intensities and (b) three repetitions of the same performance test every day for ten months. Results. We find that performance tests of serverless applications are comparatively stable if conducted on the same day. However, we also observe short-term performance variations and frequent long-term performance changes. Conclusion. Performance tests for serverless applications can be stable; however, the serverless model impacts the planning, execution, and analysis of performance tests.

cs.DC↗
arXiv2018

How is Performance Addressed in DevOps? A Survey on Industrial Practices

DevOps is a modern software engineering paradigm that is gaining widespread adoption in industry. The goal of DevOps is to bring software changes into production with a high frequency and fast feedback cycles. This conflicts with software quality assurance activities, particularly with respect to performance. For instance, performance evaluation activities -- such as load testing -- require a considerable amount of time to get statistically significant results. We conducted an industrial survey to get insights into how performance is addressed in industrial DevOps settings. In particular, we were interested in the frequency of executing performance evaluations, the tools being used, the granularity of the obtained performance data, and the use of model-based techniques. The survey responses, which come from a wide variety of participants from different industry sectors, indicate that the complexity of performance engineering approaches and tools is a barrier for wide-spread adoption of performance analysis in DevOps. The implication of our results is that performance analysis tools need to have a short learning curve, and should be easy to integrate into the DevOps pipeline.

cs.SE↗
arXiv2018

An Efficient Method for Uncertainty Propagation in Robust Software Performance Estimation

Software engineers often have to estimate the performance of a software system before having full knowledge of the system parameters, such as workload and operational profile. These uncertain parameters inevitably affect the accuracy of quality evaluations, and the ability to judge if the system can continue to fulfil performance requirements if parameter results are different from expected. Previous work has addressed this problem by modelling the potential values of uncertain parameters as probability distribution functions, and estimating the robustness of the system using Monte Carlo-based methods. These approaches require a large number of samples, which results in high computational cost and long waiting times. To address the computational inefficiency of existing approaches, we employ Polynomial Chaos Expansion (PCE) as a rigorous method for uncertainty propagation and further extend its use to robust performance estimation. The aim is to assess if the software system is robust, i.e., it can withstand possible changes in parameter values, and continue to meet performance requirements. PCE is a very efficient technique, and requires significantly less computations to accurately estimate the distribution of performance indices. Through three very different case studies from different phases of software development and heterogeneous application domains, we show that PCE can accurately (>97\%) estimate the robustness of various performance indices, and saves up to 225 hours of performance evaluation time when compared to Monte Carlo Simulation.

cs.SE↗