Evaluating Change Point Detection Methods for Software Performance Regression Analysis

Diego Elias Costa; Michele Tucci; Luca Traini; Daniele Di Pompeo; Thomas Bach; Francois Farquet; David Daly; Simon Eismann; Petr Tůma; Vittorio Cortellessa; André van Hoorn

arXiv · 2610.09023

Evaluating Change Point Detection Methods for Software Performance Regression Analysis

Diego Elias Costa·Michele Tucci·Luca Traini·Daniele Di Pompeo·Thomas Bach·Francois Farquet·David Daly·Simon Eismann·Petr Tůma·Vittorio Cortellessa·André van Hoorn

View original source ↗Read source document ↗Export metadata ↓

Abstract

Performance issues in software systems are a critical quality issue that can erode user trust, violate service-level agreements, and ultimately affect business efficiency. Consequently, software performance engineering has shifted its focus to developing robust techniques to detect performance regressions as early as possible in the development cycle. Performance regression analysis often relies on time series of performance measurements to detect significant changes in performance behavior. Change Point Detection (CPD) methods have been widely used to automate the identification of such changes in various domains, including finance, healthcare, and performance monitoring. However, the effectiveness of these methods for software performance measurements has not been thoroughly evaluated. In this paper, we present a comprehensive study to evaluate the effectiveness of various CPD methods on real-world software performance measurement datasets. We start by collecting performance measurement data from three large software systems and characterizing the unique properties of performance time series data. Then, we undertake a large-scale effort to annotate and evaluate the consistency of human annotators' identification of potential performance changes. Thereafter, we evaluate the accuracy of twelve distinct CPD methods in detecting potential performance changes, providing insights into their applicability and effectiveness in software performance regression analysis.

Explore related subjects

cs.SE

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Diego Elias Costa, Michele Tucci, Luca Traini, Daniele Di Pompeo, Thomas Bach, Francois Farquet, David Daly, Simon Eismann, Petr Tůma, Vittorio Cortellessa, André van Hoorn. 2026-10-06. Evaluating Change Point Detection Methods for Software Performance Regression Analysis. https://arxiv.org/abs/2610.09023

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Enhancing Knowledge Density via Hierarchical Context Pruning for Retrieval-Augmented Repository-Level Code Completion

Retrieval-Augmented Generation (RAG) has advanced repository-level code completion by retrieving external code snippets, API documentation, or type definitions to expand the model's contextual awareness. However, under the finite token budget of the context window, existing retrieval-augmented methods often saturate the budget with highly similar or even redundant code, which not only wastes tokens but also suppresses critical cross-file dependencies, preventing synergistic interaction between semantically similar reference snippets and cross-file dependency definitions. Therefore, further gains are limited by low knowledge density in the retrieved context rather than by model capacity. We propose SaraCoder, a retrieval-augmented framework that enhances knowledge density via hierarchical context pruning under a fixed context budget. A Hierarchical Feature Optimization module prunes an over-complete candidate pool in four stages. The freed tokens are then allocated to an External-Aware Identifier Disambiguator, which selectively injects only the most necessary cross-file dependency definitions. SaraCoder transforms context construction into a budget-aware optimization problem, balancing reference snippets and cross-file dependencies to build a context that is both knowledge-dense and semantically complete. Experiments on CrossCodeEval and RepoEval-Updated, across multiple backbone code LLMs, show that SaraCoder consistently outperforms strong RAG baselines such as GraphCoder and Repocoder in both code-level and identifier-level metrics, without increasing token usage, and yields synergistic gains when combined with other cross-file methods.

cs.SE↗

arXiv2026

Multi-LLM Orchestration for High-Quality Code Generation: Exploiting Complementary Model Strengths

Large Language Models (LLMs) are central to automated code generation, yet most approaches use one model at every stage, from writing code to repairing and optimizing it. LLMs have complementary strengths: their failures do not fully overlap across programming languages, algorithmic categories, and development stages. For example, in Python, the model with the highest code-generation accuracy ranks last at code repair. Using one model to generate code, another to repair it, and a third to refine it can therefore achieve higher correctness and better execution performance than a single model throughout. We propose PerfOrch, a multi-LLM multi-agent system with four agents for categorization, generation, debugging, and refinement. The generation, debugging, and refinement agents select models through a Memory module that ranks candidate LLMs by their measured performance for each stage, programming language, and algorithmic category. We evaluate PerfOrch with five candidate LLMs on 4,640 problems from four benchmarks covering five programming languages (HumanEval-X, EffiBench-X, BigCodeBench, and APPS) against single-model direct generation, single-model generate-debug-refine pipelines (PerfCodeGen), and the state-of-the-art multi-agent systems MapCoder and CodeSIM. PerfOrch achieves the highest pass@1 in all twelve benchmark-language settings, exceeding the strongest baseline by an average of 8.19 percentage points, and reduces execution time for an average of 75.03% of solved problems with an average speedup of 1.18x. The Memory rankings remain effective on three unseen benchmarks without re-profiling. Sequential refinement matches exhaustive execution-time refinement coverage with 66% fewer refinement-stage tokens (57% for average-memory optimization). The complementary strengths persist after all five LLMs are replaced with their current-generation successors.

cs.SE↗

arXiv2026

Smart Hiring Redefined: An Intelligent Recruitment Management Platform

Against the backdrop of deepening digital and intelligent transformation in human resource management, traditional recruitment models struggle to fully meet enterprises' growing demand for precise talent acquisition due to limited efficiency, high costs, and information asymmetry. As a vital tool for optimizing recruitment processes, reducing labor and time costs, and enhancing core competitiveness, intelligent recruitment management systems have become an indispensable component of modern organizational talent strategies. Compared with the labor intensive tasks of resume screening, candidate position matching, and interview coordination in traditional manual recruitment, intelligent recruitment systems significantly enhance the efficiency and accuracy of the hiring process through automation and data driven approaches. These systems enable rapid parsing of massive resume volumes, intelligent matching of candidates to positions, and automated scheduling of interview processes. This substantially reduces the workload on human resources departments while improving recruitment quality and response speed. This research leverages the Java technology framework to design and implement an intelligent recruitment management system tailored for campus recruitment scenarios. The system establishes a collaborative platform connecting students, enterprises, and administrators through information technology and intelligent solutions, offering comprehensive functionalities including job posting distribution, resume submission, candidate position matching, and process management. Guided by the vision of Smart Campus Recruitment, the project delivers a more convenient job seeking experience for students and provides enterprises with more efficient talent screening and recruitment management services, thereby driving high quality development in university enterprise collaboration.

cs.SE↗