arXiv subjects

Willem Meijer

Publications and source records attributed to Willem Meijer.

5 recordsLinked to original sources

arXiv2026

Data-aware Static Analysis: Improving Detection of Semantic Faults in Machine Learning Code Using Data Characteristics

Semantic faults specific to the use of machine learning models are a common problem for machine learning developers, causing suboptimal predictions, high computational cost, or incorrect outputs. For example, one may erroneously use unscaled data to train a scale-sensitive model. Machine learning developers detect these faults after training their models and manually analyzing the results, making it an inefficient process. We propose a novel data-aware static analysis approach to detect semantic faults in machine learning code, allowing developers to reveal these bugs while writing code instead of after training the model. Our approach uses combined data and control flow analysis, and API contracts, enabling data-aware reasoning about machine learning code at a high level of abstraction. We highlight the potential of our solution by analyzing a sample of real-world machine learning notebooks, finding that we can detect faults that require a data-aware approach.

cs.SE↗

arXiv2026

Are We Lost in the Woods? Detecting Silent Semantic Faults for Random Forest Classifiers with Data-informed Static Analysis

While machine learning (ML) software necessitates effective quality assurance, ML engineers still encounter silent semantic faults, such as imbalanced datasets, that degrade prediction performance without apparent symptoms. These faults are typically detected after expensive training cycles, causing significant resource waste. We propose a data-informed static analysis technique to detect silent semantic faults in ML scripts that use the popular random forest classifier. Our approach extracts ML pipelines into directed acyclic graphs and evaluates them against formalized API contracts to detect structural, data, and hyperparameter faults. Our analysis uses aggregated data properties, enabling fault detection even when datasets are inaccessible due to confidentiality restrictions. We implemented this technique in an open-source tool, dille, and evaluated it on real-world Kaggle notebooks that use the random forest classifier. Our results demonstrate that the tool identifies relevant semantic faults with 91% precision and sub-second runtime overhead, making it suitable for integration into integrated development environments, agentic workflows, and continuous integration pipelines. Our empirical study reveals that 12% to 18% of existing ML notebooks that use the random forest classifier are affected by silent semantic faults, highlighting the immediate practical utility of data-informed static analysis in reducing the burden of ML debugging.

cs.SE↗

arXiv2024

Why do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks

Jupyter notebooks have become central in data science, integrating code, text and output in a flexible environment. With the rise of machine learning (ML), notebooks are increasingly used for prototyping and data analysis. However, due to their dependence on complex ML libraries and the flexible notebook semantics that allow cells to be run in any order, notebooks are susceptible to software bugs that may lead to program crashes. This paper presents a comprehensive empirical study focusing on crashes in publicly available Python ML notebooks. We collect 64,031 notebooks containing 92,542 crashes from GitHub and Kaggle, and manually analyze a sample of 746 crashes across various aspects, including crash types and root causes. Our analysis identifies unique ML-specific crash types, such as tensor shape mismatches and dataset value errors that violate API constraints. Additionally, we highlight unique root causes tied to notebook semantics, including out-of-order execution and residual errors from previous cells, which have been largely overlooked in prior research. Furthermore, we identify the most error-prone ML libraries, and analyze crash distribution across ML pipeline stages. We find that over 40% of crashes stem from API misuse and notebook-specific issues. Crashes frequently occur when using ML libraries like TensorFlow/Keras and Torch. Additionally, over 70% of the crashes occur during data preparation, model training, and evaluation or prediction stages of the ML pipeline, while data visualization errors tend to be unique to ML notebooks.

cs.SE↗

arXiv2024

Ecosystem-wide influences on pull request decisions: insights from NPM

The pull-based development model facilitates global collaboration within open-source software projects. However, whereas it is increasingly common for software to depend on other projects in their ecosystem, most research on the pull request decision-making process explored factors within projects, not the broader software ecosystem they comprise. We uncover ecosystem-wide factors that influence pull request acceptance decisions. We collected a dataset of approximately 1.8 million pull requests and 2.1 million issues from 20,052 GitHub projects within the NPM ecosystem. Of these, 98% depend on another project in the dataset, enabling studying collaboration across dependent projects. We employed social network analysis to create a collaboration network in the ecosystem, and mixed effects logistic regression and random forest techniques to measure the impact and predictive strength of the tested features. We find that gaining experience within the software ecosystem through active participation in issue-tracking systems, submitting pull requests, and collaborating with pull request integrators and experienced developers benefits all open-source contributors, especially project newcomers. These results are complemented with an exploratory qualitative analysis of 538 pull requests. We find that developers with ecosystem experience make different contributions than users without. Zooming in on a subset of 111 pull requests with clear ecosystem involvement, we find 3 overarching and 10 specific reasons why developers involve ecosystem projects in their pull requests. The results show that combining ecosystem-wide factors with features studied in previous work to predict the outcome of pull requests reached an overall F1 score of 0.92. However, the outcomes of pull requests submitted by newcomers are harder to predict.

cs.SE↗

arXiv2024

Experimental evaluation of architectural software performance design patterns in microservices

Microservice architectures and design patterns enhance the development of large-scale applications by promoting flexibility. Industrial practitioners perceive the importance of applying architectural patterns but they struggle to quantify their impact on system quality requirements. Our research aims to quantify the effect of design patterns on system performance metrics, e.g., service latency and resource utilization, even more so when the patterns operate in real-world environments subject to heterogeneous workloads. We built a cloud infrastructure to host a well-established benchmark system that represents our test bed, complemented by the implementation of three design patterns: Gateway Aggregation, Gateway Offloading, Pipe and Filters. Real performance measurements are collected and compared with model-based predictions that we derived as part of our previous research, thus further consolidating the actual impact of these patterns. Our results demonstrate that, despite the difficulty to parameterize our benchmark system, model-based predictions are in line with real experimentation, since the performance behaviors of patterns, e.g., bottleneck switches, are mostly preserved. In summary, this is the first work that experimentally demonstrates the performance behavior of microservices-based architectural patterns. Results highlight the complexity of evaluating the performance of design patterns and emphasize the need for complementing theoretical models with empirical data.

cs.SE↗