arXiv subjects

Jørn Eirik Betten

Publications and source records attributed to Jørn Eirik Betten.

5 recordsLinked to original sources

arXiv2026

Metamorphic Testing with the Rashomon Set: Explanation Faithfulness in Machine Learning

Multiple machine learning models can achieve near-equivalent predictive performance on the same task, yet provide divergent feature-based explanations. This is called the Rashomon effect of (explainable) machine learning, and it raises the question of which explanations, if any, are trustworthy. We propose a framework based on metamorphic testing that assesses explanation faithfulness without requiring ground-truth labels by exploring attributed feature importance from post-hoc explanation methods. Five metamorphic relations formalize expected consistency properties between model behavior and feature attributions. We apply this general framework to two tabular regression datasets and two post-hoc explainers (SHAP and LIME) to demonstrate the approach. The framework offers a practical, model-agnostic tool for selecting accurate models with reliable and trustworthy explanations.

cs.SE↗

arXiv2025

Translating the Rashomon Effect to Sequential Decision-Making Tasks

The Rashomon effect describes the phenomenon where multiple models trained on the same data produce identical predictions while differing in which features they rely on internally. This effect has been studied extensively in classification tasks, but not in sequential decision-making, where an agent learns a policy to achieve an objective by taking actions in an environment. In this paper, we translate the Rashomon effect to sequential decision-making. We define it as multiple policies that exhibit identical behavior, visiting the same states and selecting the same actions, while differing in their internal structure, such as feature attributions. Verifying identical behavior in sequential decision-making differs from classification. In classification, predictions can be directly compared to ground-truth labels. In sequential decision-making with stochastic transitions, the same policy may succeed or fail on any single trajectory due to randomness. We address this using formal verification methods that construct and compare the complete probabilistic behavior of each policy in the environment. Our experiments demonstrate that the Rashomon effect exists in sequential decision-making. We further show that ensembles constructed from the Rashomon set exhibit greater robustness to distribution shifts than individual policies. Additionally, permissive policies derived from the Rashomon set reduce computational requirements for verification while maintaining optimal performance.

cs.AI↗

arXiv2025

Prompting for Performance: Exploring LLMs for Configuring Software

Software systems usually provide numerous configuration options that can affect performance metrics such as execution time, memory usage, binary size, or bitrate. On the one hand, making informed decisions is challenging and requires domain expertise in options and their combinations. On the other hand, machine learning techniques can search vast configuration spaces, but with a high computational cost, since concrete executions of numerous configurations are required. In this exploratory study, we investigate whether large language models (LLMs) can assist in performance-oriented software configuration through prompts. We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various configurable systems, such as compilers, video encoders, and SAT solvers. Our preliminary results reveal both positive abilities and notable limitations: depending on the task and systems, LLMs can well align with expert knowledge, whereas hallucinations or superficial reasoning can emerge in other cases. These findings represent a first step toward systematic evaluations and the design of LLM-based solutions to assist with software configuration.

cs.SE↗

arXiv2025

Rashomon in the Streets: Explanation Ambiguity in Scene Understanding

Explainable AI (XAI) is essential for validating and trusting models in safety-critical applications like autonomous driving. However, the reliability of XAI is challenged by the Rashomon effect, where multiple, equally accurate models can offer divergent explanations for the same prediction. This paper provides the first empirical quantification of this effect for the task of action prediction in real-world driving scenes. Using Qualitative Explainable Graphs (QXGs) as a symbolic scene representation, we train Rashomon sets of two distinct model classes: interpretable, pair-based gradient boosting models and complex, graph-based Graph Neural Networks (GNNs). Using feature attribution methods, we measure the agreement of explanations both within and between these classes. Our results reveal significant explanation disagreement. Our findings suggest that explanation ambiguity is an inherent property of the problem, not just a modeling artifact.

cs.LG↗

arXiv2025

Reusable Test Suites for Reinforcement Learning

Reinforcement learning (RL) agents show great promise in solving sequential decision-making tasks. However, validating the reliability and performance of the agent policies' behavior for deployment remains challenging. Most reinforcement learning policy testing methods produce test suites tailored to the agent policy being tested, and their relevance to other policies is unclear. This work presents Multi-Policy Test Case Selection (MPTCS), a novel automated test suite selection method for RL environments, designed to extract test cases generated by any policy testing framework based on their solvability, diversity, and general difficulty. MPTCS uses a set of policies to select a diverse collection of reusable policy-agnostic test cases that reveal typical flaws in the agents' behavior. The set of policies selects test cases from a candidate pool, which can be generated by any policy testing method, based on a difficulty score. We assess the effectiveness of the difficulty score and how the method's effectiveness and cost depend on the number of policies in the set. Additionally, a method for promoting diversity in the test suite, a discretized general test case descriptor surface inspired by quality-diversity algorithms, is examined to determine how it covers the state space and which policies it triggers to produce faulty behaviors.

cs.SE↗