arXiv subjects

Pieter Van Gorp

Publications and source records attributed to Pieter Van Gorp.

6 recordsLinked to original sources

arXiv2026

Does Explanation Correctness Matter? Linking Computational XAI Evaluation to Human Understanding

Explainable AI (XAI) methods are commonly evaluated using functional correctness metrics, sometimes termed faithfulness or fidelity, which estimate how closely an explanation reflects the model's reasoning. Higher correctness is assumed to produce better human understanding, but this link has not been tested with controlled levels. We conducted a user study (N=200) that manipulated explanation correctness at four levels (100%, 85%, 70%, 55%) in a synthetic time series classification task where participants could not rely on domain knowledge or visual intuition. Correctness was defined against a known ground truth, not estimated from a trained model. Participants predicted a simulated AI's decisions from feature-attribution-style explanations (forward simulation), and we used their forward simulation accuracy as a proxy for understanding. Correctness affected understanding, but not at every level: forward simulation accuracy dropped at 70% and 55% relative to fully correct explanations, with no further reduction below 70% and no conclusive difference between 85% and 100%. Lower correctness reduced how many participants predicted accurately rather than lowering accuracy uniformly, and even fully correct explanations did not guarantee understanding: forward simulation accuracy was bimodal there, with only a subset of participants well above chance. In exploratory analyses, some participants predicted decisions accurately but described the wrong pattern when asked what the AI used, and self-reported ratings correlated with forward simulation accuracy only when explanations were fully correct. These findings show that not all differences in functional correctness translate to differences in human understanding, highlighting the need to validate functional metrics against human outcomes.

cs.HC↗

arXiv2025

Why Do Class-Dependent Evaluation Effects Occur with Time Series Feature Attributions? A Synthetic Data Investigation

Evaluating feature attribution methods represents a critical challenge in explainable AI (XAI), as researchers typically rely on perturbation-based metrics when ground truth is unavailable. However, recent work reveals that these evaluation metrics can show different performance across predicted classes within the same dataset. These "class-dependent evaluation effects" raise questions about whether perturbation analysis reliably measures attribution quality, with direct implications for XAI method development and evaluation trustworthiness. We investigate under which conditions these class-dependent effects arise by conducting controlled experiments with synthetic time series data where ground truth feature locations are known. We systematically vary feature types and class contrasts across binary classification tasks, then compare perturbation-based degradation scores with ground truth-based precision-recall metrics using multiple attribution methods. Our experiments demonstrate that class-dependent effects emerge with both evaluation approaches, even in simple scenarios with temporally localized features, triggered by basic variations in feature amplitude or temporal extent between classes. Most critically, we find that perturbation-based and ground truth metrics frequently yield contradictory assessments of attribution quality across classes, with weak correlations between evaluation approaches. These findings suggest that researchers should interpret perturbation-based metrics with care, as they may not always align with whether attributions correctly identify discriminating features. By showing this disconnect, our work points toward reconsidering what attribution evaluation actually measures and developing more rigorous evaluation methods that capture multiple dimensions of attribution quality.

cs.LG↗

arXiv2025

Class-Dependent Perturbation Effects in Evaluating Time Series Attributions

As machine learning models become increasingly prevalent in time series applications, Explainable Artificial Intelligence (XAI) methods are essential for understanding their predictions. Within XAI, feature attribution methods aim to identify which input features contribute the most to a model's prediction, with their evaluation typically relying on perturbation-based metrics. Through systematic empirical analysis across multiple datasets, model architectures, and perturbation strategies, we reveal previously overlooked class-dependent effects in these metrics: they show varying effectiveness across classes, achieving strong results for some while remaining less sensitive to others. In particular, we find that the most effective perturbation strategies often demonstrate the most pronounced class differences. Our analysis suggests that these effects arise from the learned biases of classifiers, indicating that perturbation-based evaluation may reflect specific model behaviors rather than intrinsic attribution quality. We propose an evaluation framework with a class-aware penalty term to help assess and account for these effects in evaluating feature attributions, offering particular value for class-imbalanced datasets. Although our analysis focuses on time series classification, these class-dependent effects likely extend to other structured data domains where perturbation-based evaluation is common.

cs.LG↗

arXiv2013

The Petri-Nets to Statecharts Transformation Case

This paper describes a case study for the sixth Transformation Tool Contest. The case is based on a mapping from Petri-Nets to statecharts (i.e., from flat process models to hierarchical ones). The case description separates a simple mapping phase from a phase that involves the step by step destruction Petri-Net elements and the corresponding construction of a hierarchy of statechart elements. Although the focus of this case study is on the comparison of the runtime performance of solutions, we also include correctness tests as well as bonus criteria for evaluating transformation language and tool features.

cs.PL↗

arXiv2013

Proceedings Sixth Transformation Tool Contest

The aim of the Transformation Tool Contest (TTC) series is to compare the expressiveness, the usability and the performance of graph and model transformation tools along a number of selected case studies. Participants want to learn about the pros and cons of each tool considering different applications. A deeper understanding of the relative merits of different tool features will help to further improve graph and model transformation tools and to indicate open problems. TTC 2013 involved 18 offline case study solutions: 6 solutions to the FlowGraphs case, 9 solutions to the Petri Nets to Statecharts case and 3 solutions to the Restructuring case. 13 of the 18 solutions have undergone a non-blind peer review before the workshop and were presented and evaluated during the workshop in Budapest. This volume contains the submissions that have passed an additional (post-workshop, blind) reviewing round.

cs.SE↗

arXiv2011

Proceedings Fifth Transformation Tool Contest

The aim of the Transformation Tool Contest (TTC) series is to compare the expressiveness, the usability and the performance of graph and model transformation tools along a number of selected case studies. Participants want to learn about the pros and cons of each tool considering different applications. A deeper understanding of the relative merits of different tool features will help to further improve graph and model transformation tools and to indicate open problems. TTC 2011 involved 25 offline case study solutions: 12 solutions to the Hello World case, 2 solutions to the GMF Model Migration case, 5 solutions to the Compiler Optimization case, and 7 solutions to the Reengineering (i.e., Program Understanding) case. This volume contains the submissions that have passed an additional (post-workshop) reviewing round.

cs.SE↗