ReBeCA: Unveiling Interpretable Behavior Hierarchy behind the Iterative Self-Reflection of Language Models with Causal Analysis

Abstract

While self-reflection can enhance language model reliability, its underlying mechanisms remain opaque, with existing analyses often yielding correlation-based insights that fail to generalize. To address this, we introduce **ReBeCA** (self-**Re**flection **Be**havior explained through **C**ausal **A**nalysis), a framework for analyzing the interpretable behavioral hierarchy governing the self-reflection outcome. By modeling self-reflection trajectories as causal graphs, ReBeCA selects observed parent candidates and evaluates their stability through a three-stage ICP-based pipeline. In a controlled Qwen3 case study, we establish three critical findings: (1) Behavioral hierarchy: Semantic behaviors of the model influence final self-reflection results hierarchically: directly or indirectly; (2) Causation matters: Generalizability in self-reflection effects is limited to just a few semantic behaviors; (3) More $\neq$ better: The confluence of seemingly positive semantic behaviors, even among direct causal factors, yield no additive gain. ICP-based verification identifies sparse causal parents achieving up to $49.6\%$ structural likelihood gains relative to the dense full-set association baseline across the studied task subsets. A controlled prompt-based behavioral intervention on a novel dataset provides out-of-distribution validation ($p = .013, η^2_\mathrm{p} = .071$). The present study focuses on the Qwen3 family under fixed-round Self-Refine.