arXiv · 2611.18611
Understanding Failures in LLM Reasoning by Learning Structured Representations of Chain-of-Thought
Abstract
Chain-of-Thought (CoT) is central to improving reasoning in large language models, yet existing evaluation methods focus on step-level signals and ignore the underlying structure of reasoning traces. This raises a key question: what aspects of reasoning quality (i.e., correctness, efficiency, and failure patterns) are encoded in the structure of a CoT trace? We introduce \textbf{CoT2Graph}, a framework that represents CoT traces as directed dependency graphs, making relationships between reasoning steps explicit and capturing patterns such as backtracking, branching, and subtask merging. We model these graphs with graph neural networks and use attribution-based explanations to identify structural patterns associated with reasoning quality. CoT2Graph reveals distinct patterns of successful and failed reasoning: correct CoTs place greater importance on early planning, whereas incorrect CoTs are associated with overloaded synthesis steps and dependency violations. We further show that graph-based signals better predict whether a CoT leads to a correct answer, outperforming previous tree-based structural methods by up to 2.5\% and simple length-based baselines by 7.8\%, while also improving Best-of-$N$ selection. Overall, our results demonstrate that reasoning structure encodes rich signals about reasoning quality, motivating a shift from sequence-level evaluation to structural analysis.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Tommaso Felice Banfi, Mubashara Akhtar, Jingwei Ni, Mrinmaya Sachan. 2026-09-06. Understanding Failures in LLM Reasoning by Learning Structured Representations of Chain-of-Thought. https://searcharxiv.com/abs/2611.18611
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.