arXiv · 2611.19173
When Does Confidence Reveal Problem Difficulty?
Abstract
While confidence signals are commonly evaluated for their ability to predict a model's own correctness, their relationship with problem difficulty remains less understood. In this work, we investigate a wide range of confidence-related signals and whether they can serve as indicators of problem difficulty across various task domains. We find that perturbation inconsistency, semantic entropy, and solution length exhibit strong associations with population difficulty than token-level uncertainty. Notably, these relationships persist even when restricting the analysis to problems correctly answered by the focal model, suggesting that such signals capture information about difficulty beyond their ability to predict the model's own correctness. We further explore applications of these signals, illustrating the potential and limitations of using confidence-related signals to guide inference-time decisions. Our findings highlight the utility of model-specific signals as imperfect but informative proxies for population-level problem difficulty.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Xiyu Bao, Langyuan Cui, Hwee Tou Ng, Chun Kai Ling. 2026-11-14. When Does Confidence Reveal Problem Difficulty?. https://searcharxiv.com/abs/2611.19173
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.