When Does Confidence Reveal Problem Difficulty?
Xiyu Bao, Langyuan Cui, Hwee Tou Ng, Chun Kai Ling · 14 Nov 2026 · arXiv
Abstract
While confidence signals are commonly evaluated for their ability to predict a model's own correctness, their relationship with problem difficulty remains less understood. In this work, we investigate a wide range of confidence-related signals and whether they can serve as indicators of problem difficulty across various task domains. We find that perturbation inconsistency, semantic entropy, and solution length exhibit strong associations with population difficulty than token-level uncertainty. Notably, these relationships persist even when restricting the analysis to problems correctly answered by the focal model, suggesting that such signals capture information about difficulty beyond their ability to predict the model's own correctness. We further explore applications of these signals, illustrating the potential and limitations of using confidence-related signals to guide inference-time decisions. Our findings highlight the utility of model-specific signals as imperfect but informative proxies for population-level problem difficulty.
Comments
No comments yet. Share your own experience below.
Visitor posts are moderated before they appear and are the opinion of their authors, not DEEKOYA AI. Names are not verified. See our privacy policy.