Reasoning Confidence Is Not Policy-Invariant: Calibrating Prefix Success under Model and Budget Shift

Abstract

Inference-time scaling of LLM reasoning relies on a confidence score over each partial trace, yet existing estimators treat prefix quality as intrinsic to the prefix. A given prefix can succeed at very different rates and rankings under different continuation policies. Reasoning confidence is therefore not policy-invariant, and a squared-loss lower bound shows that policy-blind predictors retain residual error proportional to cross-policy disagreement. Calibration must condition on the continuation model and inference budget. To bridge this gap, we propose the Comparable Policy-Conditioned Calibrator (CPCC), built from a shared frozen-LLM prefix encoder, comparable scalar policy features under per-policy z-score normalization, and a Beta–Binomial head trained on bootstrap success counts. Across six continuation policies and three reasoning benchmarks (GSM8K, SVAMP, MATH-500), all evaluated under majority-vote aggregation, the CPCC family attains the lowest Brier on every evaluation cell, and CPCC is strongest on the held-out model families. It also avoids the held-out-family collapse of raw-hidden conditioning, lowers ECE by 13 to 26 percent on out-of-distribution math, and reduces downstream selective risk by 10 to 30 percent.

Qiyu Wei 魏启宇

PhD Student

My research interests include Probabilistic Machine Learning, Bayesian Optimization, and LLM agents.