rlhf-tuned-chat-models-raw-conditional-probabilities-consistently-overconfident-confidence CLM-0269observationsingle paperReviewed by AI
For RLHF-tuned chat models the raw conditional probabilities are consistently overconfident, and a confidence the model states in words or as a number in its output is better calibrated, often halving expected calibration error on TriviaQA, SciQ and TruthfulQA; asking the model to list several candidate answers before giving confidences improves it further, while chain-of-thought does not.
Capability: Saying it does not know
Observed on
ChatGPT (gpt-3.5-turbo), GPT-4, Claude 1 and 2 and Llama-2-70B-Chat, 2023; short-answer factual QA; gains on Llama-2-Chat were smaller and less consistent..
Model cards: GPT-4
Sources
- Validated by: Expected calibration error, Brier score and selective accuracy on TriviaQA, SciQ and TruthfulQA comparing raw log-probabilities, verbalized numbers, verbalized words and multi-candidate prompting across five RLHF models; chain-of-thought variant tested and found not to help.. Three factual QA benchmarks and five models; the weak result on Llama-2-Chat suggests the effect depends on the training recipe. The 'relative 50%' figure is a typical, not universal, reduction.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 30 open LLaMA/Mistral/Qwen models and 8 classification tasks, a model's self-reported numeric confidence tracks its own logit-based confidence only weakly at the instance level (mean Pearson r about 0.135) even though setting-level averages of the two correlate moderately, and instruction-tuned models show larger linguistic-vs-internal gaps and worse calibration error than base models.Saying it does not know · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- For spoken language models answering questions over long dialogue histories, converting the history to transcripts beats keeping it as audio when the answer is a lexical detail (exact slot values across dense turns: mean 71.3% vs 42.6%), but loses when the answer depends on how something was said, where audio history wins for six of seven models and by a larger margin once lexical emotion cues are removed.Understanding speech and audio · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Recalling facts it was trained on · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Newer RLHF models whose raw probabilities are as well calibrated as verbalized confidences, or tasks (long-form, reasoning) where verbalized confidence is no better than log-probabilities.