confidence-gated-retrieval-agents-commit-retrieve-or-defer-against-fixed CLM-0556In confidence-gated retrieval agents that commit, retrieve or defer against a fixed numerical confidence threshold, applying a post-hoc calibration map to the same confidence signal raises accuracy among committed answers (15.8-41.0 points across three models on HotpotQA and MuSiQue) purely by moving the controller to a more selective operating point, so coverage falls and retrieval cost rises and overall accuracy can go either way (up to +15 points on HotpotQA, down to -17 on MuSiQue).
Capability: Saying it does not know
Observed on
Multi-hop QA (HotpotQA, MuSiQue) with fixed benchmark-grounded evidence plans, three retrieval depths, verbalized confidence, isotonic calibration fitted on depth-zero states, thre.
Sources
- Validated by: Used to compare raw vs isotonic-calibrated verbalized confidence under an identical policy, budget and cost model on 2,457 test episodes across three models and two datasets, reporting overall accuracy, committed accuracy, coverage and mean retrieval actions jointly; no external baseline protocol compared.. Measured on 1,852 HotpotQA and 605 MuSiQue episodes, three models, deterministic temperature-zero single generations, LLM judge for correctness with no human validation. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.Recalling facts it was trained on · reviewed by AI
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- When a calculator-using model is trained with reinforcement learning on verifiable final-answer rewards for an arithmetic search task (Countdown), the gain lands almost entirely at low k — pass@1 rises sharply while pass@16 moves little or falls — because the update can only reinforce correct trajectories the starting policy already samples, and prompt groups with no correct sample supply no gradient.Using the tools it is given · reviewed by AI
- For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A matched-trajectory replay where calibration raises committed accuracy while leaving coverage and retrieval cost essentially unchanged, or where overall accuracy improves consistently across datasets, would show the effect is not merely an operating-point shift.