closed-book-multi-hop-qa-accuracy-gain-decomposed-prompting-shrinks-little CLM-0295mechanismsingle paperReviewed by AI
In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.
Capability: Recalling facts it was trained on
Observed on
Direct, assistive and incremental prompting on multi-hop QA benchmarks across model scales, 2026; abstention policy built on cross-regime disagreement..
Sources
- Validated by: F1 and AUROC as an error detector against verbalized-confidence and consistency baselines on multi-hop QA across model scales; training-free.. Also bears on abstention-and-self-knowledge; the mechanism (stable facts, stochastic hallucinations) is stated and tested on QA only.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- In confidence-gated retrieval agents that commit, retrieve or defer against a fixed numerical confidence threshold, applying a post-hoc calibration map to the same confidence signal raises accuracy among committed answers (15.8-41.0 points across three models on HotpotQA and MuSiQue) purely by moving the controller to a more selective operating point, so coverage falls and retrieval cost rises and overall accuracy can go either way (up to +15 points on HotpotQA, down to -17 on MuSiQue).Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- For 0.6B-8B Qwen3 models on math, logic and multiple-choice reasoning, quantifying reasoning uncertainty from a directed acyclic graph built by sampling several chains and merging semantically equivalent steps with an NLI model separates correct from incorrect generations better than sequence-level entropy, likelihood and self-prompting uncertainty measures.Carrying a chain of inference through to its conclusion · reviewed by AI
- For large audio language models on multiple-choice audio reasoning benchmarks (MMAR, MMAU-mini), accuracy does not track whether the model's chain of thought is acoustically grounded or causally drives the answer: several models answer correctly from perception steps that assert unsupported labels, and for some models replacing the reasoning chain with vacuous or fabricated content leaves the final answer unchanged.Understanding speech and audio · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Domains where a memorized wrong fact is stable across regimes (consistent misconceptions), which the method would pass as known.