factual-qa-817-item-truthfulqa-multiple-choice-plus-300-item-natural-questions CLM-0388observationsingle paperReviewed by AI
On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.
Capability: Saying it does not know
Observed on
Eleven open-weight and hosted instruction-tuned models; short factual QA with deterministic scoring; confidence elicited from the same model that answers; no fine-tuning or logit a.
Sources
- Validated by: Compared against forced-choice Direct and chain-of-thought baselines on 817 balanced-option TruthfulQA-MC items across 11 models: wrong-commitment rate fell from 13.1% to 8.9% and answered accuracy rose from 86.9% to 89.7% at 87.6% coverage, with 11/11 models improving on both axes (paired one-sided Wilcoxon p=0.00049); an abstention audit paired each abstention with the CoT outcome, and a 300-item NQ short-answer run with 5 models showed the same direction.. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 14 frontier and open model routes scored on 3,000 boundary questions (fabricated entities, post-cutoff events, hyper-niche facts, redacted-document values), the rate at which a model restrains itself and calibrates confidence varies by a factor of about 2.6 (composite 0.31 to 0.81) while all models answer a document-grounded restored-span probe at near-ceiling (pooled 99.2%, no detectable between-model variation), so epistemic restraint is not explained by the ability to report information placed in front of the model.Saying it does not know · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- On SelfAware, a set of unanswerable questions (no scientific consensus, imagination, philosophical, too many variables, subjective) paired with answerable ones, models' ability to flag the unanswerable ones rises with parameter count, with instruction tuning and with in-context examples, yet the best model tested (GPT-4, F1 about 75) stays well below the human reference of about 85, so self-knowledge is present but incomplete.Saying it does not know · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A replication where the same gate either fails to lower the wrong-commitment rate for most models, or where the abstained-on questions are mostly ones the baseline got wrong (correct-abstention share above 50%), would break it.