across-14-frontier-open-model-routes-scored-3-000 CLM-0539Across 14 frontier and open model routes scored on 3,000 boundary questions (fabricated entities, post-cutoff events, hyper-niche facts, redacted-document values), the rate at which a model restrains itself and calibrates confidence varies by a factor of about 2.6 (composite 0.31 to 0.81) while all models answer a document-grounded restored-span probe at near-ceiling (pooled 99.2%, no detectable between-model variation), so epistemic restraint is not explained by the ability to report information placed in front of the model.
Capability: Saying it does not know
Observed on
Single-turn, temperature 0, retrieval and history disabled, non-reasoning responses only; the capability probe is deliberately easy and a harder capability measure might still corr.
Sources
- Validated by: Applied to 42,000 model-item records across 14 routes; classifier checked against two blinded human coders on a stratified 560-response sample (human-human kappa .913, classifier-human kappa .414 four-class, .667 binary); matched restored-span control gave 96.6% restraint on missing-span and 99.2% correct on restored-span, 95.8% passing both sides.. Measured on one frozen run of 14 routes over 3,000 items with bootstrap intervals; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- On SelfAware, a set of unanswerable questions (no scientific consensus, imagination, philosophical, too many variables, subjective) paired with answerable ones, models' ability to flag the unanswerable ones rises with parameter count, with instruction tuning and with in-context examples, yet the best model tested (GPT-4, F1 about 75) stays well below the human reference of about 85, so self-knowledge is present but incomplete.Saying it does not know · reviewed by AI
- On self-contained linguistic Rosetta Stone puzzles where a single context example uniquely carrying a question-required token is deleted, frontier models told to output N/A when context is insufficient almost never abstain, and on some puzzles answer the damaged variant as well as or better than the original.Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A capability probe of comparable simplicity that separates the same panel and correlates with composite EHQ, or a replication where restraint scores cluster tightly across models rather than spanning 0.31-0.81. Drafted stance toward selfaware-set-unanswerable-questions-no-scientific-consensus-imagination-philosophical: supports -- Both find that ability to flag questions outside the model's knowledge is partial and not predicted by ordinary task competence, here extended to determinate-but-unavailable answers rather than in-principle unanswerable ones.