factual-recall CAP-0017 · activeRecalling facts it was trained on
Whether a model retrieves a fact from its weights when asked, especially long-tail facts; the failure is a wrong or missing answer, measured against ground truth, not the confidence of a made-up one.
Tags: knowledge, general
Filed from Russ's capability map (2026-09-12). The neighbour is stating false facts confidently, which is about fabrication under uncertainty; this is about whether the fact is there to be recalled at all, which scales with how often it appeared in pretraining. The reversal curse is a specific recall asymmetry and stays its own entry.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
In scope when the paper measures recall accuracy against ground truth as a function of fact frequency, entity popularity, or model size. NOT in scope when the finding is about confident fabrication or calibration (hallucination, uncertainty), or about facts supplied in context (retrieval utilization).
Claims
- mechanismsingle paper↑1↓0Asked which of two entities is larger on a numeric attribute, models often contradict numbers they can state correctly, following entity popularity, mention order and co-occurrence instead; for 7-8B models a logistic regression on those three surface cues predicts the model's choice better than the model's own numeric recall does, while 32B models selectively use the numbers when they are reliable.
- mechanismsingle paper↑1↓0In closed-book multi-hop QA the accuracy gain from decomposed prompting shrinks to little in frontier models, but answers still change between task-equivalent prompt regimes (agreement only about 70%, and about 60% on MuSiQue), and because stable facts survive re-prompting while hallucinations do not, disagreement across regimes detects errors better than standard uncertainty baselines on F1 and AUROC without retrieval or training.
- observationsingle paper↑1↓0On county-level US knowledge (14,782 questions over 526 counties), the best of 13 models reaches only about 57% on narrative questions and no model exceeds 13% on numeric Census facts, larger models are not reliably better, and web search helped Gemini (+14 points) while hurting GPT models (about -11), so hyper-local recall is thin and retrieval does not fix it uniformly.
- observationsingle paper↑1↓0Prompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.
Techniques
- Abstain on disagreement between task-equivalent prompt regimpromptingAsk the same closed-book question under two or three equivalent prompt structures (direct, decomposed, incremental); if the answers disagree, treat the fact as unknown and abstain, since memorized facts are stable across regimes and halluci
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs2025-09-26↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.