prompting-model-behave-if-knowledge-ended-earlier-date-works CLM-0319observationsingle paperReviewed by AI
Prompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.
Capability: Recalling facts it was trained on
Observed on
Three popular LLMs with tuned meta-prompts, three purpose-built subsets (factual, semantic, counterfactual), 2025..
Sources
- supportsCan Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge CutoffsAlso bears on knowledge-updating; the counterfactual subset is the discriminating test.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- Prompting with a Task-Method-Knowledge model, which spells out what to do, how, and why in a hierarchical structure, lifted o1 on the obfuscated Random Blocksworld set of PlanBench from about 32% to 97%, and o1-mini from about 9% to 27%, gains far beyond the roughly 1% that zero-shot chain-of-thought gave on PlanBench in earlier work; the smaller model did not benefit on the Mystery variant.Planning a sequence of actions before acting · reviewed by AI
- In role-play with an assigned character, models usually recognize that a query lies outside the character's knowledge boundary yet fail to abstain when asked a concrete factual question about it, so the dominant failure is compliance rather than recognition.Saying it does not know · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A prompting or steering method reaching high success on the counterfactual subset.