self-contained-linguistic-rosetta-stone-puzzles-where-single-context-example CLM-0634On self-contained linguistic Rosetta Stone puzzles where a single context example uniquely carrying a question-required token is deleted, frontier models told to output N/A when context is insufficient almost never abstain, and on some puzzles answer the damaged variant as well as or better than the original.
Capability: Saying it does not know
Observed on
53 UKLO Rosetta Stone puzzles, single-example deletion, zero-shot single pass, three frontier models (Claude Sonnet 4.6, Gemini 3 Flash, GPT-5.4) with explicit abstention instructi.
Sources
- Validated by: Applied to 53 UKLO puzzles, classifying 45 as fragile and 8 as robust; cross-checked by LLM-as-judge agreement with the structural label (Gemini 71% recall at 94% precision), and used to surface 21 puzzles where Gemini matched or beat its original score on a judged-unsolvable variant. Measured on 53 puzzles x 3 conditions x 3 models, one sample per puzzle. Deletion-attributable abstentions after excluding pre-existing ones: GPT 0, Gemini 2, Claude 21 split evenly across random and targeted deletion. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across 14 frontier and open model routes scored on 3,000 boundary questions (fabricated entities, post-cutoff events, hyper-niche facts, redacted-document values), the rate at which a model restrains itself and calibrates confidence varies by a factor of about 2.6 (composite 0.31 to 0.81) while all models answer a document-grounded restored-span probe at near-ceiling (pooled 99.2%, no detectable between-model variation), so epistemic restraint is not explained by the ability to report information placed in front of the model.Saying it does not know · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- On SelfAware, a set of unanswerable questions (no scientific consensus, imagination, philosophical, too many variables, subjective) paired with answerable ones, models' ability to flag the unanswerable ones rises with parameter count, with instruction tuning and with in-context examples, yet the best model tested (GPT-4, F1 about 75) stays well below the human reference of about 85, so self-knowledge is present but incomplete.Saying it does not know · reviewed by AI
- In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Stating false facts confidently · reviewed by AI
- When an LLM answers a question and gives an explanation, removing from the input the concepts its explanation did not credit and re-querying on the reduced input raises measured explanation faithfulness over both standard prompting and prompts that ask for faithfulness, because the removal deletes the unmentioned influences rather than asking the model to report them.Saying what actually drove the answer · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Models given the targeted-deletion variants abstain on the damaged questions at rates close to the structural damage count, and their accuracy on those questions drops to near zero. Drafted stance toward selfaware-set-unanswerable-questions-no-scientific-consensus-imagination-philosophical: supports -- Both find models answer rather than flag insufficient information, here with the unanswerability induced by a single matched deletion rather than inherently unanswerable questions.