unlearning-methods-make-model-answer-i-don-t-know CLM-0279Unlearning methods that make a model answer "I don't know" about a forget set can leave the knowledge in place: the strongest IDK-style fine-tuning method still answered those same questions accurately when probed in a different format, and across nine unlearning methods on three model families none met a joint standard of utility and honesty on retained knowledge plus consistent, acknowledged forgetting; aligning representations after feature randomization (ReVa) roughly doubled the rejection rate that survives a second conversational turn.
Capability: Retaining skills and safety after fine-tuning
Observed on
Nine unlearning methods, three open model families, 2026; Q&A and multiple-choice probes of the forget set, rejection rate measured after two rounds of interaction..
Sources
- The headline finding is a failure mode shown by re-probing, which is direct; the proposed fix is evaluated by the same authors' new metrics. Whether hidden retention is exploitable in practice is not tested.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Instruction tuning that forces an answer to every question teaches models to fabricate when the question lies outside their parametric knowledge; splitting the tuning data by whether the base model already answers correctly and appending "I am unsure" to the rest (refusal-aware tuning) raises average precision on known-versus-unknown separation over vanilla tuning, transfers to out-of-domain datasets, and yields better calibrated confidence than post-hoc uncertainty scoring.Saying it does not know · reviewed by AI
- When an LLM answers a question and gives an explanation, removing from the input the concepts its explanation did not credit and re-querying on the reduced input raises measured explanation faithfulness over both standard prompting and prompts that ask for faithfulness, because the removal deletes the unmentioned influences rather than asking the model to report them.Saying what actually drove the answer · reviewed by AI
- When an LLM gives an answer plus an explanation, deleting from the input those concepts the explanation does not credit and re-querying on the reduced input raises measured faithfulness over both standard prompting and prompts that ask for faithful explanations, because the re-query removes the unmentioned factors that could have influenced the original answer.Saying what actually drove the answer · reviewed by AI
- When vision-language models are given an explicit abstention option plus a stated payoff for correct, wrong and declined answers, quadrupling the penalty for a wrong answer barely changes how often they answer, so their abstention rate is largely insensitive to the stated cost of error across text, synthetic video and natural video recall tasks.Saying it does not know · reviewed by AI
- Prompting a model to behave as if its knowledge ended at an earlier date works for facts asked about directly (about 82% unlearning success) and for word-sense shifts (about 70%), but fails when the post-cutoff knowledge is only causally connected to the question (about 19%), so a simulated cutoff cannot be trusted to remove contamination from temporal prediction tests.Recalling facts it was trained on · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An unlearning method whose refusal on the forget set holds across probe formats and follow-up turns without loss on the retain set.