language-model-updated-fine-tuning-substituted-fact-facts-already-answered CLM-0203When a language model is updated by fine-tuning on a substituted fact, the facts it already answered correctly that are most likely to flip to wrong are those whose answer entity is referenced by many other facts, and updates to such structurally popular facts corrupt connected facts out to five hops in the fact graph — so popularity, not long-tail rarity, predicts collateral corruption among already-correct knowledge.
Capability: Stating false facts confidently
Observed on
LoRA fine-tuning with counterfactual object substitutions on four open-weight instruction-tuned models (2B-31B, Qwen and Gemma families), evaluated on single-answer Wikipedia-deriv.
Sources
- Measured on a constructed 100k-node / 432k-edge Wikipedia fact graph; flip rates pooled over hops 1-5 for four models, with a within-neighborhood control showing string similarity correlates near zero with flipping (aggregate r=0.05), and an attention-perturbation probe showing larger |ΔAttLift| for popular-source updates in three models. Popularity is proxied by object in-degree, not proposition frequency, so entity connectivity and relation rarity are not disentangled. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Knowledge-editing methods such as ROME and fine-tuning insert a stated fact reliably but do not carry its logical consequences: on multi-hop questions generated from knowledge-graph rules that the edit entails, accuracy is up to 24 points lower than on the directly edited fact, so an edited model can assert a new fact and still answer as if its implications never changed.Absorbing and flexibly using new knowledge · reviewed by AI
- When new knowledge is a set of interconnected fictional future events rather than isolated fact edits, updating 4-8B instruct models by mid-training on teacher-generated long-form articles plus preference tuning (preferring reasoned answers over outdated, hallucinated, or needlessly abstaining ones) beats graph-retrieval and in-context editing baselines on multi-hop and causal questions, while remaining worse than in-context methods on exact numeric single-hop recall unless retrieved facts are also supplied.Absorbing and flexibly using new knowledge · reviewed by AI
- Accuracy on a fact scales with how many pretraining documents mention it, so long-tail facts are systematically unreliable.Stating false facts confidently
- In ESG-report RAG question answering with retrieval held fixed at a single top-1 chunk, open-weight 2B-30B generators vary widely in how well they stay anchored to the retrieved passage (faithfulness 0.607-0.822) yet all land in a narrow low band of factual correctness (0.387-0.449), and the most faithful model is the least factually correct, so groundedness to context and correctness of extracted ESG facts do not track each other.Competence in a specialist domain · reviewed by AI
- For factual paired-candidate QA where the model already holds the needed knowledge, hallucination can be detected by removing the query keyword that most drives the chosen answer and seeing whether the prediction systematically recovers to the alternative — erroneous answers show that recovery while correct answers merely become ambiguous — but the signal weakens when the knowledge is absent and largely fails on multi-step arithmetic reasoning, where errors are distributed across steps rather than bound to one keyword.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Under the same protocol, grouping already-correct neighbor facts by object in-degree shows equal or higher flip rates for low-popularity facts, or flip rates fall to baseline beyond one hop. Proposed technique, not catalogued: Popularity-based anchoring (KL-regularize on a small set of high-in-degree facts during updates).