training-language-model-differential-privacy-whether-dp-sgd-pre-training-orTraining a language model with differential privacy — whether DP-SGD pre-training or DP LoRA fine-tuning — makes it generate more false claims about facts in its training data than a non-private counterpart, and the effect grows as the privacy budget tightens, because per-example gradient clipping plus noise prevents low-frequency facts from being acquired at all while flattening the next-token distribution onto incorrect alternatives.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Stating false facts confidently
Observed on
Open-ended long-form generation of Wikipedia-style articles, scored by atomic-claim verification; GPT-J 6B fine-tuned at epsilon 8 and 16, and VaultGemma-1B pre-trained at epsilon .
Sources
- FactScore on three Wikipedia sets plus human annotation by 5 graduate students on 30 stratified AI articles; ordering non-DP > eps=16 > eps=8 held on both datasets, though one comparison was not significant and eps=8 was near the untrained base model on the AI set. Mechanism supported by nucleus/entropy measurements and a synthetic bigram replication rather than by intervention. Model-human agreement on claim support was low, so the automated metric is an imperfect proxy.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · unreviewed
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · unreviewed
- When fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.Stating false facts confidently · unreviewed
- When training small instruction-tuned models with reinforcement learning plus factual supervision, routing each atomic fact's verification score only to the tokens that produced it, and down-weighting verifier judgements that do not change when their key evidence is removed, improves factuality benchmark scores over trajectory-level or reasoning- step-level factual rewards.Checking claims against evidence · unreviewed
- For persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Remembering across sessions · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A DP-trained model at a strict privacy budget matching or beating its non-private counterpart on atomic-fact accuracy for facts present in the training data, with no reduction in output informativeness. Drafted stance toward accuracy-fact-scales-how-many-pretraining-documents-mention: supports -- A controlled experiment varying fact frequency shows DP raises the repetition count needed to acquire a fact, so the frequency-accuracy dependence sharpens under DP.