safety-realignment-after-poisoned-fine-tune-brittle-three-ways-existing CLM-0278observationsingle paperReviewed by AI
Safety realignment after a poisoned fine-tune is brittle in three ways: existing defenses (SSRD, RESTA, soft-SFT, rollback) often destroy the downstream skill they were meant to keep, lose more than 30% of their effect when the defender's prompt template differs from the attacker's, and a realigned model can be re-jailbroken by switching the system prompt; distilling on-policy from two frozen teachers, the original model for refusal and the fine-tuned model for the task, routed per token, cut attack success to about 2% under the defender's template while holding task score near pre-attack level.
Capability: Retaining skills and safety after fine-tuning
Observed on
Llama-2, Qwen2.5 and Gemma-2 bases, 2026; tasks SQL, SAMSum, NL2Bash; 700 BeaverTails harmful prompts judged by Qwen2.5-32B; residual risk remains when deployment and attacker temp.
Sources
- Validated by: Attack success rate and task score across three bases and three tasks under matched and mismatched attacker/defender templates against four realignment baselines; system-prompt re-jailbreak test.. Comprehensive template-mismatch design and honest reporting of residual cross-template risk; open models up to mid-size only; the harm judge is itself a model.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- As long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.Following instructions hidden in data · reviewed by AI
- Rewriting a harmful request in an emotional or narrative style (fearful, curious, compassionate) while keeping its intent raises jailbreak success by up to 57 percentage points across 16 instruction-tuned models, contextual rewrites beating fixed templates, and the vulnerability does not shrink with model scale; a preprocessing step that has a second model strip stylistic cues cuts the success rate substantially.Refusing harmful requests under obfuscation · reviewed by AI
- For black-box jailbreak defense at test time, abstracting each successful attack into a rule that names the structural wrapper (nested role-play, code obfuscation, flipped text) rather than the harmful topic lets one rule induced from a single prompt cover the rest of that attack family, so attack success falls to near zero after the first batch of ~20 streamed prompts without any parameter updates.Improving its own scaffolding from failures · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- Across 25 open-weight instruction-tuned models (4B-744B), the ability to modulate a named concept's residual-stream projection on plain instruction does not track parameter count or recency — the best scorer is an 8B 2024 model and some of the newest models score lowest — while within a single model line the score rises steadily across ordinary pre-training and post-training checkpoints, so activation controllability is acquired incidentally and must be measured per model rather than inferred from scale.Strategic deception and detecting it · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Attacks that poison the task teacher's own outputs so the routed distillation reintroduces the harm, or template gaps large enough that the residual ASR the paper reports becomes the common case.