fine-tuning-retention CAP-0018 · activeRetaining skills and safety after fine-tuning
Covers loss of general capability, out-of-domain skill, refusal behavior, or output diversity caused by domain, multimodal, alignment or RL post-training; excludes in-domain accuracy gains and inference-time behavior shifts.
Also called: catastrophic forgetting, alignment tax, mode collapse, output diversity, exploration collapse
Tags: general
Proposed from the ingestion pipeline rather than chosen by hand. 5 papers independently pointed at this same competence, arriving under 4 different names (catastrophic-forgetting, alignment-tax, multimodal-adaptation-forgetting, finetuning-alignment- fragility), which is the signal that it is a real recurring topic and not one author's framing. Recurs across four independent framings (capability forgetting and safety erosion) and is not covered by any existing entry. Absorbed output-diversity (2026-09-19): mode collapse, entropy decline and repeated attempts retreading one strategy after fine-tuning or RL are the same failure seen in the distribution rather than in accuracy, so post-training narrowing is filed here as a facet.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Covers loss of general capability, out-of-domain skill, refusal behavior, or diversity of outputs and solution paths caused by domain, multimodal, alignment or RL post-training; excludes in-domain accuracy gains, inference-time behavior shifts, and stylistic variety alone.
Claims
- mechanismsingle paper↑1↓0For 1.5B-8B base models given a domain SFT expert (math or medical), keeping both models and routing token by token — invoking the expert only on tokens where a trained router predicts the base would fail — preserves general-benchmark accuracy within about 0.5% of the base while exceeding the expert's in-domain accuracy, whereas loss-modifying SFT regularizers still lose up to 16% general accuracy.
- mechanismsingle paper↑1↓0When 7-8B open-weight instruct models are fully fine-tuned on a single domain dataset, confining gradient updates to FFN neurons that activate stably across the training set and adding an MSE loss that pulls activations back toward pre-computed activation anchors retains far more general-benchmark ability (MMLU, BBH, TyDiQA) than full-parameter fine-tuning, by about 9.9 points on average, while also raising in-domain scores.
- mechanismsingle paper↑1↓0In DPO alignment of 7-8B instruct models, the alignment tax is predictable from properties of the preference pairs themselves: pairs whose chosen/rejected log-probability margin under the reference model is large in magnitude, or whose chosen/rejected token-length difference is large, cause the most loss of general capability, so dropping them keeps most of the alignment gain while cutting forgetting.
- observationsingle paper↑1↓0In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.
- mechanismsingle paper↑1↓0In GRPO-style RLVR on math for 7B models, injecting partial reasoning prefixes from a small out-of-family model (Gemma-2-2B, LLaMA-3.2-1B) into a fraction of rollouts keeps policy entropy higher early in training and raises pass@k, with gains growing as k rises; the effect comes from distributional unfamiliarity rather than prefix quality, since prefixes from same-family models of equal or higher quality give no gain over vanilla GRPO.
- mechanismsingle paper↑1↓0Refusal learned by alignment tuning lives in a low-rank subspace of hidden-state activations that recurs across model families; ablating directions in that subspace lowers harmful-request refusal, and a scalar computed from the subspace's geometry before any fine-tuning (the Geometric Fragility Score) picks out which initially safe model will keep the most refusal after small benign LoRA fine-tuning.
- observationsingle paper↑1↓0Safety realignment after a poisoned fine-tune is brittle in three ways: existing defenses (SSRD, RESTA, soft-SFT, rollback) often destroy the downstream skill they were meant to keep, lose more than 30% of their effect when the defender's prompt template differs from the attacker's, and a realigned model can be re-jailbroken by switching the system prompt; distilling on-policy from two frozen teachers, the original model for refusal and the fine-tuned model for the task, routed per token, cut attack success to about 2% under the defender's template while holding task score near pre-attack level.
- observationsingle paper↑1↓0Unlearning methods that make a model answer "I don't know" about a forget set can leave the knowledge in place: the strongest IDK-style fine-tuning method still answered those same questions accurately when probed in a different format, and across nine unlearning methods on three model families none met a joint standard of utility and honesty on retained knowledge plus consistent, acknowledged forgetting; aligning representations after feature randomization (ReVa) roughly doubled the rejection rate that survives a second conversational turn.
Techniques
- Complete prefixes from a weaker out-of-family model during RtrainingFor a fraction of RLVR rollouts, prepend a truncated partial reasoning trajectory generated by a smaller, different-family model (cut at the largest step-to-step entropy drop) and train the target model to complete it.
- Filter preference pairs by reference margin, length gap, lextrainingScore each preference pair by a normalized composite of |reference-model per-token log-prob margin|, chosen-minus-rejected token length difference, and TF-IDF similarity to a general-capability corpus, then train DPO only on the lowest-risk
- Geometric fragility score from the safety subspacetoolingCollect hidden states on harmful and harmless prompts, recover the low-rank safety subspace per layer, and compress its geometry into one score that ranks aligned checkpoints by how easily benign fine-tuning will erase refusal, so the fragi
- Route token by token between base model and SFT expertdecodingTrain a lightweight two-stage router on the base model's hidden states, labelled by tokens where the base's greedy prediction is wrong and the expert's is right, then at decoding mix the two next-token distributions with momentum smoothing
- Routed on-policy distillation from safety and task teacherstrainingSample from the compromised student, route each token to either the original aligned model (refusal prior) or the fine-tuned task model, and train the student with a top-K KL loss toward the routed teacher, so realignment targets the distri
- Stratify rollouts by confidence and outcome when forming RL trainingWithin each RL step, bucket the step's own rollouts by length-normalized surprisal and reward into anchors, hard negatives and low-confidence successes, and assemble half the optimization groups from a fixed ratio of these buckets before cl
- Update only stable neurons and anchor their activationstrainingSelect FFN neurons with low activation coefficient of variation, mask gradients to those neurons only, and after cross-entropy converges add an MSE term pulling current activations toward a pre-computed activation anchor.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Models2026-09-05↑ supports a claim here
- about this capability · no claim drafted yet
- about this capability · no claim drafted yet
- ↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Skin-Deep: A Geometric Diagnostic for Alignment Fragility in Large Language Model Representations2026-06-21↑ supports a claim here
- ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.