dpo-alignment-7-8b-instruct-models-alignment-tax-predictable-properties CLM-0378In DPO alignment of 7-8B instruct models, the alignment tax is predictable from properties of the preference pairs themselves: pairs whose chosen/rejected log-probability margin under the reference model is large in magnitude, or whose chosen/rejected token-length difference is large, cause the most loss of general capability, so dropping them keeps most of the alignment gain while cutting forgetting.
Capability: Retaining skills and safety after fine-tuning
Observed on
Offline DPO on HH-RLHF with Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct, 5% data budget, general capability measured on MMLU, IFEval, ARC-Challenge, HumanEval, GSM8K..
Sources
- Validated by: Against Full, Random, and nine SFT/alignment data-selection baselines at a fixed 5% budget on HH-RLHF: general-capability drop 0.86/0.38 points (Llama/Qwen, helpfulness) versus 5.35/2.31 for full-data DPO, while alignment gain stayed near the best baseline; bucket experiment shows forgetting rising from 0.08 to 3.48 across reference-margin quantiles; ablation removing each score degrades either forgetting or alignment.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In GRPO-style RLVR on math for 7B models, injecting partial reasoning prefixes from a small out-of-family model (Gemma-2-2B, LLaMA-3.2-1B) into a fraction of rollouts keeps policy entropy higher early in training and raises pass@k, with gains growing as k rises; the effect comes from distributional unfamiliarity rather than prefix quality, since prefixes from same-family models of equal or higher quality give no gain over vanilla GRPO.Retaining skills and safety after fine-tuning · reviewed by AI
- In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.Retaining skills and safety after fine-tuning · reviewed by AI
- For instruction tuning an 8B open model on text-to-SQL, a curated set of under 900 examples selected to cover every table in the target schemas and score well on question-SQL alignment reaches higher execution accuracy than pipelines trained on tens of thousands to millions of samples, and dropping the schema-coverage term from the selection objective costs more accuracy than dropping any single quality term.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- When post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.Predicting future events · reviewed by AI
- When fine-tuning 7-8B code models on an LLM-synthesized Verilog instruction corpus, training on a 20-25% subset selected for functional correctness, instruction-solution alignment, and coverage of Verilog structural clusters (AST, CFG, netlist features) yields higher Pass@1 than training on the full corpus, because the automated corpus carries noise and redundancy that filtering removes.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Partitioning a preference dataset by reference-margin magnitude or length difference and training DPO on each bucket shows no monotone relation to general-benchmark degradation, or the composite filter loses as much general capability as random selection at equal budget.