grpo-style-rlvr-math-7b-models-injecting-partial-reasoning-prefixes CLM-0402In GRPO-style RLVR on math for 7B models, injecting partial reasoning prefixes from a small out-of-family model (Gemma-2-2B, LLaMA-3.2-1B) into a fraction of rollouts keeps policy entropy higher early in training and raises pass@k, with gains growing as k rises; the effect comes from distributional unfamiliarity rather than prefix quality, since prefixes from same-family models of equal or higher quality give no gain over vanilla GRPO.
Capability: Retaining skills and safety after fine-tuning
Observed on
Qwen2.5-7B and Qwen2.5-Math-7B trained with GRPO on MATH, evaluated on six math benchmarks; prefix injection probability 0.2, entropy-based truncation; only mathematical reasoning .
Sources
- Validated by: Average pass@1 to pass@128 over AIME 2024/2025, AMC 2023, MATH 500, Minerva, OlympiadBench against vanilla GRPO and the untrained base model; ablations on injection probability, random vs entropy truncation, and same-family prefix generators; LLM-graded prefix quality distribution.. Measured on two 7B targets, one training set, six benchmarks, single runs with no seed variance reported; pass@128 still below the untrained base model in most rows, so the method narrows rather than removes coverage loss. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.Retaining skills and safety after fine-tuning · reviewed by AI
- For adapting Qwen3 instruction-tuned models from 0.6B to 32B to tool-calling corpora, LoRA supervised fine-tuning gives the best in-distribution exact-match accuracy (best in 15 of 18 settings, never below zero-shot), while GRPO holds only a sub-one-point average edge on cross-dataset transfer and SFT followed by GRPO under a matched data budget is rarely best at either.Using the tools it is given · reviewed by AI
- When post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.Predicting future events · reviewed by AI
- For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.Grounding answers in the image · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Prefixes from a same-family model of comparable quality producing pass@k gains equal to out-of-family prefixes, or out-of-family prefixes failing to raise pass@k above vanilla GRPO at matched compute.