grpo-style-rl-post-training-7b-multimodal-reasoning-models-composing-each CLM-0401observationsingle paperReviewed by AI
In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.
Capability: Retaining skills and safety after fine-tuning
Observed on
Qwen2.5-VL-7B-Instruct base, verifiable multimodal math/diagram reasoning, ~400 update steps, group size 16, vision encoder frozen; single reported run; does not cover open-ended g.
Sources
- Validated by: Ablation on MathVista/MathVision/MMMU Pass@8: GRPO 60.7 avg, DAPO 62.0, +HSR alone 64.7, +PAQM alone 63.8, both 65.8; plus path-entropy curves staying ~0.35-0.60 nats versus baseline collapse. Baselines reported at peak pre-collapse. One run, no seed variance.. Measured on three benchmarks in one ablation table plus training curves; the PAQM-vs-full comparison isolates group composition from the teacher-hint curriculum, but baselines are scored at hand-picked peak steps, only one run is reported, and the whole result rests on a single base model.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In GRPO-style RLVR on math for 7B models, injecting partial reasoning prefixes from a small out-of-family model (Gemma-2-2B, LLaMA-3.2-1B) into a fraction of rollouts keeps policy entropy higher early in training and raises pass@k, with gains growing as k rises; the effect comes from distributional unfamiliarity rather than prefix quality, since prefixes from same-family models of equal or higher quality give no gain over vanilla GRPO.Retaining skills and safety after fine-tuning · reviewed by AI
- When post-training a small LLM (Qwen2.5 0.5B-3B) to produce weekly asset-return forecasts from news, rewarding rank correlation of a full listwise asset ordering gives a dense learning signal where exact-match GRPO gives almost none (all sampled actions scoring zero on ~35% of steps), and yields higher Spearman correlation and backtest Sharpe than the SFT checkpoint, standard GRPO, and prompted commercial models.Predicting future events · reviewed by AI
- For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.Grounding answers in the image · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- When training 4B-8B Qwen3 agents with group-based RL (GRPO, DAPO, GSPO) on sparse-reward long-horizon tasks, injecting hints into the environment's feedback — procedural action guidance in early episode steps and early training, switching to enriched state observations in later steps and later training — raises success on the unmodified environments by a few points on average, while lowering performance on tasks that require questioning whether the request is answerable at all.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched-compute replication where stratified group assembly shows the same entropy decline and late-training degradation as GRPO, or where the ablation gain disappears across seeds.