8b-multimodal-models-preference-tuning-plain-outcome-reward-grpo-lower CLM-0353For 8B multimodal models, preference tuning and plain outcome-reward GRPO lower hallucination rates while making stated confidence worse calibrated than the untuned backbone, because rank- or correctness-based rewards never score the numerical accuracy of the model's own confidence; adding a Brier-plus-group-ECE calibration term to the GRPO reward cuts calibration error sharply and leaves accuracy metrics largely unchanged.
Capability: Grounding answers in the image
Observed on
InternVL3-8B and Qwen3-VL-8B trained on ~100K VQA/hallucination prompts, evaluated on POPE-adv and HallusionBench with verbalised answer confidence.
Sources
- Validated by: ECE (15 bins), Brier, AUROC of 1-confidence, Selective-Risk@90 on POPE-adv and HallusionBench versus base, HA-DPO, RLHF-V and vanilla GRPO; ECE on POPE-adv 0.272 (GRPO) to 0.085, with hallucination metrics (POPE F1, AMBER CHAIR, MMHal) also best and MME/MMBench/MMStar/SEED unchanged; ablation removing the calibration reward returns ECE to 0.268 with accuracy nearly flat.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- In GRPO-style RL post-training of 7B multimodal reasoning models, composing each optimization group from rollouts stratified by length-normalized sequence surprisal and outcome (confident-correct anchors, confident-wrong negatives, low-confidence successes) within the same step keeps path entropy from collapsing and avoids the late-training performance decline GRPO and DAPO show after a few hundred steps.Retaining skills and safety after fine-tuning · reviewed by AI
- For 2B-8B video multimodal models, adding a small fraction of time-reversed video rollouts into each GRPO group (e.g. 1 reversed out of 7) raises accuracy on action and temporal-reasoning video QA over standard GRPO, while raising the reversed share further (2 or 3 of 7) gives back the gain.Reasoning about time in video · reviewed by AI
- For adapting Qwen3 instruction-tuned models from 0.6B to 32B to tool-calling corpora, LoRA supervised fine-tuning gives the best in-distribution exact-match accuracy (best in 15 of 18 settings, never below zero-shot), while GRPO holds only a sub-one-point average edge on cross-dataset transfer and SFT followed by GRPO under a matched data budget is rarely best at either.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A run where outcome-reward GRPO leaves calibration error at or below the untuned backbone, or where adding the calibration reward lowers ECE only by also lowering task accuracy (so the ablation removing R_cal shows accuracy loss rather than calibration loss).