rl-trained-multimodal-image-quality-models-applying-reasoning-reward-only-reasoning CLM-0630In RL-trained multimodal image-quality models, applying the reasoning reward only to reasoning tokens and the human-rating reward only to rating tokens keeps rating alignment at the level of rating-only training while turning reasoning-guided edit quality gain positive, and it prevents the collapse to a single universal edit that an unmasked shared reward produces despite that variant's larger raw gain.
Capability: Saying what actually drove the answer
Observed on
4B-class MLLM actor (Qwen3.5-4B) trained with GRPO on KonIQ-10k, with a frozen diffusion editor and a frozen MLLM judge supplying the reasoning signal; faithfulness measured as jud.
Sources
- Measured on one actor backbone and one training split, with the final E5 comparison confounded by a difference in KL scope (the authors flag this and defer the controlled comparison to an appendix). Faithfulness is operationalized as judge-scored quality gain, which the paper itself concedes conflates identifying the limiting factor with proposing a good fix; semantic drift in edits is unmeasured.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In RL-trained multimodal image-quality models, supervising reasoning with the rating reward alone raises rating alignment while making the stated reasoning less faithful — rating-only training reached an average PLCC/SRCC of 0.836/0.815 yet its reasoning-guided edits lowered judged image quality (mean gain -0.260, worse than the untrained baseline's -0.136).Saying what actually drove the answer · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
- For 4-8B multimodal models on long-video QA, RL training on questions authored from short scene segments but answered over the whole video improves benchmark accuracy more than RL on questions authored from the whole video, because segment-authored questions target details a whole-video view overlooks; including the segment timestamp in the training instruction adds a further gain by removing referential ambiguity, and is not needed at inference.Reasoning about time in video · reviewed by AI
- In multi-reward RL fine-tuning where several valid reward dimensions are scalarized with fixed weights, the policy can raise total reward by saturating the easiest and densest dimensions while the hardest one degrades, because the scalar projection makes different reward profiles indistinguishable to the advantage estimator and shortcut profiles then crowd informative samples for the neglected dimension out of rollout groups.Gaming the measure instead of doing the task · reviewed by AI
- Tool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.Fixing its own mistakes · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: An ablation under matched KL scope and compute where masked token-specific credit gives no higher edit-induced quality gain than a shared reward at equal solution diversity, or where the unmasked variant does not collapse to a universal edit. Drafted stance toward rl-trained-multimodal-image-quality-models-supervising-reasoning-rating-reward-alone: supports -- Rating-only training here matched the best rating alignment while producing edit gains worse than the untrained baseline, repeating the finding that rating reward alone does not buy faithful reasoning.