8b-14b-generative-reward-models-trained-judges-making-model-generate CLM-0350For 8B-14B generative reward models trained as judges, making the model generate input-adaptive evaluation criteria inside its own reasoning chain and rewarding the quality of those criteria with a stronger teacher model raises agreement with preference labels, and the gain scales with the teacher's capability rather than with the presence of the rubric step alone.
Capability: Judging another's work as well as a careful reviewer would
Observed on
Qwen3-8B/14B backbones, SFT on distilled judge traces then GRPO, evaluated on RewardBench, M-RewardBench, MM-Eval, JudgeBench and RewardBench v2.
Sources
- Validated by: Pairwise accuracy on RewardBench, MM-Eval and JudgeBench against the untrained Qwen3-8B backbone (overall 0.809) and against ablations: dropping the rubric reward gives 0.854 and the full model 0.863; swapping the rubric-scoring teacher from Qwen3-32B to Qwen3-Flash to Qwen3-Max moves overall from 0.840 to 0.849 to 0.863.. Measured on four pairwise benchmarks plus one listwise, with reward-component and teacher-capability ablations on the 8B model only. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- In label-free RL post-training of reasoning models, deriving each model's reward from a separately-trained peer model's majority vote — rather than from its own majority vote — avoids the reward collapse and length degeneration that self-rewarding runs show, and the benefit grows as the paired models differ in family or size because their errors overlap less.Getting more from several models than from one · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
- For a same-family pair of reasoning and instruction-tuned 4B models, learning one interpolation ratio per module (attention block, FFN block, embedding, final norm) with a reward that scores correctness and ranks correct responses by length cuts average generated tokens by about 27% relative to the reasoning model while holding average accuracy, and beats fixed global-ratio merging and CMA-ES search over the same ratios.Cutting to the chase · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An equally trained judge whose self-generated rubrics are rewarded by a weak teacher, or not rewarded at all, matching the strong-teacher version's accuracy on the same benchmarks; or the rubric segment adding nothing over a plain reasoning chain of similar length.