reward-models-scored-sequence-log-probability-contextual-preference-pairs-abstentions CLM-0477mechanismsingle paperReviewed by AI
When reward models are scored by sequence log-probability on contextual preference pairs, abstentions are systematically shorter than the unsupported answers they are preferred over, so measured grounding/abstention judgement partly tracks response length; rewriting rejected and refusal responses to match the preferred ones in length while keeping the grounding failure raises pairwise accuracy far more than adding more in-domain legal preference data (up to +25.6 points over baseline for Ministral-8B).
Capability: Judging another's work as well as a careful reviewer would
Observed on
Pairwise contextual reward models (0.5B-27B open-weight) refined with DPO and scored by (length-normalised) sequence log-probability on legal RAG preference pairs built from Victor.
Sources
- Validated by: Pairwise accuracy on LegalRewardBench: Ministral-8B 59.3 baseline, 67.6 with CJB-only DPO, 66.0 with CJB+original legal pairs, 81.8 with length-augmented legal pairs, 84.9 with CJB+length-augmented; same ordering for Gemma 2-2B.. Measured on 156 test preference pairs of one benchmark, two fine-tuned models for the main table; the augmentation is confounded with rewriting (a rewrite may change more than length), and no ablation separates length from other rewrite effects. All labels are LLM-judge derived (91.7% pair-direction agreement across four judges).
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For agentic retrieval-augmented multi-hop QA with 7B-class policy models, scoring each intermediate step for both logical validity and grounding in the retrieved passages — rather than scoring steps by the final answer — yields higher EM/F1 than outcome-reward and final-answer-based process-reward training, because it removes preference pairs where flawed retrieval happened to reach the right answer.Carrying a chain of inference through to its conclusion · reviewed by AI
- When an LLM judge must pick the human-preferred response out of more than two candidates, the assumption behind confidence-threshold abstention — that higher estimated confidence means lower disagreement with humans — degrades as the candidate count grows, because probability mass is normalized across more options; restricting the decision to a conformal top-k shortlist and thresholding a margin inside it restores the monotone relation.Saying it does not know · reviewed by AI
- In label-free RL post-training of reasoning models, deriving each model's reward from a separately-trained peer model's majority vote — rather than from its own majority vote — avoids the reward collapse and length degeneration that self-rewarding runs show, and the benefit grows as the paired models differ in family or size because their errors overlap less.Getting more from several models than from one · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- For small instruction-tuned models (3-4B) answering closed-form true/false questions under a stated user opinion, scoring each sampled response by how much more common its answer is than the group itself predicted, and using that score as the GRPO reward, reduces answer flips under user pressure and raises accuracy without any labels, roughly matching label-supervised desycophancy fine-tuning at higher compute cost.Telling the user what they want to hear · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Length-balanced augmentation giving no gain over the unaugmented legal pairs under a length-normalised scoring function, or the gain persisting when response lengths in the original pairs are already matched.