Judging another's work as well as a careful reviewer would

Claims

• observationsingle paper↑1↓0
  For 8B-14B generative reward models trained as judges, making the model generate input-adaptive evaluation criteria inside its own reasoning chain and rewarding the quality of those criteria with a stronger teacher model raises agreement with preference labels, and the gain scales with the teacher's capability rather than with the presence of the rubric step alone.
• observationsingle paper↑1↓0
  When a critique's worth is measured by whether a fixed downstream repair agent can improve the interface from it, eight frontier models differ meaningfully in how actionable their UX critiques are, lead on different product-surface categories, and vary in reliability fixture by fixture; the best repair lift was about +0.22 on the benchmark's scale, so the task is unsaturated and not one-dimensional.
• observationsingle paper↑1↓0
  For image-aesthetics models fine-tuned for critique and scoring, competence at producing reference-aligned aesthetic critique does not carry over to judging whether an image suits a stated purpose, audience or cultural setting: all four aesthetic specialists tested scored below 30% accuracy on 301 expert-curated suitability cases (best 29.57%) versus 91.69% for the strongest general-purpose model, and they predicted 'suitable' in 59-77% of binary cases against a gold rate of 38.14%.
• observationsingle paper↑1↓0
  When LLM-generated peer reviews are compared with human reviews point by point under theory-guided critique frameworks, the model and human critique differ in kind rather than in quality: LLM reviews carry more explanatory depth, integrative reasoning and explicit argument structure, while human reviews more often identify higher-order weaknesses and give revision-oriented guidance, and an expert prompt does not close that gap but amplifies the LLM-side tendencies.
• observationsingle paper↑1↓0
  When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.
• observationsingle paper↑1↓0
  When LLMs are used as relevance judges on a specialized non-English (Brazilian Portuguese legal) retrieval collection, they assign systematically higher relevance grades than human experts (mean absolute error 0.46-0.66 on a 0-2 scale, Cohen's kappa with humans only 0.32-0.53), yet the resulting system rankings stay close to the human reference for rank-aware metrics (nDCG@10, MRR, Kendall's tau >= 0.90) and diverge for P@10 and R@10.
• observationsingle paper↑1↓0
  For long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.
• observationsingle paper↑1↓0
  When open-source vision-language models are post-trained with RL or critique tuning to act as judges of chart-to-code outputs, they become more likely to output Accept rather than better at verification: pairwise-comparison accuracy barely improves under an order-swapped protocol, while acceptance rates rise and specificity falls, so they ratify flawed but plausible-looking charts.
• mechanismsingle paper↑1↓0
  When reward models are scored by sequence log-probability on contextual preference pairs, abstentions are systematically shorter than the unsupported answers they are preferred over, so measured grounding/abstention judgement partly tracks response length; rewriting rejected and refusal responses to match the preferred ones in length while keeping the grounding failure raises pairwise accuracy far more than adding more in-domain legal preference data (up to +25.6 points over baseline for Ministral-8B).
• observationsingle paper↑1↓0
  Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.
• mechanismsingle paper↑1↓0
  For VLM-based judging of camera-conditioned world-model videos, splitting a long video pair into action-aligned chunks with structured visual evidence (source image, frame-grid overview, per-action first/last-frame panels) and voting chunk verdicts into video-level preferences agrees with human preferences better than judging the whole video pair at once, because short-lived action evidence is otherwise diluted by the full context.
• mechanismsingle paper↑1↓0
  Within one model family, the relative distribution of reasoning failure modes stays similar from 1.5B to 72B, so critiques mined offline from small models' wrong answers, retrieved by failure mode and inserted as in-context exemplars, raise a large model's math accuracy above random, fixed, or embedding-similarity exemplar selection at a single generation.

Techniques

• Interrupt the agent when a step-level critic flags critical process
  A lightweight generative reward model, trained on causal-chain-backtracked step labels from failed trajectories plus injected errors in successful ones, asynchronously diagnoses the agent's last completed step and forces regeneration of the
• Judge video pairs in action-aligned chunks, then voteprompting
  Decompose a paired video comparison into chunks of four consecutive actions, judge each chunk's action consistency and visual quality from a six-image structured evidence panel, and aggregate chunk verdicts by voting into video-level prefer
• Length-balance rejected responses in preference datatraining
  Rewrite refusal and rejected responses to match the length and structure of the preferred grounded answer while preserving the underlying grounding failure, so preference learning cannot key on length.
• Profile review critique by function, not human-likenessprocess
  Annotate each review point against knowledge type, argument structure, question depth, cognitive complexity and feedback function so model and human critique are compared by what functions they perform rather than by textual overlap.
• Retrieve in-context critiques by weak-model failure modeprompting
  Build an offline bank of wrong answers from small same-family models labelled with clustered failure modes plus natural-language critiques, then retrieve critique exemplars for a large model either by its dominant family-level failure profi
• Reward the judge's self-generated rubric, scored by a teachetraining
  The judge model emits an analysis segment, then task-generic and instruction-specific scoring criteria, then its evaluation, and during GRPO a stronger teacher model scores the generated criteria as one term of the reward.
• Score a critique by the repair it enablesprocess
  Instead of rating a critique's text, hand it to a fixed repair agent and measure the improvement of the artefact after repair, so a critique is judged by what a reader can do with it.
• Strip prior-score metadata from judge promptsprompting
  Remove revision, attempt-index and prior-score fields from the judge's context so each evaluation sees only task, answer and rubric.

Sources

Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.

• ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation2026-09-21
  ↑ supports a claim here
• FLARE: A Full-Lifecycle Dense Supervision Paradigm for Long-Horizon Coding Agents via Generative Reward Model2026-09-20
  ↑ supports a claim here
• Building Legal Reward Models for Grounding and Abstention2026-09-13
  ↑ supports a claim here
• UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms2026-09-05
  ↑ supports a claim here
• WorldReward: Reward Modeling for Camera-Conditioned World Models2026-09-03
  ↑ supports a claim here
• Beyond Human-Likeness: Mapping the Scientific Critique Profiles of LLMs and Human Reviewers2026-09-01
  ↑ supports a claim here
• AesCanvas: A Large-Scale Dataset and Benchmark for Aesthetic Critique and Contextual Suitability2026-08-27
  ↑ supports a claim here
• NormasTCU --- A Brazilian Portuguese IR Dataset and an Evaluation of LLM-as-a-Judge for Relevance Assessment2026-08-27
  ↑ supports a claim here
• CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes2026-08-27
  ↑ supports a claim here
• Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence2026-08-26
  ↑ supports a claim here
• Training-Free Inference-Time Self-Reflection and Cost-Bounded Early Stopping for Large Language Models2026-08-19
  about this capability · no claim drafted yet
• UXBench: Measuring the Actionability of LLM-Generated UX Critiques2026-06-15
  ↑ supports a claim here
• Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators2025-04-21
  ↑ supports a claim here