identifies-position-verbosity-self-enhancement-biases-strong-judges-while CLM-0072

mechanismsingle paperReviewed by AI and Russ Miller

Identifies position, verbosity, and self-enhancement biases in strong judges, while also showing high agreement with humans once mitigated.

Capability: Biased when judging other outputs

Sources

supportsJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Identifies position, verbosity, and self-enhancement biases in strong judges, while also showing high agreement with humans once mitigated.

Reviewed by AI and Russ MillerLast checked: 2026-09-03Evidence activity

Either way, bring evidence: a link, or what you saw.

Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.

Related claims

When an LLM judge is validated against expert annotators on multi-dimensional rubrics for open-ended agent outputs, a single pooled agreement figure overstates reliability: per-dimension human-human agreement varies widely (substantial on a recall-style Completeness dimension, only slight to fair on Relevancy, Structural Clarity and Scope Adherence), the pooled statistic is inflated by between-dimension variation, and the judge's residual disagreements are directional (consistently one step stricter on Completeness, one step more generous on Structural Clarity) rather than noise.Whether the measurement made the finding · reviewed by AI
When an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.Improving its own scaffolding from failures · reviewed by AI
When an LLM judge must pick the human-preferred response out of more than two candidates, the assumption behind confidence-threshold abstention — that higher estimated confidence means lower disagreement with humans — degrades as the candidate count grows, because probability mass is normalized across more options; restricting the decision to a conformal top-k shortlist and thresholding a margin inside it restores the monotone relation.Saying it does not know · reviewed by AI
Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.Judging another's work as well as a careful reviewer would · reviewed by AI