llm-judge-given-prior-evaluation-score-context-metadata-revision CLM-0596When an LLM judge is given a prior evaluation score as context metadata on a revision, its rubric scores shift toward that uninformative prior — seven of eight tested models (API and open-weight, 0-5 scale, 20 fixed texts) show a negative total shift, and on a categorical compliance task the metadata blocks 48% of error corrections and flips about 10% of previously correct judgments to the anchored wrong label.
Capability: Judging another's work as well as a careful reviewer would
Observed on
Single-answer rubric grading on a 0-5 scale with anchors drawn below the 4.0 acceptance threshold; 20 generated-and-screened texts across four task categories; one 441-sample indus.
Sources
- Validated by: Compared C0 (no metadata) against C2 (anchored metadata) over 185,271 valid evaluations with task-stratified bootstrap intervals; Chain-of-Thought and an explicit disregard-metadata warning were tested as alternatives and neither reduced the total effect on the numerical pair, though the warning zeroed the paired accuracy effect on the categorical task.. Measured: 192,000 attempted calls, 185,271 valid, 8 models, 4 temperatures, 20 tasks, with task-stratified bootstrap rather than treating decodings as replicates. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When an LLM judge must pick the human-preferred response out of more than two candidates, the assumption behind confidence-threshold abstention — that higher estimated confidence means lower disagreement with humans — degrades as the candidate count grows, because probability mass is normalized across more options; restricting the decision to a conformal top-k shortlist and thresholding a margin inside it restores the monotone relation.Saying it does not know · reviewed by AI
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- In LLM fact-checking over supplied evidence, high veracity accuracy does not imply the verdict depends on that evidence: when the model's own cited sentences are iteratively removed, off-the-shelf models (GPT-4o-mini, Gemini-2.5-Flash-Lite, Qwen-2.5-32B, Llama-3.1-8B) keep their SUPPORT/REFUTE labels at a non-zero plateau instead of abstaining, and training with paired supervision that requires NOT_ENOUGH_INFO on the evidence-ablated input removes most of that persistence without losing accuracy.Checking claims against evidence · reviewed by AI
- When LLMs are used as relevance judges on a specialized non-English (Brazilian Portuguese legal) retrieval collection, they assign systematically higher relevance grades than human experts (mean absolute error 0.46-0.66 on a 0-2 scale, Cohen's kappa with humans only 0.32-0.53), yet the resulting system rankings stay close to the human reference for rank-aware metrics (nDCG@10, MRR, Kendall's tau >= 0.90) and diverge for P@10 and R@10.Judging another's work as well as a careful reviewer would · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A design that includes prior-score metadata with a matched control containing an equally salient irrelevant number, showing no difference in mean judge score or in correction/induction rates against human labels.