judge-position-bias CAP-0025 · activeBiased when judging other outputs
A model used as a judge favors whichever answer is shown first, longer answers, and its own outputs.
Also called: LLM-as-judge bias, position bias, verbosity bias, self-enhancement bias
Tags: evaluation, llm-as-judge
When scoring or ranking candidate answers, a capable judge gives the same verdict regardless of the order of candidates, their length, or which model produced them, and agrees with careful human raters.
Claims
- mechanismsingle paper↑1↓0Identifies position, verbosity, and self-enhancement biases in strong judges, while also showing high agreement with humans once mitigated.
- observationreplicated↑2↓0As a raw pairwise judge with no order-swap mitigation, GPT-4 can have its verdict flipped simply by swapping which candidate answer is shown first; applying a position-swap protocol restores agreement with careful human raters to close to human-human agreement levels.
- mechanismsingle paper↑1↓0Swapping candidate order alone can flip the judge's preference, so an evaluator can be gamed by ordering.
Techniques
- Judge both orders and reconcileprocessRun the judge with candidates in both orders and count a preference only when it holds in both.
- Separate the verifier from the producerprocessThe agent that produced an artifact does not judge it; a deterministic sensor or a separate verifier agent does, and reports rather than rewrites.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaThis source: Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 10936 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetchclick for Semantic Scholar2023↑ supports a claim here · ↑ supports a claim here
- Large Language Models are not Fair EvaluatorsThis source: Large Language Models are not Fair EvaluatorsHow much the field cites it — very heavily cited in the last 12 months506 in the last 12 months · 1242 totalpublished 2023checked 2026-09-042023↑ supports a claim here · ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.