As a raw pairwise judge with no order-swap mitigation, GPT-4 can have its verdict flipped simply by swapping which candidate answer is shown first; applying a position-swap protocol restores agreement with careful human raters to close to human-human agreement levels.

Evidence for: Law of triviality (holds)

Capability: Biased when judging other outputs

Observed on

GPT. 2023, GPT-4 class. LLM as judge.

Model cards: GPT-4

Sources

supportsLarge Language Models are not Fair Evaluators
Swapping the order of two candidate answers alone could flip GPT-4's preference between them.
supportsJudging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Identifies the same position bias (plus verbosity and self-enhancement bias) in strong judges, and shows that mitigating it restores high agreement with human raters.

Reviewed by AI and Russ MillerLast checked: 2026-09-03Evidence activity

Either way, bring evidence: a link, or what you saw.

Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.

Related claims

Swapping candidate order alone can flip the judge's preference, so an evaluator can be gamed by ordering.Biased when judging other outputs
GPT-4 could reliably name a well-known celebrity's parent, but was far less reliable naming the celebrity when given the parent — the same directional-recall asymmetry demonstrated in smaller fine-tuned models, showing up in a deployed model that wasn't fine-tuned for the test.Not generalizing "A is B" to "B is A"