swapping-candidate-order-alone-can-flip-judges-preference CLM-0163mechanismsingle paperReviewed by AI and Russ Miller
Swapping candidate order alone can flip the judge's preference, so an evaluator can be gamed by ordering.
Capability: Biased when judging other outputs
Sources
- Swapping candidate order alone can flip the judge's preference, so an evaluator can be gamed by ordering.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- As a raw pairwise judge with no order-swap mitigation, GPT-4 can have its verdict flipped simply by swapping which candidate answer is shown first; applying a position-swap protocol restores agreement with careful human raters to close to human-human agreement levels.Biased when judging other outputs
- When reranking sampled Verilog candidates with both a generated-testbench execution signal and an LLM judge, letting the judge see the execution outcomes in its prompt lowers final Pass@1 relative to acquiring the two signals separately and combining them only as a weighted score at selection time, because the judge anchors on the test verdicts (approving all-pass candidates with uncovered logic errors, over-penalizing failures caused by bad testbenches) and stops contributing information the tests miss.Generating and editing working code · reviewed by AI