no-one-should-judge-their-own-cause ADG-0019

No one should be judge in their own cause

“A party with a stake in an outcome cannot be trusted to judge it.”

Also: Nemo iudex in causa sua · Don't grade your own homework

Origin

Roman law maxim (nemo iudex in causa sua), a pillar of natural justice in common-law systems. About conflict of interest in people: a judge who benefits from a verdict will, consciously or not, reach it.

Why it should, or should not, apply to models

A model has no stake in the ordinary sense, so if the maxim transfers it is for a different reason: the model that produced an output holds the same context, the same assumptions and the same blind spots that produced the error, so its review re-runs the mistake rather than catching it. That predicts the maxim holds when the reviewer shares the producer's context and weakens when the reviewer has independent grounding (a test result, a verifier, a different model with different context).

Standing: Holds

Standing counts claims a person has reviewed. Claims reviewed by AI alone are listed and marked, and do not move it.

Where it holds only under a condition

Claims that keep the heuristic but bound it.

When an agent labels its own past attempts as successes or failures to build an experience memory, the labels can be substantially wrong without much cost: the LLM judge used was 72.7% accurate against ground truth on WebArena-Shopping, and simulating judges anywhere from 70% to 90% accurate gave similar success rates, with ground-truth labels best — so for distilling lessons, a rough self-judgment is close to enough, although a perfect one still helps.
Improving its own scaffolding from failuresReviewed by AI
For distilling lessons across many attempts, a self-judge only 72.7% accurate cost little; the maxim bites hardest on a single verdict and on errors that are systematic rather than random.
In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
Fixing its own mistakesReviewed by AI
Where a deterministic check exists it should come first; a model judge is acceptable only for properties no rule can express, and then as an advisory signal. The maxim holds for model judges, less for sensors.

Where it holds

Claims showing the heuristic applying to models.

Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.
Fixing its own mistakesReviewed by AI and Russ Miller
Without an external signal, a model's own critique is unreliable and revision often turns a correct answer wrong.
The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.
Fixing its own mistakesReviewed by AI
Argued from the mechanism above: the producer holds the context that skewed the output, so verification belongs with a sensor or a separate verifier.

Add a case where it holds or breaks

A break is worth more than a hold. If you have seen this fail for a model, with the setup written down, that is the most useful thing you can add here.