deterministic-sensors-should-precede-llm-judges CLM-0043mechanismmechanism reasoningReviewed by AI
In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
Evidence for: No one should be judge in their own cause (narrows)
Capability: Fixing its own mistakes · Agentic, Coding agent
Applies when
- Effect on the capability:
- helps
- Needs:
- an environment where code can run
- Cost:
- low
- Fails when:
- The property has no deterministic check. A model judge is then an advisory signal only, until it has been measured against human review.
Sources
- Table III sets out the cost and determinism contrast and the design rule that follows from it. Argued from the properties of the two sensor kinds, with a worked cost example, not measured.
- The adjacent finding: with no external signal, a model's own review does not improve reasoning and often degrades it. A deterministic sensor is the external signal; an LLM judge without grounding is closer to self-review.
- A follow-up devoted to sensors for coding agents, with the same ordering: mechanical checks first, model-based judgment where rules run out.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.Fixing its own mistakes · reviewed by AI
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.Fixing its own mistakes · contested
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.Generating and editing working code · reviewed by AI
Notes
Sits beside external-feedback-repair-works-only-with-real-grounding and says why: the grounding that makes reflect-and-retry work is exactly what a computational sensor provides and an inferential one does not.