deterministic-sensors-should-precede-llm-judgesmechanismmechanism reasoningpending review
In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Fixing its own mistakes · Agentic, Coding agent
Sources
- Table III sets out the cost and determinism contrast and the design rule that follows from it. Argued from the properties of the two sensor kinds, with a worked cost example, not measured.
- The adjacent finding: with no external signal, a model's own review does not improve reasoning and often degrades it. A deterministic sensor is the external signal; an LLM judge without grounding is closer to self-review.
- A follow-up devoted to sensors for coding agents, with the same ordering: mechanical checks first, model-based judgment where rules run out.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.Fixing its own mistakes · unreviewed
- Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.Fixing its own mistakes · contested
- In multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Stating false facts confidently · unreviewed
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
- Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Fixing its own mistakes
Notes
Sits beside external-feedback-repair-works-only-with-real-grounding and says why: the grounding that makes reflect-and-retry work is exactly what a computational sensor provides and an inferential one does not.