self-repair CAP-0044 · activeFixing its own mistakes
Whether and how a model can diagnose and correct a wrong answer once it's already made.
Tags: agentic, coding-agent, math
This topic covers self-correction and self-critique behavior: does reviewing or revising an answer make it better or worse, and what turns out to matter is what's driving the revision — the model's own unaided judgment, or a concrete external signal like a failing test. The two are easy to conflate under one banner ("self-correction") and behave very differently, which is exactly the kind of thing a bare "capability score" would have hidden.
Claims
- observationsingle paper↑1↓0For 8B instruction-tuned backbones run in an iterative typed-error-operator correction loop, accuracy gains concentrate in the first few iterations and then plateau rather than decline as the denoising budget grows, so a small step budget captures nearly all the benefit.
- mechanismsingle paper↑1↓0For 8B instruction-tuned backbones, training a LoRA corrector to diagnose a typed, executable error operator (error type, step location, corrupted and repaired proposition) and apply its inverse as a localized edit raises accuracy over free-form self-revision baselines (Self-Refine, RISE, SCoRe) on math, code and commonsense tasks, and dropping either the operator-prediction loss or the adjacent-state reconstruction loss loses most of the gain.
- observationsingle paper↑2↓0With weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.
- mechanismmechanism reasoning↑3↓0In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
- mechanismreplicatedcontested↑2↓1Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.
- observationsingle paper↑1↓0For a frozen LLM run in a generate-critique-revise loop with no external feedback, the accuracy gain is concentrated where single-shot reasoning is already unreliable: on clean multi-step reasoning (BBH, ~74% single-shot) depth makes no difference beyond the confidence interval, while on harder math (MATH, 26.2% single-shot) the same loop gains +14.2 points, with GSM8K in between (+4.2).
- mechanismsingle paper↑1↓0When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.
- observationsingle paper↑1↓0For LLM-driven heuristic discovery on combinatorial optimization tasks (TSP, CVRP, bi-objective and multi-component variants), letting an agent decide when to read, edit, diagnose and evaluate code while persisting each evaluation outcome as a retrievable experience card reaches better final heuristic quality than fixed evolutionary pipelines (EoH, ReEvo, FunSearch, MCTS-AHD, MEoH, MOTIF) at a fraction of their evaluation budget, and re-running the same task warm-started from the previous run's best code improves further round after round under a fixed budget.
- observationsingle paper↑1↓0For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.
- mechanismsingle paper↑1↓0In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.
- observationsingle paper↑1↓0In open-ended editing tasks with no unique correct output (multi-slide design editing), a separate judge model that scores the structural diff between the original and edited artifact against the instruction, and returns its critique verbatim as the next-turn instruction, raises instruction-following over a single pass, but the gain is selective: most tasks pass on the first attempt, and a minority of the ones that enter the loop get worse unless an earlier iteration is restored when the judge's own score declines.
- mechanismsingle paper↑1↓0Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.
- mechanismsingle papercontested↑1↓1Prompting a model to generate its own feedback on a draft output and revise accordingly improves quality on open-ended generation tasks (dialogue, code, review-writing) over a single-shot attempt.
- mechanismmechanism reasoning↑3↓0The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.
- mechanismsingle paper↑1↓0Tool-augmented multimodal models trained only on how to call vision tools accept tool outputs uncritically; adding supervised trajectories where tool evidence is checked against the image and re-queried, plus a reward for correction that reaches a correct answer, raises accuracy on perception-heavy benchmarks (counting, depth, spatial relations) in 3B and 7B Qwen2.5-VL, while slightly lowering it on knowledge-heavy ones.
Techniques
- Correct reasoning by inverting typed error operatorstrainingCorrupt correct reasoning traces by composing typed executable error operators, train an adapter to predict the active operator plus the adjacent lower-noise trace, and at inference iteratively predict and apply inverse operators to the mod
- Deterministic checks before LLM judgesprocessWire linters, test suites and schema validators into the agent's loop before adding any model-based judgment.
- Let the agent revise its own harness, regression-gatedprocessCluster failed traces into signatures, have the same fixed model propose a few minimal harness edits, and promote only edits that improve one split without degrading another.
- Repair with external feedbackprocessGive the model a concrete external signal, such as test output, and let it reflect and retry.
- Separate the verifier from the producerprocessThe agent that produced an artifact does not judge it; a deterministic sensor or a separate verifier agent does, and reports rather than rewrites.
- Stop the self-critique loop on a CONFIRMED sentinelpromptingRun generate-critique-revise to a fixed max depth but exit as soon as the critique emits a CONFIRMED token, instead of trusting a self-reported numeric confidence tag.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- Towards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical Study2026-09-19↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here · ↑ supports a claim here
- ↑ supports a claim here
- Large Language Models Cannot Self-Correct Reasoning YetThis source: Large Language Models Cannot Self-Correct Reasoning YetHow much the field cites it — very heavily cited in the last 12 months503 in the last 12 months · 1164 totalpublished 2023checked 2026-09-042023↑ supports a claim here · ↑ supports a claim here · ↑ supports a claim here · ↓ contests a claim here · ↑ supports a claim here
- Maintainability sensors for coding agents2026 · post↑ supports a claim here
- Reflexion: Language Agents with Verbal Reinforcement LearningThis source: Reflexion: Language Agents with Verbal Reinforcement LearningHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 5075 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch2023↑ supports a claim here · ↑ supports a claim here
- Self-Refine: Iterative Refinement with Self-FeedbackThis source: Self-Refine: Iterative Refinement with Self-FeedbackHow much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 4530 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch2023↓ contests a claim here · ↑ supports a claim here
- Harness engineering for coding agent users2026 · post↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.