external-feedback-repair-works-only-with-real-grounding CLM-0045Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.
Capability: Fixing its own mistakes · Agentic, Coding agent
Observed on
2023, GPT-3.5/GPT-4 class.
Model cards: GPT-4
Applies when
- Effect on the capability:
- helps
- Needs:
- an external signal (a test, a compiler, an environment outcome)
- Fails when:
- The feedback in the loop is the model's own unaided critique rather than a test result, compiler error or environment outcome.
Sources
- Reflection on a real environment signal (test result, action outcome), fed back for a retry, substantially raised pass rates on coding and decision-making tasks.
- The complementary direction: with no external feedback, review-and- revise did not improve reasoning accuracy and often degraded it — isolating grounding as the active ingredient.
- Reports that self-generated feedback alone improves outputs across several generation tasks, without requiring an external signal — which is broader efficacy than this claim allows.
Suspected axis of disagreement(a guess, not verified)
Probably task family and how "improvement" was measured. The contesting result is strongest on open-ended generation, where better is a fuzzy judgment a second pass can nudge; the supporting results are on checkable tasks with a pass/fail signal. A secondary possibility is baseline setup — whether the no-revision comparison got the same inference budget. I have not reconciled the experimental setups myself.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.Fixing its own mistakes · reviewed by AI
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
- When self-reflection is used as a training signal for long-horizon agent and math tasks, resetting to the original prompt with a compact reflection patch prepended beats accumulating reflections inside the ongoing trajectory, and overly long reflections (more than ~10 points) hurt compared with 2-5 bullet points.Completing tasks that take many steps or many hours · reviewed by AI
- For long-horizon LLM agents in environments where an erroneous action changes world state, restoring an earlier checkpoint and carrying forward a structured reflection memory that excludes branch-local state claims beats either intervention alone, because reflection alone cannot undo environmental damage and rollback alone re-explores the same failure.Completing tasks that take many steps or many hours · reviewed by AI
Notes
Filed as an efficacy claim rather than as a property of the technique, so the scope condition ("only with real grounding") and the contest are both first-class. Marked replicated because two independent papers converge on grounding being the active ingredient from opposite directions, not because any single result was rerun.