external-feedback-repair-works-only-with-real-groundingReflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.
Capability: Fixing its own mistakes · Agentic, Coding agent
Observed on
2023, GPT-3.5/GPT-4 class.
Sources
- Reflection on a real environment signal (test result, action outcome), fed back for a retry, substantially raised pass rates on coding and decision-making tasks.
- The complementary direction: with no external feedback, review-and- revise did not improve reasoning accuracy and often degraded it — isolating grounding as the active ingredient.
- Reports that self-generated feedback alone improves outputs across several generation tasks, without requiring an external signal — which is broader efficacy than this claim allows.
Suspected axis of disagreement(a guess, not verified)
Probably task family and how "improvement" was measured. The contesting result is strongest on open-ended generation, where better is a fuzzy judgment a second pass can nudge; the supporting results are on checkable tasks with a pass/fail signal. A secondary possibility is baseline setup — whether the no-revision comparison got the same inference budget. I have not reconciled the experimental setups myself.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.Fixing its own mistakes · unreviewed
- Without an external, ground-truth signal — a failing test, a compiler error, a verifier's output — a model's own critique of its reasoning is not a reliable improvement signal, and asking it to review and revise a correct answer often turns it into a wrong one.Fixing its own mistakes
- The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.Fixing its own mistakes · unreviewed
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- In multi-step visual QA, a critic step run by the same model improves final accuracy only when it is shown the candidate evidence alone and denied the generator's textual rationale; conditioned on that rationale, the critic ratifies the generator's initial hypothesis and the loop performs worse than no critic at all.Fixing its own mistakes · unreviewed
Notes
Filed as an efficacy claim rather than as a property of the technique, so the scope condition ("only with real grounding") and the contest are both first-class. Marked replicated because two independent papers converge on grounding being the active ingredient from opposite directions, not because any single result was rerun.