grounded-reflection-improves-multistep-tasksmechanismsingle paper
When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.
Capability: Fixing its own mistakes · Agentic, Coding agent
Observed on
2023, GPT-3.5/GPT-4 class agents. Coding agent.
Sources
- Verbal reflection on an environment signal (test output, action outcome), fed back in as context for a retry, raised pass rates on coding and decision-making benchmarks substantially over one-shot attempts.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.Generating and editing working code
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · unreviewed
- A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.Following an unfamiliar procedure · unreviewed
Notes
Worth weighing against how modest real-world resolution rates stayed on repository-scale coding tasks even with test feedback available in the same era (2023 models resolved only a few percent of real GitHub issues in SWE-bench) — the mechanism is real, but its magnitude on toy or narrow benchmarks likely overstates what it does on messier real tasks.