llm-multi-agent-systems-reflect-after-failure-restricting-reflection-single CLM-0425In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.
Capability: Getting more from several models than from one
Observed on
Task-oriented collaborations with clear success criteria and small agent teams (Magentic-One over HotPotQA, ChartQAPro, Mind2Web), frozen GPT-4o-mini actors with a fine-tuned 8B re.
Sources
- Validated by: Success rate on 100 sampled tasks per dataset over five trials: +22/26/27 points over initial success on HotPotQA, ChartQAPro, Mind2Web, above Reflexion, Retroformer and COPPER; ablation removing diagnosis+correction gave the largest drop; the attribution component beat All-at-once/Step-by-step/Binary-search on agent- and step-level accuracy on held-in and held-out Who&When data.. Measured on three benchmarks with 100-task test sets and five trials, against three reflection baselines, plus ablations. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- For long-horizon LLM agents in environments where an erroneous action changes world state, restoring an earlier checkpoint and carrying forward a structured reflection memory that excludes branch-local state claims beats either intervention alone, because reflection alone cannot undo environmental damage and rollback alone re-explores the same failure.Completing tasks that take many steps or many hours · reviewed by AI
- In multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Stating false facts confidently · reviewed by AI
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A setup where all-agent reflection matches or beats decisive-agent-only reflection at equal compute, or where reflections written by correctly-behaving agents are shown not to degrade their later performance.