llm-multi-agent-systems-failing-long-horizon-web-tasks-re-running-whole CLM-0598For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.
Capability: Getting more from several models than from one
Observed on
536 evaluator-confirmed failure trajectories from 200 WebArena-Verified Hard and AssistantBench tasks run on AG2, CrewAI and Magentic-One, all with one model (deepseek-v4-flash) at.
Sources
- Validated by: Pass rate on 536 recorded failures against three task-level baselines given three attempts each and two node-level controls (random node, last node) given the same single replay intervention: 20.15% vs 6.90% best task-level, 3.73% random-node, 1.31% last-node; best on each of the three frameworks after Holm correction. Failure reproduction rose from 67.97% to 80.78% per execution.. Measured on one dataset, three frameworks, one model at temperature 0; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- For LLM agents that score their own actions during exploration in text-based games and then reuse that experience, the accuracy of those self-scores against the environment verifier does not predict later gains on held-out stricter configurations: block-level correlations between judgment agreement or calibration and the next checkpoint's score change sit near zero, so recognizing a good action is not the bottleneck — converting it into a transferable policy is.Improving its own scaffolding from failures · reviewed by AI
- For long-horizon LLM agents in environments where an erroneous action changes world state, restoring an earlier checkpoint and carrying forward a structured reflection memory that excludes branch-local state claims beats either intervention alone, because reflection alone cannot undo environmental damage and rollback alone re-explores the same failure.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A run where self-reflection or critic-agent repair beats unguided full rerun at equal attempt budget on these trajectories, or where random-node or last-node intervention under selective replay matches symptom-guided intervention. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: contests -- Self-reflection and critic-agent repair, which in this setting do receive the external evaluator outcome of a failed run, performed no better than unguided rerun under an equal attempt budget.