automatically-rewriting-prompts-multi-agent-llm-system-failures-locating-responsible CLM-0365When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.
Capability: Improving its own scaffolding from failures
Observed on
Multi-agent pipelines of a few LLM agents with a scalar reward and a training set; tested on HotpotQA, HoVer, PUPA, IFBench and MATH with GPT-5-mini and Qwen3-8B..
Sources
- Validated by: Against MIPROv2, TextGrad, GEPA and a no-optimization baseline on five MAS benchmarks, three seeds: average gain over no-optimization +11.76 with GPT-5-mini (vs +9.24 for GEPA) and +9.67 with Qwen3-8B; wall-clock optimization time 2.5x lower than the next-fastest baseline; ablation adds target identification (+1.44 HotpotQA, +3.84 PUPA), agent-level supervision, and gradient clustering separately; optimized prompts also transferred best on five unseen in-domain benchmarks.. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM multi-agent systems that reflect after failure, restricting reflection to the single agent responsible for the first incorrect step — located by a trained step-level correctness scorer and paired with a counterfactually generated corrected step — raises task success more than having every agent reflect on the trajectory, because reflections written by agents that behaved correctly enter their memory as misleading guidance for later tasks.Getting more from several models than from one · reviewed by AI
- For LLM agents that must compose and then execute domain tool workflows, training separate adapters for the two roles splits the gains cleanly: supervised fine-tuning of the planner on reference tool sequences mainly improves workflow-order fidelity, while node-level RL on argument-match rewards mainly improves tool-argument grounding and final-answer quality, and combining both beats either on a 9B backbone.Planning a sequence of actions before acting · reviewed by AI
- When a tool-using agent is trained with outcome-only RL and a surface prompt cue co-occurs with one tool in most training examples, the agent invokes that tool on tasks where it is useless, but only for tools it has already learned to use well and mainly when the cue is semantically related to the tool.Using the tools it is given · reviewed by AI
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For LLM web agents on cluttered pages containing many concurrent interface components, success falls sharply as component count grows, and having the agent first emit a localized sub-goal plus a task-relevant sub-observation before choosing an action recovers much of the loss, because self-assessment of the action is more reliable in the reduced context.Using the tools it is given · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An ablation or replication in which locating the target agent by hint injection and using its corrected output gives no advantage over whole-system critique at matched rollout budget, or fails to transfer to unseen benchmarks.