long-horizon-repository-level-coding-agents-small-4b-critic-trained-diagnose CLM-0432For long-horizon repository-level coding agents, a small (4B) critic trained to diagnose each just-completed step with a severity label, error type and causal rationale, and to redirect the next action when severity is critical, repairs more initially-failing tasks with a single branch (14.1% fail-to-pass) than five full resamples of the base agent (13.7%), at about one-fifth the base-agent output tokens.
Capability: Judging another's work as well as a careful reviewer would
Observed on
Four large backbones (GLM-5, Kimi K2.5, Qwen3.5-Plus, DeepSeek V3.2) under Claude Code and Cline-style scaffolds on four SWE benchmarks; conditional protocol where the failing subs.
Sources
- Validated by: Fail-to-pass Pass@N on initially failed tasks and pass-to-pass Avg@N on initially successful tasks, against global resampling, blind breakpoint rollout, and post-hoc RADAR-guided replay at matched branch budgets, with agent output tokens and turns reported; also ROC-AUC of the derived trajectory score and downstream SFT/RL pass rates.. Measured across four models and two scaffolds on four SWE benchmarks with per-slice breakdowns and component ablations. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- When RL-training a central-agent multi-agent harness with 1.5B-3B Qwen2.5 backbones on multi-hop QA, optimizing only the coordinating central policy and leaving sub-agents frozen scores higher than jointly training central and sub-agent policies (average F1 47.79 vs 45.36, accuracy 42.88 vs 40.49 at 3B).Getting more from several models than from one · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- For a small agent policy (Qwen3.5-4B) repeatedly deployed on the same task family with environment-level feedback, distilling only the trajectories whose textual-state guidance is newly or persistently useful relative to state-free execution raises the state-free policy's accuracy more than distilling all state-conditioned trajectories, and the resulting policy keeps improving over alternating rounds of state editing and consolidation.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Under a protocol that charges the critic's own tokens and does not give global rollout the initial failing trajectory as sunk cost, single-branch critic-guided repair matches or falls below matched-token global resampling in fail-to-pass rate.