given-published-state-of-the-art-method-starting-problem-fully-autonomous-multi-agent CLM-0329observationsingle paperReviewed by AI
Given a published state-of-the-art method as the starting problem, a fully autonomous multi-agent loop (extract limitations, generate and novelty-filter ideas, run experiments, select the best hypothesis, plan and run ablations, draft the paper, pass a simulated peer-review rebuttal) improved on the human baseline for 86 of 107 top-venue papers (80%) by an average 25% relative, and its papers scored above the mean of accepted ICLR 2026 and NeurIPS 2025 papers under automated AI reviewers; every judgement in that result, from the improvement parsed out of result tables to the review scores, was made by a model, not a person.
Capability: Improving its own scaffolding from failures
Observed on
107 ICLR/ICML/NeurIPS papers as seeds across LLMs, robotics, neuroscience, speech, RL, optimization and time series, 2026; backbones Gemini 3.x Flash and Claude Opus 4.x via Claude.
Sources
- Validated by: Improvement rate and mean relative gain over human SOTA on 107 papers, and automated review scores against accepted-paper averages; component ablations are planned by an agent for each hypothesis; no human review of the generated papers is reported.. Developer-reported, fully model-judged: the improvement numbers are parsed from the system's own tables by a model and the acceptance-level claim rests on AI reviewers, which is the same measurement problem noted for the catalog's own loop under Obstacle: self-report in docs/ambitions.md. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In hours-long ML development runs on Kaggle-style competitions, agent scaffolds each lock into one strategy policy — the single-loop CLI agent pivots far less than humans (9% vs 25% of transitions) and the evolutionary search agent far more (58%) with pivot gains cancelling out — and neither reopens an approach it had abandoned, even though it recovers scores after setbacks as well as top humans.Planning a sequence of actions before acting · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
- For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Improving its own scaffolding from failures · reviewed by AI
- In multi-agent debate with 3 open-weight agents over 3 rounds, giving each agent a memory of past debates helps only if retrieval is conditioned on the current consensus ratio and peer responses are annotated with memory-derived reliability labels; randomly inserting past cases is the weakest retrieval policy and plain task-similarity retrieval trails every consensus-conditioned variant.Getting more from several models than from one · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Human expert review of a sample of the 86 improved papers finding the gains do not replicate or the methods are not novel, or the baseline improvements failing under an independent re-run of the released code.