production-ads-ranking-stack-agent-chains-hypothesis-generation-experiment-execution CLM-0335observationsingle paperReviewed by AI
In a production ads-ranking stack, an agent that chains hypothesis generation, experiment execution, result analysis and a shared knowledge store, with human checkpoints at stage boundaries, delivered several times the completed ML iterations per engineer-week of the manual baseline and a 2.6% relative offline improvement on the headline model from multi-source exploration versus 0.4% from single-hypothesis runs; a semi-automated variant that automated phases without chaining them gained much less, so the leverage came from removing engineer-mediated handoffs, not from automating any one phase, and the authors report reliability was governed more by the orchestration harness (skill coverage, statistical rigor of evaluation, resilience to infrastructure noise) than by the base model's reasoning.
Capability: Improving its own scaffolding from failures
Observed on
Industrial ads-ranking model portfolio, 2026, one evaluation window; developer-reported; throughput multiple stated qualitatively as 'multiple times'; five recurring failure modes .
Sources
- Industrial system with no public code or data; the numbers are the authors'. The 'harness dominates model' lesson is stated from experience and partly supported by their cross-LLM study (filed as a separate claim). The failure-mode list is concrete and reusable.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For single-agent procedural tasks where the relevant state fits a fixed schema known in advance, replacing the appended conversation transcript with a structured execution state that the model rewrites each step — discarding the reasoning trace once the state patch is validated — holds accuracy flat as the horizon grows while history-appending runtimes degrade, and the gain comes from the structure rather than the shorter prompt.Completing tasks that take many steps or many hours · reviewed by AI
- For a frozen-backbone multi-agent scientific discovery pipeline on HypoBench tasks, distilling reusable procedural skills separately per stage — from discriminator critique of a whole hypothesis batch for generation, and from execution outcomes compared to ground-truth results for testing — beats no skill, an LLM-written skill, and a hand-written skill, and removing each stage's feedback signal costs most of the gain.Improving its own scaffolding from failures · reviewed by AI
- On cross-system urban tasks where several services must be chained under matched tools and budgets, the task-success gap between orchestration strategies is not explained by required-tool coverage, call ordering, or asking for missing inputs — those are near-saturated across strategies — but by whether the agent goes beyond the minimal toolchain to refine coarse results and verify each candidate against the stated constraints before answering.Using the tools it is given · reviewed by AI
- Given a published state-of-the-art method as the starting problem, a fully autonomous multi-agent loop (extract limitations, generate and novelty-filter ideas, run experiments, select the best hypothesis, plan and run ablations, draft the paper, pass a simulated peer-review rebuttal) improved on the human baseline for 86 of 107 top-venue papers (80%) by an average 25% relative, and its papers scored above the mean of accepted ICLR 2026 and NeurIPS 2025 papers under automated AI reviewers; every judgement in that result, from the improvement parsed out of result tables to the review scores, was made by a model, not a person.Improving its own scaffolding from failures · reviewed by AI
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A matched deployment where automating individual phases without end-to-end chaining yields the same throughput, or where swapping the base model changes outcomes more than changing the harness does.