a-fixed-model-can-improve-its-own-harness-with-a-regression-gated-loop CLM-0007observationsingle paperReviewed by AI
With weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.
Evidence for: Boyd's law of iteration (narrows), Hashimoto's rule (holds), Practice makes perfect (narrows)
Capability: Fixing its own mistakes · Agentic, Coding agent
Observed on
2026, MiniMax M2.5 / Qwen3.5-35B-A3B / GLM-5 on Terminal-Bench 2.0, SWE-bench Verified, AppWorld.
Applies when
- Effect on the capability:
- helps
- Needs:
- a held-out evaluation set; an external signal (a test, a compiler, an environment outcome)
- Helps most:
- weaker models
- Cost:
- high
- Fails when:
- The promotion gate reads the held-out split, so the reported gain is not clean generalisation, and there is no matched-compute baseline to say the same evaluations spent otherwise would not have done as well.
Sources
- Model, tools, budget, evaluator and splits held fixed, so the harness is isolated as the cause. Largest relative gain 132%, weakest 52.0% to 55.5%. Two repeated attempts per score; no comparison to a human-engineered harness; edits may be benchmark-specific. The authors state the held-out caveat themselves.
- Different harness (an ontology layer for data agents), same gate: in ablation, accepting every candidate edit instead of gating on a paired backbone-conditional evaluation was the largest single loss (-11.2 trajectory accuracy), because unfiltered edits admitted regressions later rounds could not undo. Added 2026-09-21.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For long-video QA with a frozen VLM and fixed answering interface, an outer loop that proposes, executes and retains only development-accuracy-improving rewrites of the executable context-construction program can advance past a strong hand-crafted keyframe-selection harness, and the gain survives a matched cumulative visual-token control and direct reuse on other long-video benchmarks.Improving its own scaffolding from failures · reviewed by AI
- When an automated search over agent-harness mechanisms is gated on token efficiency with only a tolerance band on capability, the resulting full stack buys large token savings at a small but consistent score loss, and scores below the best single discovered mechanism — on EdgeBench the four-mechanism stack cut token traffic 44.7-49.0% and cost about a third while scoring 42.0 vs 44.8 for the base harness on GPT-5.6 Sol, whereas adding ObservationPack alone scored 47.2.Improving its own scaffolding from failures · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- The large reported gains from fine-tuning on tool-call trajectories come from small open models starting far behind — distilling a few hundred trajectories from a stronger model moves them a long way. Whether the same procedure helps a model that is already good at tool use is a separate question this evidence does not answer.Using the tools it is given
- When an agent harness is evolved from execution traces under a fixed interaction budget, verifying each candidate edit on tasks chosen for the behavior it targets and accepting it only when the gain is attributable to that behavior beats scoring every candidate on a shared fixed, random or coreset task split, and larger verification budgets do not compensate: fixed-split methods using up to 24x the rollouts fell below the unmodified harness in half of their harness-benchmark pairs.Improving its own scaffolding from failures · reviewed by AI
Notes
The closest existing implementation of this project's own first ambition, and its stated weakness is the one docs/ambitions.md names: a loop that grades its own edits on the split it will be scored on has closed the circle. The fix it lacks — a held-out set the loop never sees and cannot grow — is the rule already in force for the backtest here.