arxiv-2606-09498 · paper

Self-Harness: Harnesses That Improve Themselves

Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, Shuyue Hu

Created: 2026-06-08 · Ingested: 2026-09-11

The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which ident

In brief

An agent can rewrite the scaffolding it runs inside, using only its own base model, and get large pass-rate gains without any stronger external model or human engineer. The improvement comes from bounded, evidence-grounded edits to prompts, tools, and runtime policy rather than weight changes.

Self-Harness runs a 3-stage loop: cluster failed execution traces into verifier-grounded failure signatures, propose K minimal distinct harness edits with the same fixed model, then promote only edits that improve 1 split without degrading the other. Tested on Terminal-Bench-2.0 (64-case subset), SWE-bench Verified (100 cases, 67 held-in / 33 held-out), and AppWorld (180 examples), with MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5, starting from a minimal DeepAgent harness.

All 9 model-benchmark pairs improve on both splits. Largest relative gain 132% (Qwen3.5-35B-A3B, AppWorld); largest absolute gain 40.6 percentage points for GLM-5 on AppWorld, 44.4% to 85.0%. Weakest gain: GLM-5 on SWE-bench Verified, 52.0% to 55.5%. Weak models gain most.

Model, tools, budget, evaluator, and split assignments are held fixed, so the harness is isolated as the cause, and the held-out split was never shown to the proposer. But the promotion gate itself uses held-out pass counts, so held-out is not clean generalization; scores come from 2 repeated attempts; no comparison against Meta-Harness or a human-engineered harness; no matched compute baseline; benchmarks are fixed and edits may be benchmark-specific.

If you tune harnesses by hand per model, this suggests the loop can be automated, at the cost of many full benchmark re-evaluations.

Written from the abstract by claude-opus-5 on 2026-09-11, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsWith weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.