arxiv-2603-28052 SRC-0085 · paper

Meta-Harness: End-to-End Optimization of Model Harnesses

Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, Chelsea Finn

Created: 2026-03-30 · Ingested: 2026-09-14

The performance of large language model (LLM) systems depends not only on model weights, but also on their harness: the code that determines what information to store, retrieve, and present to the model. Yet harnesses are still designed largely by hand, and existing text optimizers are poorly matched to this setting because they compress feedback too aggressively. We introduce Meta-Harness, an outer-loop system that searches over harness code for LLM applications. It uses an agentic proposer that accesses the source code, scores, and execution traces of all prior candidates through a filesystem. On online text classification, Meta-Harness improves over a state-of-the-art context management system by 7.7 points while using 4x fewer context tokens. On retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across …

In brief

Giving an automated harness optimizer unrestricted filesystem access to raw execution traces of all prior candidates, rather than scalar scores or LLM summaries, is what makes harness search work. Meta-Harness runs a coding-agent proposer (Claude Code with Opus-4.6) that reads prior harness source code, scores, and traces via grep and cat, proposes a new single-file Python harness, evaluates it, and logs everything back; a typical run evaluates roughly 60 harnesses over 20 iterations.

On online text classification with GPT-OSS-120B across LawBench, Symptom2Disease and USPTO-50k, the discovered harness reached 48.6% average accuracy versus 40.9 for ACE and 40.0 for MCE, using 11.4K context tokens against 50.8K and 28.5K. It also held up on 9 unseen datasets (73.1% vs 70.2% for ACE). A single discovered retrieval harness improved 200 IMO-level problems by 4.7 points on average across 5 held-out models, and 1.3 points over BM25. On TerminalBench-2 it hit 76.4% with Opus 4.6 (vs 74.7 for Terminus-KIRA, 81.8 for ForgeCode) and 37.6% with Haiku 4.5 (vs 35.5 for Goose).

The interface ablation isolates the cause: scores-only gave 34.6 median accuracy, scores-plus-summary 34.9, full traces 50.0. Baselines were matched on evaluation budget (Best-of-N, OpenEvolve, TTT-Discover, GEPA), and Meta-Harness matched the best of them in 0.1x the evaluations.

Boundaries: 1 proposer agent, and on TerminalBench-2 search and final evaluation used the same 89 tasks, with overfitting checked only by inspection and regex audits.

If you hand-tune scaffolds, the trace-access result is the part to act on.

Written from the abstract by claude-opus-5 on 2026-09-15, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.