synthesis-2026-harness-engineering-playbook · postHarness Engineering — Agent = Model + Harness: The 6-Layer Production Playbook
Unattributed (independent synthesis)
Created: 2026-08-01 · Ingested: 2026-09-11
https://drive.google.com/file/d/1wZeuP0t4WOA2COMxcrU-Uv34IutM13UR/view(opens in a new tab)A nine-page playbook presenting production agent design as six layers around the model: guides (feedforward instructions), sensors (feedback checks), a bounded agentic loop, persistent memory, permissions and budgets, and observability. Its organising idea is the ratchet: every failure becomes a permanent fix in the harness rather than a patch to a prompt. It compiles harness-only performance gains from five secondhand sources and supplies a vocabulary — guides versus sensors, computational versus inferential sensors, trip wires, capability budgets, escalation packets — that maps unusually well onto this catalog's own machinery.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsIn an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
- supportsA correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
- supportsA large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
- contestsGiving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
- supportsThe agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.