fixes-in-the-harness-compound-fixes-in-the-prompt-do-not CLM-0055mechanismmechanism reasoningReviewed by AI
A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
Evidence for: Hashimoto's rule (holds), Practice makes perfect (breaks), Simon's law (holds)
Capability: Following an unfamiliar procedure · Agentic, Coding agent
Applies when
- Effect on the capability:
- helps
- Cost:
- low
- Fails when:
- The correction is re-applied as a prompt each session instead of being encoded as a guide rule, sensor or permission; it is then gone when the session ends.
Sources
- The ratchet principle, sections II.B and IX. The six-step loop — reproduce, classify, choose the strongest layer, implement, verify, regression-test — and the observation that a mature harness adds one rule a week where a new one adds five a day. Argued and illustrated from the Ghostty AGENTS.md history; not measured against a control.
- The primary. Hashimoto's rule for the Ghostty AGENTS.md — "anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again" — and the file growing one line per failure over months. Practitioner account, no control, but first-hand rather than secondhand.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.Following an unfamiliar procedure · contested · reviewed by AI
- Guides, sensors and checkpoints pay for themselves only when the same workflow runs repeatedly, mistakes have real consequences, and no person is watching each run; single-turn questions, brainstorming and one-off tasks gain nothing from them and pay their cost in tokens and setup — so the decision to build a harness is a decision about the shape of the work, not about the model.Improving its own scaffolding from failures · reviewed by AI
- In file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.Completing tasks that take many steps or many hours · reviewed by AI
- When an agent system evolves its own harness from failure traces, batching failures across tasks before editing buys efficiency but not better failure attribution: pooled across three agent benchmarks, the share of harness edits that merely accommodate the current model rose from 60.0% under per-failure serial editing to 75.6% under cross-task aggregation, and only fell to 45.5% when a separate analyst/critic/engineer refinement stage attributed each recurring pattern before the edit was written.Improving its own scaffolding from failures · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
Notes
Read alongside repository-context-files-do-not-raise-task-success-on-benchmark-coding: the measured evidence is that a context file per se does not lift task success, so whatever compounds here is the specific failure-derived rule, not the presence of a file. This is the claim closest to the project's own ambition of improving itself. docs/ambitions.md now uses this document's vocabulary to describe what that loop would be.