fixes-in-the-harness-compound-fixes-in-the-prompt-do-notA correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following an unfamiliar procedure · Agentic, Coding agent
Sources
- The ratchet principle, sections II.B and IX. The six-step loop — reproduce, classify, choose the strongest layer, implement, verify, regression-test — and the observation that a mature harness adds one rule a week where a new one adds five a day. Argued and illustrated from the Ghostty AGENTS.md history; not measured against a control.
- The primary. Hashimoto's rule for the Ghostty AGENTS.md — "anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again" — and the file growing one line per failure over months. Practitioner account, no control, but first-hand rather than secondhand.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.Following an unfamiliar procedure · contested · unreviewed
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · unreviewed
- Handing an agent context-editing tools (delete, summarize, note, fold) by prompt alone makes mid-size models worse than leaving their context untouched — Qwen3-8B, Qwen3-14B and Gemma4-E4B all scored lower on long-context QA with the tools than without them; only after fine-tuning on trajectories that use the tools does the toolset become a gain.Keeping its own context clean · unreviewed
Notes
Read alongside repository-context-files-do-not-raise-task-success-on-benchmark-coding: the measured evidence is that a context file per se does not lift task success, so whatever compounds here is the specific failure-derived rule, not the presence of a file. This is the claim closest to the project's own ambition of improving itself. docs/ambitions.md now uses this document's vocabulary to describe what that loop would be.