The ratchet
The site is named for a claim: a fix placed in structure applies on every later run, while a fix given in conversation dies with the session. These are the sources about systems that try to build that ratchet — a loop that reads its own failures and rewrites the harness it runs inside — together with the claims drawn from each. It is the catalog looking at its own subject: a system like this could read these claims, apply them, and file what broke.
Four tests separate a ratchet from a fitted harness, and they are what to read each paper against: freeze the harness before evaluation; forbid the improver from editing its own tests or notes; require that failed paths stay observable; promote only changes that survive a new session or a disjoint task set. The recurring finding across these sources is that the acceptance gate, not the improvement engine, is what makes a loop ratchet rather than drift.
The hypothesis we are trying to prove
argued, not yet measuredModels trained on a literature that hides its failures inherit its optimism: asked what works, they overrate it, and they miss the conditions under which it stops working.
The argument is Sungmin Lee’s, in LLMs Have Made Failure Worth Publishing (2026), filed here as a claim reviewed by AI. The paper proposes experiments to test it and does not run them. If it holds, it is the strongest reason for this catalog to exist: an agent asking a model what works would be asking a reader of a tilted sample, and needs the failures written down somewhere else.
The experiment, and what would count
- For techniques the catalog has evidence on — above all the contested ones, and those whose claims record when they fail — ask models from several families, without the catalog, whether the technique helps and when it does not.
- Score each answer against the filed evidence: does it call a contested technique settled, does it name the failure condition the evidence records, does it claim an effect where the evidence records none or harm.
- Prediction, written before the run: models rate techniques more favourably than the evidence, and omit failure conditions far more often than they invent them.
- What would refute it: answers that match the evidence, or that err as often toward pessimism as toward optimism.
Two limits, stated in advance. The catalog draws on the same published literature, so it is a less tilted reference, not ground truth. And scoring the answers is itself a judgment: it needs a model from a different family than the ones tested, and a person checking a sample.
Why it files what failed
The published record is tilted toward what worked. In psychology, 96% of standard papers reported a positive first result against 44% of Registered Reports, which journals accept before the results are known; across all disciplines the share of papers reporting support for their hypothesis rose from 70% in 1990 to 86% by 2007 (the claim and its two sources). Machine learning has its own version: beating a benchmark is the publishable result, so methods that did not work go unreported and the same dead ends get paid for again (Karl et al., 2024). In their words, registering what does not work is no less important than registering what works.
Language models make this costlier. They are trained on the tilted record, so a model asked what works is a reader of a biased sample, and models are now asked exactly that (Lee, 2026). An agent trying to improve itself needs the boundary conditions most: when a technique stops helping, and what the evidence against it says.
So the catalog is built around the missing half. Every efficacy claim states the condition under which it fails; a claim with evidence on both sides is filed as contested, not averaged; heuristics are tested for where they break; and a technique with nothing measured is listed as a research brief, not omitted. The catalog is honest about its own limit: its sources are mostly published papers, so it inherits the same tilt. The counterweights are failures filed from use — a person’s own observations, and later, failures reported back by agents.
What exists today is the bank: the shared record of what works, what breaks, and under which conditions. The ratchet is the loop that would read it when something fails and file back what it learns. That loop is the open experiment, and the sources below are about systems that try to build one.
Techniques that serve the loop
Of 29 flagged techniques, these are grouped by the job they do for a loop. The rest of the catalog is object-level: it improves the work, not the loop that improves the work. A fix at the object level pays once; a better gate reduces error that would otherwise accumulate over every later round, which is why the gate keeps coming out load-bearing above.
gate 20
Decides what is kept, so the loop cannot slip back: acceptance criteria, held-out evaluation, independent verification, and the action gates that prevent unrecoverable loss.
- Actor-judge-editor RL on solver feedback for NL-to-PDDL ·
- Bound what the agent may do, not just what it may not
- Check long answers fact by fact
- Conformal abstention from self-consistency ·
- Critic-gated refinement stages with a simulated rebuttal ·
- Deterministic checks before LLM judges
- Gate irreversible actions outside the model
- Gated, attributed, typed edits to an agent-queried ontology ·
- Geometric fragility score from the safety subspace ·
- Judge both orders and reconcile
- Probe-derived value signal as a free process reward ·
- Promote a fix only against targeted, protected and negative- ·
- Scan generated code for insecure patterns
- Score a critique by the repair it enables ·
- Score recovery from the model's own real failure states ·
- Separate the verifier from the producer
- Split text-to-SQL evaluation by database, not by question ·
- Taxonomy-driven hallucination injection for review verificat ·
- Tool-feedback autoformalization with compiler and judge pane ·
- Verify packages exist before installing
loop 6
Extracts the signal a loop runs on: detecting, localizing or attributing a failure, or proposing the fix.
persistence 3
Puts the fix where later runs inherit it without anyone remembering.
Papers 7
- FINSKILLOPS: A Self-Evolving Multi-Agent System for SEC Filing QA2026-09-17 · paper
- ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI2026-09-17 · paper
- EvoOntology: A Self-Evolving Ontology Layer for Data Agents2026-09-14 · paper
- With weights frozen and no stronger model in the loop, an agent can raise its own pass rate by rewriting the scaffolding it runs inside — clustering its failures, proposing minimal edits to prompts, tools and policy, and keeping only edits that improve one split without degrading another: all 9 of 9 model-by-benchmark pairs improved, by up to 40.6 percentage points (GLM-5 on AppWorld, 44.4% to 85.0%), with weak models gaining most; but the promotion gate reads the held-out split, so the reported held-out gain is not clean generalisation, and there is no matched-compute baseline.
- For agents working over heterogeneous data, exposing a schema/content/tool ontology as an MCP server the agent queries, and refining it by a loop of diagnose, attribute-to-level, typed patch, and a gate that accepts an edit only if a backbone-conditional paired evaluation shows a margin, raised trajectory accuracy on DDR-Bench by an average 17.8 points across six backbones and text-to-SQL EX on BIRD by about 7 points; injecting the same semantic content as a static prompt did not help consistently and cost 15 points on one backbone, and in ablation removing the acceptance gate was the largest loss (-11.2), then attribution (-6.3), diagnosis (-4.8), typed patches (-1.7).
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement2026-09-14 · paper
- Agentic ML Exploration (A-MLE) for Ads Ranking2026-09-08 · paper
- With the agent loop, skills and prompts held fixed and only the model swapped, models split into two clusters on operating a multi-step asynchronous ML workflow: Claude Sonnet 3.5-4.0, Gemini 2.5 and GPT-5 completed tasks at 93-100%, while GPT-4 scored 13% and others could not run the loop at all, hallucinating workflow ids or failing to wait for asynchronous jobs; on open-ended exploration the effect of a stressful, competitive prompt reversed by family, with Sonnet 4.0 producing the largest improvement of any configuration under stress and GPT-5 turning conservative and giving back most of its basic-prompt gains.
- In a production ads-ranking stack, an agent that chains hypothesis generation, experiment execution, result analysis and a shared knowledge store, with human checkpoints at stage boundaries, delivered several times the completed ML iterations per engineer-week of the manual baseline and a 2.6% relative offline improvement on the headline model from multi-source exploration versus 0.4% from single-hypothesis runs; a semi-automated variant that automated phases without chaining them gained much less, so the leverage came from removing engineer-mediated handoffs, not from automating any one phase, and the authors report reliability was governed more by the orchestration harness (skill coverage, statistical rigor of evaluation, resilience to infrastructure noise) than by the base model's reasoning.
- The Economics of Recursive Self-Improvement2026-07-13 · paper
- Reporting how much a lab's models are used in its own research — tokens, lines of code, inference compute, experiments per researcher — cannot show that the models are accelerating the research, because every one of those is an input. Establishing the loop needs an outcome variable over time: algorithmic efficiency as a function of capability, which is the edge that turns a pipeline into a feedback loop.
- On the best current calibration, AI is not yet accelerating its own development in a self-sustaining way: the modelled threshold is that a one-unit gain in model capability must buy at least 15% higher AI R&D productivity, and the back-of-envelope figure from reported engineer uplift since coding agents launched is about 9% — below it, but rising, so the gap is a current reading rather than a ceiling.
Practice and commentary 10
- Guides, sensors and checkpoints pay for themselves only when the same workflow runs repeatedly, mistakes have real consequences, and no person is watching each run; single-turn questions, brainstorming and one-off tasks gain nothing from them and pay their cost in tokens and setup — so the decision to build a harness is a decision about the shape of the work, not about the model.
- In an agent's verify-and-fix loop, deterministic checks — tests, linters, schema validators — should be exhausted before any model-based judge is added, because they are free, fast and give the same verdict every time, while an LLM-as-judge costs tokens per run, returns non-deterministic verdicts, and so belongs only on properties no rule can express, and even then as an advisory signal until measured against human review.
- A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.(contests)
- The agent that produced an artifact is a biased judge of it — it holds the context and the incentives that skew its assessment — so verification belongs with a deterministic sensor or a separate verifier that reports failures back rather than rewriting the output, and in a multi-agent system the verifier is the one component no agent may override.
- Giving a coding agent a raw shell is worse than giving it a designed interface: SWE-agent's constrained Agent-Computer Interface — a file viewer windowed to 100 lines, a search tool that truncates large output, and an editor that lints before a change persists — roughly doubled SWE-bench resolution over raw shell access with the same model, because verbose output and unmanaged state are where the model loses its place.
- A large share of an agent benchmark score is attributable to the harness around the model rather than the model itself: holding the model fixed and changing only guides, sensors, retries and state management is reported to move the same model from 30.9% to 74.6% on GAIA and 30th to 5th on Terminal Bench — so a score reported for "a model" on an agentic benchmark is a score for a model-plus-harness, and comparisons across labs with different harnesses are not comparisons of models.
- Harness engineering: leveraging Codex in an agent-first world2026-02-11 · vendor-doc
No claim drawn from it yet.
- Harness engineering for coding agent users2026 · post
- Improving Deep Agents with harness engineering2026 · vendor-doc
- My AI Adoption Journey – Mitchell Hashimoto2026 · post
- A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.
- Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.(contests)