repository-context-files-do-not-raise-task-success-on-benchmark-coding CLM-0130Giving a coding agent a repository context file — AGENTS.md, CLAUDE.md — does not raise its success rate on benchmark coding tasks and costs about 20% more inference: across 4 agents, 2 benchmarks and 3 conditions, LLM-generated files hurt slightly in 5 of 8 settings while developer-written ones gained 2.4% (p=0.21), and agents obey the files — which is why they spend more — so the files do not carry success-relevant information rather than being ignored.
Evidence for: Hashimoto's rule (narrows)
Capability: Following an unfamiliar procedure · Coding agent, Context
Observed on
2026, Claude Code / Codex / Qwen Code on SWE-bench Lite and CTXbench.
Applies when
- Effect on the capability:
- no effect
- Cost:
- moderate
- Fails when:
- The file is LLM-generated, which hurt slightly in 5 of 8 settings. Developer-written files gained 2.4%, not significant, at about 20% more inference either way.
Sources
- Measured with significance tests and ablations over generating model, prompt, length and section. Cost up 20% and 23%, steps up 2.45 and 3.92. Time to first touch of a gold-patch file did not drop, so the overview does not help navigation. Python only; one sample per instance.
- The practitioner position the study is testing: a guide file that grows one rule per observed failure, over months, in one repository. No measurement, but a different outcome — recurrence of specific past mistakes — from the study's task success rate.
- Treats the guide file as the first harness layer and "organisational memory". Secondhand; relays Hashimoto and OpenAI.
Suspected axis of disagreement(a guess, not verified)
Probably the outcome variable, and possibly the file's origin. The study measures success on tasks the agent has never seen, in repositories where the file was written generically or generated by the agent; the practitioner claim is about a file that accumulates the exact failures one team's agent made in one codebase, measured by whether those failures recur. A file of non-obvious project deviations might do the second job while adding nothing to the first — the study's own developer-written condition trends that way (+2.4%, not significant). If that is right, the technique's value is in what goes in the file, not in having one. Not tested against each other.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A correction made in conversation fixes one exchange and is gone when the session ends; the same correction encoded as a guide rule, a sensor, or a permission fixes every future run — so an agent system improves over time only to the extent that failures are converted into harness structure rather than re-applied as prompts, and the rate of new guide rules per week falling is the sign the conversion is working.Following an unfamiliar procedure · reviewed by AI
- In file-based sandbox agent tasks with distractor artefacts, scoring only whether the task goal is met overstates reliability, because agents frequently complete the target while modifying unrelated files: across nine LLM agents, pass rate exceeds a success-plus-preservation metric by roughly 10-17 points, with the gap widening as model capability falls.Completing tasks that take many steps or many hours · reviewed by AI
- On SWE-bench Lite, asking the model to name suspicious files from the issue plus a repository tree finds the ground-truth file in about 79% of cases and embedding retrieval over the issue text in about 68%, and taking the union of the two raises file-level localization to about 82% because the two methods miss different files; a fixed three-step narrowing (files, then classes and functions, then edit lines) with no agent loop then supported the best open-source resolve rate at the time (32%).Finding what matters in a large codebase · reviewed by AI
- On repository-grounded LLM-infrastructure engineering tasks, frontier agents score far below ceiling (best 36.53% overall), and every one of eight models scores lower on long-horizon multi-file implementation than on single-file kernel completion, even though the full repository and test cases are visible.Completing tasks that take many steps or many hours · reviewed by AI
- Giving an agent a parsed code graph (files, classes, functions with import, call and inheritance edges) and tools to search and traverse it lets a fine-tuned 32B open model localize issues about as well as proprietary agents at roughly an eighth of the cost, reaching about 93% file-level accuracy, and better localization carried through to about 12 points more resolved issues at Pass@10 when plugged into a fixed repair pipeline.Finding what matters in a large codebase · reviewed by AI
Notes
Filed the day the guide-file technique was, because the first measured study of it cuts against the practitioner consensus it came from. That is the catalogue working as intended: the technique record says what a guide file is, and this claim says the evidence on whether it helps is split, and along which axis.