behaviour-preserving-whole-repository-tasks-such-stack-migrations-where-starting-repository CLM-0024For behaviour-preserving whole-repository tasks such as stack migrations, where the starting repository already passes its test suite, a fixed behavioural test suite cannot distinguish a completed migration from an untouched or shim-wrapped repository, so adding more tests does not close the gap and a non-behavioural audit with veto power is required.
Capability: Whether the measurement made the finding
Observed on
Tasks where the starting state is already green — behaviour-preserving refactors, migrations, ports — not bug-fix tasks that have a red-to-green signal..
Sources
- supportsSWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?Argued from task structure (the original repository passes any suite drawn from its own behaviour by construction) and measured on 520 runs across 8 models and 26 configurations on 20 migrations: 118 runs passed every one of 130,118 fixed checks, but 30 of those had not migrated at all and 60 more were broken by post-hoc agentic verifiers, leaving 28 (5.4%) accepted. Audit judge agreed with two human annotators on 140/156 runs, with disagreements skewed strict. Does not isolate how much of the audit's rejection power generalises beyond these 20 hand-built criteria sets.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.Writing secure code and dependencies · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- In a hierarchical multi-agent system where a controller gates acceptance of worker outputs on semantic entropy and semantic density computed over resampled completions, the intervention helps only where failures come from semantic instability (vague or contradictory outputs) and hurts where they come from evidence selection or reasoning-chain errors \u2014 gaining on ambiguity/hallucination-prone benchmarks while losing 10-14 points on multi-hop QA.Getting more from several models than from one · reviewed by AI
- When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · reviewed by AI
- Treating post-deployment improvement as scoped behavioural maintenance, where each recurring failure becomes a skill patch with a stated scope and injection point that is promoted only after passing its targeted cases, a protected regression suite and negative controls, raised correctness from 3.70 to 4.55 on an SEC-filing QA benchmark, and over twelve operational rounds promoted only 6 of 33 candidate skills while the monitoring non-correct rate fell from 20.0% to 12.5%; the ungated condition scored higher on each candidate's own targeted cases but admitted regressions on the protected set.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: high. Falsifier as drafted: Show a purely behavioural suite that rejects untouched or forwarding-shim submissions on behaviour-preserving migration tasks, or show that submissions passing all fixed checks essentially never fail an independent migration audit. Proposed technique, not catalogued: Three-stage migration scoring: migration audit, fixed differential tests, then adversarial agentic verifiers.