SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

Behaviour-preserving repository migration exposes a failure mode that fixed test suites cannot see: a repository handed back untouched passes every behavioural check, so agents get full credit for doing nothing. SWE Refactor Bench scores migration completeness and correctness separately, and current agents rarely achieve both.

The benchmark is 20 whole-repository migrations of real open-source projects (SQLite, zlib, libsodium, GraphHopper) across language (7), framework (7), platform (3) and build toolchain (3) debt, with 6-30 hour budgets. Scoring runs 3 stages: a model-judged Migration Audit with veto, 130,118 fixed behavioural checks recorded from the original, and 6 independent coding agents given 1 hour each to produce an executable counterexample.

Across 520 runs from 8 frontier models and 26 model-effort configurations, 28 runs (5.4%) passed all 3 stages and 13 of 20 tasks got no accepted solution; best was claude-opus-5 at 47.0/100. 30 runs preserved behaviour by skipping the migration, 252 migrated and broke behaviour. Of 340 audited-pass runs, 58% reached 99% of checks but only 26% reached 100%; of 88 clean submissions, 60 were broken by a verifier.

Evidence is measured, with human agreement on Stage I (140/156, κ=0.795), a self-preference check (verifiers broke 33.9% of own-family vs 34.3% of other submissions), and a majority-vote ablation (305 vs 340 passes). Each configuration ran each task once, so run-to-run variance is unmeasured, and verifier strength sets the bar: dropping the 2 opus verifiers would accept 46 instead of 28.

If you cite behavioural test pass rates as evidence of refactoring ability, this argues you are measuring the wrong thing.