fine-tuned-program-repair-models-score-well-humaneval-java-bug-lose-more-than CLM-0300observationsingle paperReviewed by AI
Fine-tuned program-repair models that score well on HumanEval-Java-Bug lose more than half their Pass@10 under semantics-preserving rewrites of the buggy code, with local-variable renaming alone cutting performance by 50-57%, so much of measured repair skill is tied to the exact surface form of the bug.
Capability: Finding and fixing a fault in existing code
Observed on
Five fine-tuned repair models on 1,450 transformed instances from eight transformations of HumanEval-Java-Bug, 2026; Java, function-level, Pass@10..
Sources
- Fine-tuned smaller models only, so it says more about specialised repair models than about frontier assistants; the transformation set is standard and the drop is large and consistent.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
- In LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.Stating false facts confidently · reviewed by AI
- On real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.Finding and fixing a fault in existing code · reviewed by AI
- In zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.Stating false facts confidently · reviewed by AI
- For repair of production crash bugs in large Java enterprise systems, feeding an LLM the source of the top-3 suspicious files plus a representative stack trace localizes the developer-modified code in about 71% of responses and yields a semantically valid patch in about 52%, and adding an explicit ranking of suspicious files and methods helps only when the full stack trace is withheld (per-response repair 28% to 47% with stack messages only) while slightly lowering performance when the stack trace is already present.Finding and fixing a fault in existing code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Frontier general models (not fine-tuned repair models) showing flat Pass@k across the same transformations.