zero-shot-llm-program-repair-defects4j-bugs-passing-developer-written-test CLM-0251In zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.
Capability: Stating false facts confidently
Observed on
Three frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot with extracted buggy functions and test context on 832 filtered Defects4J Java bugs; no agentic too.
Sources
- supportsBetter Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program RepairValidated by: Artifact accuracy was measured against instrumented execution (F1 for triggering-test identification and line coverage, validity of generated tests) and compared between Pass and Fail repairs; Pass repairs had higher artifact accuracy on average, but the distributions overlapped and the paper did not build or evaluate an actual patch-reliability classifier.. Measured: automatic execution-based evaluation of all 832 bugs plus open-coded manual annotation of 812 sampled repairs stratified across nine task-model groups. Plausible-patch rates ran 21.0-55.9%. [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.Stating false facts confidently · reviewed by AI
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
- For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.Fixing its own mistakes · reviewed by AI
- Fine-tuned program-repair models that score well on HumanEval-Java-Bug lose more than half their Pass@10 under semantics-preserving rewrites of the buggy code, with local-variable renaming alone cutting performance by 50-57%, so much of measured repair skill is tied to the exact surface form of the bug.Finding and fixing a fault in existing code · reviewed by AI
- For repair of production crash bugs in large Java enterprise systems, feeding an LLM the source of the top-3 suspicious files plus a representative stack trace localizes the developer-modified code in about 71% of responses and yields a semantically valid patch in about 52%, and adding an explicit ranking of suspicious files and methods helps only when the full stack trace is withheld (per-response repair 28% to 47% with stack messages only) while slightly lowering performance when the stack trace is already present.Finding and fixing a fault in existing code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A manual audit of test-passing LLM patches on Defects4J finding that nearly all implement the developer-intended repair semantics, with correct causal localization. Drafted stance toward function-level-algorithmic-tasks-translated-c-model-s-functional-test: supports -- Both find that passing functional tests carries little information about the deeper quality of model-written code, here about semantic correctness and fault localization in patches.