llm-program-repair-workflows-built-self-correction-generated-knowledge-or-multi-agent-scaffolding CLM-0426For LLM program-repair workflows built on self-correction, generated-knowledge or multi-agent scaffolding, the gain over plain prompting is unstable across benchmarks and can turn negative once extrinsic defect information (error messages, triggering tests, bug comments) is withheld, while simply putting the error message plus the failing test into the prompt yields more correct repairs than the scaffolds do.
Capability: Fixing its own mistakes
Observed on
Three enhancement methods (SRepair, FixAgent, RefinedTRP) on Defects4J, HumanEval-Java and 54 type-related BugsInPy defects, with GPT-3.5-Turbo (and GPT-5.4-mini for the main compa.
Sources
- supportsTowards the Generalizability of Leveraging ChatGPT in APR via Self-enhancing: An Empirical StudyMeasured: repair counts with human-adjudicated semantic review across three benchmarks and two models, with and without extrinsic information. Absolute numbers are small on BugsInPy (7 to 10 of 54), single execution per defect at non-zero temperature, so run-to-run variance is unmeasured; benchmarks also differ in language and defect origin, so cross-benchmark differences are not isolated to enhancement quality.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.Stating false facts confidently · reviewed by AI
- In search-based automated program repair of Simulink/Stateflow models, replacing hand-crafted domain-specific mutation operators with an LLM that proposes label rewrites from static component context lowers repair coverage under a fixed wall-clock budget, because unconstrained generative edits widen the candidate space while each candidate costs an expensive simulation and the generator gets no behavioural feedback about why a patch failed.Generating and editing working code · reviewed by AI
- In zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.Stating false facts confidently · reviewed by AI
- For LLM multi-agent systems failing on long-horizon web tasks, re-running the whole task from scratch — plain, with self-reflection, or with a critic agent — repairs few failures (at most 6.90% within three attempts, with feedback variants below plain rerun), while reconstructing the recorded execution prefix and intervening once at a symptom-flagged trace node repairs 20.15% with a single attempt.Getting more from several models than from one · reviewed by AI
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A study showing the same enhancement workflows, with no extrinsic information supplied, deliver consistent positive gains across several benchmarks and exceed prompts that directly include error messages and failing tests. Drafted stance toward external-feedback-repair-works-only-with-real-grounding: supports -- Scaffolds stripped of external defect information gave zero or negative gains, while supplying real execution feedback raised repair counts.