arxiv-2609-04909 SRC-0206 · paperBetter Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair
Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
Created: 2026-09-04 · Ingested: 2026-09-19
https://arxiv.org/abs/2609.04909(opens in a new tab)Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence.
In brief
Patches that pass a project's own test suite are often still not grounded in a correct understanding of the bug. Across 3 models, manual inspection found repair hallucinations in 72.7% of 812 sampled repairs, including some patches that pass all available tests.
The study runs GPT-5, DeepSeek-R1, and Claude Sonnet 4.5 zero-shot on 832 filtered Defects4J bugs, and makes intermediate reasoning observable through 3 tasks: identifying the bug-triggering test methods, predicting which lines execute under those tests (buggy and self-patched versions, checked with JaCoCo), and generating additional tests that must fail on the buggy code and pass on the developer fix.
Only 21.0%–55.9% of patches pass the developer test suite; the baseline prompt yields 21.0%–26.3%. Models exactly identify triggering tests for 23.0%–40.9% of bugs. Valid generated tests: 30.5% Claude, 34.6% DeepSeek, 48.4% GPT-5. Passing repairs have better intermediate artifacts (GPT-5 F1 0.607 vs 0.214 on trigger identification), but 45.7%–67.7% of tests tied to Pass repairs are still invalid. 39 of 44 inspected low-coverage cases (88.6%) involve branching control flow.
Patch outcomes are measured against execution ground truth, and a matched baseline repair setting is included, so the per-task differences are attributable to prompt formulation. The intermediate-artifact link is correlational only. Annotator agreement was weak for understanding hallucinations (κ = 0.49), the benchmark is Java Defects4J at function granularity, and no agentic or tool-using systems were tested.
If you score APR by test pass rate, treat that number as an upper bound on understanding.
Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsIn LLM-based program repair on Defects4J bugs with frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5) prompted zero-shot, passing the developer-written test suite does not mean the patch is grounded in the bug: manual inspection of 812 sampled repairs found ungrounded repairs in 72.7% of cases including test-passing patches, most often from incorrect causal localization (45.9%) rather than a wrong edit style, and the same models frequently misidentify the bug-triggering tests and generate additional tests that miss the triggering condition.
- supportsIn zero-shot LLM program repair on Defects4J bugs, passing the developer-written test suite does not indicate that the patch is grounded in a correct understanding of the bug: manual inspection of sampled repairs found ungrounded patches in about three quarters of cases including ones that passed all available tests, with incorrect causal localization the largest category.