arxiv-2604-21579 SRC-0352 · paper

A Metamorphic Testing Approach to Diagnosing Memorization in LLM-Based Program Repair

Created: 2026-04-23 · Ingested: 2026-09-19

LLM-based automated program repair (APR) techniques have shown promising results in reducing debugging costs. However, prior results can be affected by data leakage: large language models (LLMs) may memorize bug fixes when evaluation benchmarks overlap with their pretraining data, leading to inflated performance estimates. In this paper, we investigate whether we can better reveal data leakage by combining metamorphic testing (MT) with negative log-likelihood (NLL), which has been used in prior work as a proxy for memorization. We construct variant benchmarks by applying semantics-preserving t

In brief

Rewriting buggy Java code in semantics-preserving ways drops LLM repair success rates, and the drop is largest exactly on the code the model finds most familiar, which points at memorization rather than reasoning.

CodeCocoon applies 9 natural transformations (identifier renaming via synonyms, for-to-while, nested else-if, operand swaps, unary expansion) to Defects4J (854 bugs) and GitBug-Java (199 bugs, 44 excluded). 7 models (Claude-3.7-Sonnet, GPT-4o, GPT-4o-mini, Llama 3.1 8B, Gemma 2 27B, Mistral 7B v0.3, StarCoder2 7B) were prompted 10 times per bug with Chain-of-Thought, function-level fault localization, and test-suite validation.

On Defects4J all 7 models dropped significantly (p < 0.001): -4.01% for GPT-4o up to -15.98% for Llama-3.1, with worst cases of -100.00% (Claude-3.7 on Jsoup-15). Easy bugs suffered most. GitBug-Java showed no consistent drop (Claude-3.7 +0.27%, p = 0.819), though individual bugs fell from 100% to 30%. For the 4 open models, Spearman correlation between NLL and success-rate drop was r = 0.303 (Gemma-2), 0.218 (Mistral), 0.165 (Llama), 0.401 for StarCoder2 (p = 0.0798).

The drop is measured against paired original bugs; the leakage attribution is correlational, uses NLL values partly borrowed from prior work, and cannot cover proprietary models. Patch validity is test-suite pass, not manual correctness. Java and APR only.

If you report Defects4J numbers, report transformed variants too.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsEvery state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.