code-generation-benchmarks-deciding-whether-benchmark-item-leaked-into CLM-0370For code generation benchmarks, deciding whether a benchmark item leaked into a model's training set is done better by a classifier over code-specific signals (test pass rate, CodeBLEU, edit distance, perplexity, CodeBERT embeddings of generated vs reference code) than by perplexity alone, because memorized items show abnormally high functional correctness and similarity to the reference solution, not just low perplexity.
Capability: Whether the measurement made the finding
Observed on
Black-box query access with token log-probabilities; shadow model fine-tuned on a disjoint benchmark subset; small code LLMs (1.3B-3B) as targets; leakage mostly simulated by fine-.
Sources
- Validated by: Eight benchmarks, four target models, against eight MIA baselines: average gains of 0.05-0.16 Precision, 0.06-0.11 Recall, 0.17-0.26 MCC, 0.09-0.13 AUC over the strongest baseline per model; recovered over 65% of 108 known leaked APPS samples in StarCoder-7B.. Measured on eight benchmarks and four small code models, but membership is mostly created by the authors' own LoRA fine-tuning, which likely leaves stronger memorization traces than incidental pretraining leakage; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI
- On natural-language-to-SQL synthesis over multi-table databases, a model's accuracy overstates its ability whenever the same database or the same query appears in both training and test: splitting by database rather than by example cut the best system's exact-match accuracy from 33.0% to 12.4%, and accuracy falls further as the target database has more foreign keys.Writing a reusable locator: regex, SQL, XPath · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.Whether the measurement made the finding · reviewed by AI
- For a domain-specific language essentially absent from pretraining corpora (FORM symbolic algebra), QLoRA fine-tuning an 8B model on a few thousand examples validated by executing them through the language's own interpreter beats frontier open-weight models of up to 756B parameters given syntax documentation in context, on execution rate and strict output match for the larger benchmarks, while general reasoning scores drop under 3 points.Generating and editing working code · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: An evaluation where adding functional-correctness and similarity features to a perplexity-only detector gives no improvement in MCC/AUC, or where the classifier's advantage disappears on real (not fine-tuning-simulated) leakage. Drafted stance toward static-code-benchmarks-saturate-need-continuous-refresh: supports -- The paper documents that widely used code benchmarks leak into training data and that leaked items yield inflated pass@1, which is the pressure behind continuous benchmark refresh.