static-code-benchmarks-saturate-need-continuous-refresh CLM-0158mechanismsingle paperReviewed by AI and Russ Miller
Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
Evidence for: Goodhart's law (holds)
Capability: Generating and editing working code · Reasoning
Observed on
2024, evaluating 52 models on 2023-2024 problems. Coding agent.
Sources
- LiveCodeBench continuously collects newly-published problems to stay contamination-free, evaluating 52 models across base and instruction-tuned variants.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Every state-of-the-art model tested repairs fewer Defects4J and GitBug-Java bugs after semantics-preserving transformation of the buggy code, from -4% for GPT-4o to -16% for Llama-3.1, and the size of each drop correlates with how likely the model was to have memorized the original (negative log-likelihood), so part of published repair success on these benchmarks is recall of seen fixes rather than debugging.Finding and fixing a fault in existing code · reviewed by AI
- On real C/C++ faults with reproducing tests, 24 current models repair only about 11% of the 248 bugs and about 6% of the 102 vulnerabilities, far below their Java and Python repair rates on comparable benchmarks, and fine-tuning on the benchmark's commit corpus raised Pass@k in 21 of 28 comparisons by an average 85% relative, so the gap is partly a training-data gap.Finding and fixing a fault in existing code · reviewed by AI
- Fine-tuned program-repair models that score well on HumanEval-Java-Bug lose more than half their Pass@10 under semantics-preserving rewrites of the buggy code, with local-variable renaming alone cutting performance by 50-57%, so much of measured repair skill is tied to the exact surface form of the bug.Finding and fixing a fault in existing code · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · reviewed by AI
Notes
Pairs with self-repair's SWE-bench claim (arxiv-2310-06770) for the repo-scale, real-issue-resolution side of code competence, versus isolated-problem generation here.