static-code-benchmarks-saturate-need-continuous-refreshmechanismsingle paper
Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.
Evidence for: Goodhart's law (holds)
Capability: Generating and editing working code · Reasoning
Observed on
2024, evaluating 52 models on 2023-2024 problems. Coding agent.
Sources
- LiveCodeBench continuously collects newly-published problems to stay contamination-free, evaluating 52 models across base and instruction-tuned variants.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- On repository-level issue-fixing tasks (SWE-bench Lite/Verified/Pro), spending effort upstream — expanding the issue text into a structured requirement spec from retrieved repo context, then testing and refining that spec via generated code and tests — raises resolve rate over agents that consume the raw issue description, and raises the share of patches that even apply cleanly by a larger margin than it raises correctness.Generating and editing working code · unreviewed
- In UI-to-code generation, where a rendered screenshot gives genuine external feedback, iterative self-critique still drifts below the initial draft over ten rounds, because a code edit changes the rendering non-locally and a fix for one region breaks regions that were already faithful; restricting each round to one typed, scoped repair target and carrying past targets forward as an avoid-list reduces that drift.Generating and editing working code · unreviewed
- When a model gets a real environment signal after acting — a test result, a tool error, a task-success indicator — reflecting on that signal in words and retrying substantially improves success rates on multi-step coding and decision-making tasks over a single attempt.Fixing its own mistakes
- Grading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.Generating and editing working code
- In agent pipelines that generate whole domain-specific ML codebases (medical imaging) from scratch, execution/assembly validation and domain knowledge bases fix different failures: removing runtime and assembly testing mainly costs autonomy (more human interventions to debug), while removing the curated domain knowledge base mainly costs output quality (syntactically fine but domain-inappropriate pipelines) with little change in intervention count.Generating and editing working code · unreviewed
Notes
Pairs with self-repair's SWE-bench claim (arxiv-2310-06770) for the repo-scale, real-issue-resolution side of code competence, versus isolated-problem generation here.