functional-correctness-testing-is-the-standard-for-grading-code CLM-0060mechanismsingle paperReviewed by AI and Russ Miller
Grading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.
Capability: Generating and editing working code · Reasoning
Observed on
2021, Codex/GPT-3 class. Coding agent.
Sources
- Introduced HumanEval and the functional-correctness grading methodology (execute against held-out tests) alongside the pass@k estimator, now the field standard.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For code generation benchmarks, deciding whether a benchmark item leaked into a model's training set is done better by a classifier over code-specific signals (test pass rate, CodeBLEU, edit distance, perplexity, CodeBERT embeddings of generated vs reference code) than by perplexity alone, because memorized items show abnormally high functional correctness and similarity to the reference solution, not just low perplexity.Whether the measurement made the finding · reviewed by AI
- Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.Generating and editing working code
- On function-level algorithmic tasks translated to C#, a model's functional test pass rate carries almost no information about the structural quality or runtime cost of the code it writes: across 340 solutions from four frontier models, correctness correlated near zero with a composite quality score (r = 0.075) and model rankings by correctness reversed against rankings by quality, while the structurally simplest generator was also the most memory-hungry.Whether the measurement made the finding · reviewed by AI
- When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · reviewed by AI
- For a 7B code model fine-tuned on competitive-programming tasks, training it to emit a fixed-schema requirement specification (goals, input/output constraints, examples, edge cases, dependencies) before the code raises Pass@1 more than training it to emit free-form reasoning or a stepwise implementation plan of similar length, with the largest margin on an out-of-distribution benchmark.Generating and editing working code · reviewed by AI