Grading generated code by whether it actually passes held-out functional tests, rather than by surface similarity to a reference solution, is what makes code generation gradeable at scale — and it's how the field has graded it since the earliest large benchmarks.

Capability: Generating and editing working code · Reasoning

Observed on

2021, Codex/GPT-3 class. Coding agent.

Sources

supportsEvaluating Large Language Models Trained on Code
Introduced HumanEval and the functional-correctness grading methodology (execute against held-out tests) alongside the pass@k estimator, now the field standard.

Status: activeLast checked: 2026-09-04Evidence activity

Contest this claim

Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.

Related claims

Static, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.Generating and editing working code
When grading LLM-generated unit tests by pass rate and line coverage alone, a share of passing test suites never directly call the function under test — they reach it through wrappers, mocks, or unrelated public APIs — so pass rate overstates test quality, and this gap widens under agentic repository-access workflows and richer file-level context.Whether the measurement made the finding · unreviewed