arxiv-2403-07974 · paper

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica

Created: 2024 · Ingested: 2026-09-04

1000+ citations in the last 12 months · 2073 total · checked 2026-09-04

https://arxiv.org/abs/2403.07974

Continuously collects newly-published coding problems to stay contamination-free, evaluating 52 models; static fixed-problem benchmarks like HumanEval saturate and leak into training data over time, so an honest current read needs a benchmark that keeps adding problems.

In brief

Strong HumanEval scores from fine-tuned open code models do not transfer to fresh competition problems, and several models show sharp performance drops exactly at their training cutoff dates. LiveCodeBench collects 511 problems released May 2023-May 2024 from LeetCode, AtCoder and CodeForces, each tagged with a release date, and scores 18 base and 34 instruction-tuned models across code generation, self-repair, code execution, and test output prediction using Pass@1 over 10 samples.

DS-Base-33B falls from Pass@1 ~60 on May LeetCode problems to ~0 on September ones; GPT-4-O drops after its November cutoff; Codestral goes from 36.5 to 28.3 after Feb'24. HumanEval+ and LCB-Easy correlate only 0.72: DS-Ins-1.3B gets 59.8% on HumanEval+ but 26.3% on LCB-Easy. DS-Ins-33B trails GPT-4-Turbo by 4.3 points on HumanEval+ but 16.2 points (69%) on LCB generation, and by 96% and 134% on test output prediction and code execution. Scenario correlations exceed 0.88, so ranking is largely stable across tasks.

The contamination claim is inferred from time-segmented performance curves, not from inspecting training data; there is no controlled ablation separating contamination from problem-difficulty drift across months. Post-cutoff evaluation uses 349 problems, with 1-1.5% estimated variation from set sampling; scope is Python and competition-style problems only, with untuned prompts.

Treat HumanEval-topping open fine-tunes as unproven until checked on post-cutoff problems.

Written from the abstract by claude-opus-5 on 2026-09-05, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsStatic, fixed-problem coding benchmarks saturate and leak into training data over time, so getting an honest read on current code-generation ability requires a benchmark that continuously adds newly-published problems rather than reusing an older fixed set like HumanEval.