AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents

Whole projects generated by coding agents run in a clean environment only about two-thirds of the time, and most of the failures are ordinary code bugs, not missing dependency declarations. That cuts against the framing that dependency specification is the main barrier.

Claude Code (Opus 4.1), OpenAI Codex (0.52.0), and Gemini Code (2.5 Pro) each answered the same 100 prompts (40 Python, 35 JavaScript, 25 Java) that explicitly demanded complete dependency files and 100% reproducibility. The 300 projects were run on reset AWS EC2 instances holding exactly 91 baseline packages, with SciUnit, npm list, and mvn dependency:tree capturing runtime imports.

205 of 300 (68.3%) executed out-of-the-box. By language: Python 89.2%, JavaScript 61.9%, Java 44.0%. By agent: Claude 73.0%, Gemini 72.0%, Codex 60.0%, with Gemini at 100% on Python but 28.0% on Java and Claude at 80.0% on Java. Of the 95 failures, 50 (52.6%) were code bugs and only 10 (10.5%) missing dependencies. Average runtime multiplier 13.5x; one ML project declared 3 packages and loaded 52.

Everything is measured on a controlled clean environment, which isolates the dependency question well. There is no repeated sampling per prompt, no retry or agentic self-debug condition, 1 version of each agent, and the 15-minute debugging figure is the authors' own manual timing.

Use it to argue that pass@1 correctness benchmarks miss environment and structural failures entirely.