Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.

Capability: Writing secure code and dependencies

Sources

supportsWe Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.

Status: activeLast checked: 2026-09-03Evidence activity

Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.

Related claims

GPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.Writing secure code and dependencies
Checking that every package an agent names actually exists catches the invented ones but not the missing ones: what agents declare is routinely a fraction of what their code imports at runtime, so generated projects fail in a clean environment even when every declared package is real. Existence is a necessary check, not a sufficient one.Writing secure code and dependencies
On code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.Stating false facts confidently · unreviewed