secure-coding-package-hallucination-rate-gpt-4 CLM-0144observationsingle paperReviewed by AI and Russ Miller
GPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.
Capability: Writing secure code and dependencies
Observed on
GPT. 2024, GPT-4 class. Coding agent.
Model cards: GPT-4
Sources
- Across many models and hundreds of thousands of samples, GPT-4 had a lower but still nonzero package-hallucination rate, with repeated package names across runs.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.Writing secure code and dependencies
- GPT-4 could reliably name a well-known celebrity's parent, but was far less reliable naming the celebrity when given the parent — the same directional-recall asymmetry demonstrated in smaller fine-tuned models, showing up in a deployed model that wasn't fine-tuned for the test.Not generalizing "A is B" to "B is A"
- When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.Generating and editing working code · reviewed by AI