secure-coding-package-hallucination-rate-gpt-4observationsingle paper
GPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.
Capability: Writing secure code and dependencies
Observed on
GPT. 2024, GPT-4 class. Coding agent.
Sources
- Across many models and hundreds of thousands of samples, GPT-4 had a lower but still nonzero package-hallucination rate, with repeated package names across runs.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.Writing secure code and dependencies
- GPT-4 could reliably name a well-known celebrity's parent, but was far less reliable naming the celebrity when given the parent — the same directional-recall asymmetry demonstrated in smaller fine-tuned models, showing up in a deployed model that wasn't fine-tuned for the test.Not generalizing "A is B" to "B is A"
- When a code-generation prompt combines a prompt-enforced output format (JSON or XML), a persona, and urgency framing at once, pass@1 on function-level Python problems can fall well below the sum of each constraint's individual effect, even when each constraint alone is neutral or helpful — observed in the GPT-4o family and absent in the GPT-4.1 family and o3-mini.Generating and editing working code · unreviewed