secure-coding · activeWriting secure code and dependencies
Generated code carries known vulnerability patterns, and models suggest packages that do not exist.
Also called: insecure code generation, package hallucination, slopsquatting
Tags: coding-agent, security
A capable coding model avoids well-known vulnerability classes, uses safe defaults, and only imports dependencies that exist and are trustworthy, checking the registry rather than recalling names from memory.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Security properties of code the model writes, and of the dependencies it recommends: injection-prone patterns, hardcoded secrets, unsafe deserialization, and hallucinated or malicious packages. NOT in scope: the general word "vulnerability" used to mean a model weakness or robustness gap, which is ordinary English in this literature and not about generated code.
Claims
- mechanismsingle paperAcross many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.
- observationsingle paperChecking that every package an agent names actually exists catches the invented ones but not the missing ones: what agents declare is routinely a fraction of what their code imports at runtime, so generated projects fail in a clean environment even when every declared package is real. Existence is a necessary check, not a sufficient one.
- mechanismsingle paperWhen hardware description code (Verilog, SystemVerilog, VHDL, Amaranth-Python) is generated from a functional specification that omits security obligations — as real SoC docs do, keeping countermeasures in separate config files — frontier models pass functional tests far more often than security tests, and the model with the highest functional pass rate is not the safest; naming the relevant CWE in the prompt raises the security pass rate sharply, showing the limit is missing awareness of the required obligation rather than inability to write defensive RTL.
- mechanismsingle paperModels complied with insecure completions a large fraction of the time, and more capable models were more likely to suggest insecure code.
- mechanismsingle paperRoughly forty percent of Copilot completions in security-relevant scenarios were vulnerable.
- observationsingle paperGPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.
- mechanismsingle paperUsers with an AI assistant wrote less secure code and were more confident it was secure.
Techniques
- Scan generated code for insecure patternstoolingRun a static insecure-pattern scanner on every generated change before it is accepted.
- Verify packages exist before installingprocessCheck every suggested dependency against the registry and an allowlist before it is installed.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.