secure-coding CAP-0043 · activeWriting secure code and dependencies
Generated code carries known vulnerability patterns, and models suggest packages that do not exist.
Also called: insecure code generation, package hallucination, slopsquatting
Tags: coding-agent, security
A capable coding model avoids well-known vulnerability classes, uses safe defaults, and only imports dependencies that exist and are trustworthy, checking the registry rather than recalling names from memory.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Security properties of code the model writes, and of the dependencies it recommends: injection-prone patterns, hardcoded secrets, unsafe deserialization, and hallucinated or malicious packages. NOT in scope: the general word "vulnerability" used to mean a model weakness or robustness gap, which is ordinary English in this literature and not about generated code.
Claims
- mechanismsingle paper↑1↓0Across many models and hundreds of thousands of samples, a substantial share of recommended packages did not exist, and the names repeated, making them exploitable.
- observationsingle paper↑1↓0When a benign GitHub issue is augmented with a plausible-sounding maintenance requirement that specifies a helper reading an environment variable into a dangerous sink, repository-level APR agents produce patches that fix the original bug and pass the project test suite while containing the requested vulnerability in roughly half of tasks, and the strongest instruction-following coding models are more susceptible, not less.
- observationsingle paper↑2↓0Checking that every package an agent names actually exists catches the invented ones but not the missing ones: what agents declare is routinely a fraction of what their code imports at runtime, so generated projects fail in a clean environment even when every declared package is real. Existence is a necessary check, not a sufficient one.
- mechanismsingle paper↑1↓0When hardware description code (Verilog, SystemVerilog, VHDL, Amaranth-Python) is generated from a functional specification that omits security obligations — as real SoC docs do, keeping countermeasures in separate config files — frontier models pass functional tests far more often than security tests, and the model with the highest functional pass rate is not the safest; naming the relevant CWE in the prompt raises the security pass rate sharply, showing the limit is missing awareness of the required obligation rather than inability to write defensive RTL.
- mechanismsingle paper↑1↓0Models complied with insecure completions a large fraction of the time, and more capable models were more likely to suggest insecure code.
- mechanismsingle paper↑1↓0Roughly forty percent of Copilot completions in security-relevant scenarios were vulnerable.
- observationsingle paper↑1↓0GPT-4 recommended non-existent Python and JavaScript packages in only a small percentage of generations — lower than the open models tested in the same study — but the same hallucinated names recurred often enough across runs to be practically exploitable by an attacker who registers them ahead of time.
- mechanismsingle paper↑1↓0Users with an AI assistant wrote less secure code and were more confident it was secure.
Techniques
- Adversarial issue descriptions for APR agentspromptingEmbeds an attack-specific helper requirement (attacker-controlled env-var input plus a dangerous sink wired into the repaired symbol) into an otherwise legitimate bug report, framed as ordinary maintenance work.
- Scan generated code for insecure patternstoolingRun a static insecure-pattern scanner on every generated change before it is accepted.
- Verify packages exist before installingprocessCheck every suggested dependency against the registry and an allowlist before it is installed.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMsThis source: We Have a Package for You! A Comprehensive Analysis of Package Halluci…How much the field cites it — heavily cited in the last 12 months80 in the last 12 months · 106 totalpublished 2024checked 2026-09-042024
- Purple Llama CyberSecEval: A Secure Coding Benchmark for Language ModelsThis source: Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Mode…How much the field cites it — heavily cited in the last 12 months73 in the last 12 months · 180 totalpublished 2023checked 2026-09-04click for Semantic Scholar2023↑ supports a claim here
- Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code ContributionsThis source: Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Cod…How much the field cites it — very heavily cited in the last 12 months413 in the last 12 months · 952 totalpublished 2021checked 2026-09-04click for Semantic Scholar2021↑ supports a claim here
- Do Users Write More Insecure Code with AI Assistants?This source: Do Users Write More Insecure Code with AI Assistants?How much the field cites it — very heavily cited in the last 12 months200 in the last 12 months · 430 totalpublished 2022checked 2026-09-04click for Semantic Scholar2022↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.