prompt-injection-defenses-report-near-zero-attack-success-short-context-benchmarks CLM-0122Prompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.
Evidence for: Postel's law (breaks)
Capability: Following instructions hidden in data
Observed on
Single-pass document-centric tasks with full document in one inference call; open 3-8B models plus GPT-4o/GPT-4.1; synthetic and real-world datasets, 100 instances per synthetic su.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- access to train the weights
- Fails when:
- The injected instruction sits inside a document of thousands to tens of thousands of tokens. Separation defences and detect-localise-remove pipelines then let a large share of injections through.
Sources
- Measured: eight LLMs, six heuristic and two optimization attacks, nine detectors and six prevention defenses, four scenarios with synthetic and real-world splits. Baseline is the same defenses on OPI, InjecAgent, AgentDojo, where their reported attack success is near zero. Design shows the effect but the authors state it does not isolate the mechanism (dilution vs position). Detection defenses show extreme false-positive/false-negative tradeoffs. Single group, one benchmark.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · reviewed by AI
- For computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.Following instructions hidden in data · reviewed by AI
- As long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.Following instructions hidden in data · reviewed by AI
- For tool-using agents trained against adaptive indirect prompt injection, attack success measured against a single frozen adaptive attacker understates vulnerability compared with taking the per-case success union over a population of retained attacker checkpoints (0.25% vs 4.97% on the same 1,187 adaptive cases for the trained defender).Following instructions hidden in data · reviewed by AI
- For decoder-only 360M-8B instruct models, adding application-supplied provenance labels as an architectural signal — origin embeddings plus a learned ring-indexed attention bias that suppresses assistant and user queries attending to untrusted-source keys — cuts indirect-injection attack success on unseen task domains and injection styles further than the same two-stage fine-tuning without the architectural components, even though both are equal in-distribution.Following instructions hidden in data · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Evaluating the same defenses on long-document injection with matched attacks and finding attack success stays near the short-context level, or showing the long-context gap disappears once no-attack baseline rates and scoring criteria are matched to the short-context benchmarks. Drafted stance toward separating-prompt-data-reserved-delimiters-fine-tuning-structure-blocks: contests -- A fine-tuned separation defense (MetaSecAlign 8B) that reports near-zero attack success on short-context benchmarks reached attack success of 1.00 on the long-context paper review and resume screening datasets.