prompt-injection-defenses-report-near-zero-attack-success-short-context-benchmarksPrompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following instructions hidden in data
Observed on
Single-pass document-centric tasks with full document in one inference call; open 3-8B models plus GPT-4o/GPT-4.1; synthetic and real-world datasets, 100 instances per synthetic su.
Sources
- Measured: eight LLMs, six heuristic and two optimization attacks, nine detectors and six prevention defenses, four scenarios with synthetic and real-world splits. Baseline is the same defenses on OPI, InjecAgent, AgentDojo, where their reported attack success is near zero. Design shows the effect but the authors state it does not isolate the mechanism (dilution vs position). Detection defenses show extreme false-positive/false-negative tradeoffs. Single group, one benchmark.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · unreviewed
- For computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.Following instructions hidden in data · unreviewed
- For open-weight agents where attention matrices are accessible, detecting injection by asking which context span actually drove the tool call — aggregating attention over the decision tokens and localizing the guiding span, then checking whether that span's provider has authority for the action — catches unauthorized tool invocations at higher true-positive and lower false-positive rates than static scanners that look for malicious-looking text, because benign-looking injected text is only harmful when it is what the model actually attends to.Following instructions hidden in data · unreviewed
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · unreviewed
- Separating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.Following instructions hidden in data
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Evaluating the same defenses on long-document injection with matched attacks and finding attack success stays near the short-context level, or showing the long-context gap disappears once no-attack baseline rates and scoring criteria are matched to the short-context benchmarks. Drafted stance toward separating-prompt-data-reserved-delimiters-fine-tuning-structure-blocks: contests -- A fine-tuned separation defense (MetaSecAlign 8B) that reports near-zero attack success on short-context benchmarks reached attack success of 1.00 on the long-context paper review and resume screening datasets.