arxiv-2608-28411 · paper

LongPIBench: A Long-Context Benchmark for Prompt Injection

Created: 2026-08-28 · Ingested: 2026-09-09

Prompt injection attacks pose a serious security risk to large language models in real-world applications. However, existing prompt injection benchmarks primarily focus on short-context inputs, leaving the attacks and defenses in long-context settings largely unexplored. This gap leads to a substantial overestimation of the effectiveness of current defenses. In this paper, we bridge the gap by introducing LongPIBench, a long-context benchmark for prompt injection covering 4 realistic application scenarios: paper peer review, resume screening, code review, and email summary. For each scenario,

In brief

Prompt injection defenses that report near-zero attack success on short-context benchmarks fail once the injected text sits inside a document of thousands to tens of thousands of tokens. LongPIBench covers 4 document-centric scenarios: paper peer review, resume screening, code review, email summary, each with a synthetic set of 100 GPT-5-generated instances and a real-world set. It evaluates 6 heuristic attacks, 2 GCG variants, 9 detectors and 6 prevention defenses across 8 LLMs.

With no defense, the Combined attack reaches ASR 1.00 on synthetic paper review, and MetaSecAlign 8B, which reports near-0% on prior benchmarks, also hits 1.00 there. A new Authority spoof attack, impersonating organizers or hiring managers, is the strongest heuristic: above 0.75 on paper review and resume screening for GPT-4o and GPT-4.1, versus often below 0.6 for other heuristics, and 0.61 on GPT-5.6 against a 0.08 no-attack baseline. Detectors collapse to extremes: DataSentinel and DistilBert reach FPR near 1.00, while EVD and Deberta show FNR near 1.00. Aggregated, defenses give 0.98 ASR on this benchmark versus 0.67, 0.61, 0.28 undefended on OPI, InjecAgent, AgentDojo.

Measured, with real-world replication and ablations on injection position, format and goal, but the mechanism is not isolated: dilution versus positional effects is left open, MetaSecAlign results are 8B only, and 100 instances give a 95% Wilson half-width near 9.6 percentage points.

Agentic and multi-step pipelines were not tested. If you rely on published defense numbers, re-measure at your real document length.

Written from the abstract by claude-opus-5 on 2026-09-09, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

supportsPrompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.