prompt-injection CAP-0038 · activeFollowing instructions hidden in data
Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user.
Also called: indirect prompt injection, jailbreak via content
Tags: autonomous-agent, coding-agent, customer-support, rag-qa, security
A capable model treats content it reads through tools as data. It follows only instructions from the user and the system, and it flags text that tries to redirect it rather than acting on it.
Claims
- observationsingle paper↑1↓0For computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.
- mechanismsingle paper↑1↓0For decoder-only 360M-8B instruct models, adding application-supplied provenance labels as an architectural signal — origin embeddings plus a learned ring-indexed attention bias that suppresses assistant and user queries attending to untrusted-source keys — cuts indirect-injection attack success on unseen task domains and injection styles further than the same two-stage fine-tuning without the architectural components, even though both are equal in-distribution.
- mechanismsingle paper↑1↓0Demonstrates that instructions planted in retrieved content can hijack real LLM-integrated applications for data theft and manipulation.
- observationsingle paper↑1↓0For image-borne injected instructions delivered to tool-using agents, completion rate understates exposure: agents act on the injected instruction about ten times more often than they complete the objective (12.8% attempted vs ~1% completed across 720 runs), and most failures stop at the planning step where the model read the payload and simply did not act on it rather than at any architectural barrier.
- observationsingle paper↑1↓0In an LLM-assisted SOC pipeline where injections are planted in log fields, a deterministic SIEM pre-filter (regex decoders plus length truncation) catches only the structurally obvious payloads while an LLM self-check input/output rail catches most semantic ones, so the two layers block disjoint attack classes and neither alone is sufficient.
- mechanismsingle paper↑1↓0For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.
- mechanismreplicated↑1↓0As long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.
- mechanismsingle paper↑1↓0For open-weight agents where attention matrices are accessible, detecting injection by asking which context span actually drove the tool call — aggregating attention over the decision tokens and localizing the guiding span, then checking whether that span's provider has authority for the action — catches unauthorized tool invocations at higher true-positive and lower false-positive rates than static scanners that look for malicious-looking text, because benign-looking injected text is only harmful when it is what the model actually attends to.
- observationsingle paper↑1↓0Prompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.
- mechanismsingle paper↑1↓0Separating prompt and data with reserved delimiters and fine-tuning on that structure blocks most injections at little cost to utility.
- mechanismsingle paper↑1↓0Simple goal-hijacking and prompt-leaking attacks succeed against production models with short adversarial strings.
- mechanismreplicated↑2↓0Separating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.
- mechanismsingle paper↑1↓0For tool-integrated LLM agents where the attack works by adding attacker-controlled tools, overwriting the agent's visible toolset with the saved user toolset immediately before planning drives attack success to near zero across direct and indirect injection, memory poisoning and backdoor attacks, but its effect on benign task success is model-dependent — raising it for 7-9B open models and lowering it for GPT-3.5/GPT-4.
- mechanismsingle paper↑1↓0For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.
- mechanismsingle paper↑1↓0For tool-using agents on long-trajectory benchmarks, an injection defense that audits each candidate action against a one-step-ahead predicted tool set holds attack success low while keeping task utility near the attack-free level, whereas defenses that fix a global plan or filter whole tool outputs lose utility because the predicted full trajectory almost never matches the executed one once environmental feedback arrives.
- mechanismsingle paper↑1↓0For tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.
- mechanismsingle paper↑1↓0For tool-using agents, a safety harness searched separately for each model-and-domain deployment (jointly evolving a natural-language policy and executable pre/post-tool-call code, screened by a fresh-context adversarial reviewer) reaches a better safety-utility trade-off than fixed expert-designed defenses ported unchanged across deployments, because domain semantics determine which safety relations and trajectory state must be checked while the victim model's failure behavior determines whether a check should be semantic or deterministic and where in execution it must run.
- mechanismsingle paper↑1↓0For tool-using agents trained against adaptive indirect prompt injection, attack success measured against a single frozen adaptive attacker understates vulnerability compared with taking the per-case success union over a population of retained attacker checkpoints (0.25% vs 4.97% on the same 1,187 adaptive cases for the trained defender).
- mechanismsingle paper↑1↓0Training models to rank system, user, and tool instructions by privilege improves robustness to injections in tool outputs.
Techniques
- Bound what the agent may do, not just what it may notprocessAn explicit allow/ask/deny policy plus rate, cost and time ceilings, enforced by the harness rather than requested of the model.
- Deterministic SIEM pre-filter before semantic guardrailsprocessSanitize and length-bound log payloads with regex decoders and atomic rules at SIEM ingestion, then pass the structured alert through LLM self-check input and output rails before and after the analyst model.
- Make source provenance an architectural attention biasarchitectureTag each input token with a discrete origin ring, add a learned origin embedding scaled by a learnable gate, and add a learned ring-by-ring bias to attention logits that suppresses attention from assistant/user queries to untrusted-source k
- Report attempted and recognized rates, not just completionprocessInstrument the agent pipeline to record the furthest stage an injected payload reaches and use a judge panel to label whether the agent attempted the attack and whether it explicitly recognized the injection, alongside the deterministic com
- Restore the trusted toolset just before planningtoolingSave the user's original tool list and overwrite whatever toolset the agent can see with it immediately before the planning step, so plans can only reference user-approved tools.
- Separate instructions from datatrainingMark the boundary between trusted instructions and untrusted content, and train or prompt the model to honor it.
Sources
Every paper, post or observation behind this capability. Ones cited by a claim show the claim and its stance; ones filed directly are about the capability but have no claim drafted from them yet. Open one to add support or contest it.
- ↑ supports a claim here
- Origin Is All You Need: Provenance-Aware Transformers for Structural Trust-Boundary Separation2026-09-17↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here
- ↑ supports a claim here
- about this capability · no claim drafted yet
- ↑ supports a claim here
- An Experimental Evaluation of Multimodal Prompt Injection Attacks on Agentic AI Frameworks2026-09-08↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ECLIPSE: Self-Evolving Stealthy Prompt Injection Attack against Long-Horizon Agentic Systems2026-08-31↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- ↑ supports a claim here
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt InjectionThis source: Not what you've signed up for: Compromising Real-World LLM-Integrated …How much the field cites it — very heavily cited in the last 12 months1000+ in the last 12 months · 1870 totalpublished 2023checked 2026-09-04count capped at 1000 by the fetch2023↑ supports a claim here
- StruQ: Defending Against Prompt Injection with Structured QueriesThis source: StruQ: Defending Against Prompt Injection with Structured QueriesHow much the field cites it — very heavily cited in the last 12 months243 in the last 12 months · 394 totalpublished 2024checked 2026-09-042024↑ supports a claim here · ↑ supports a claim here
- Ignore Previous Prompt: Attack Techniques For Language ModelsThis source: Ignore Previous Prompt: Attack Techniques For Language ModelsHow much the field cites it — very heavily cited in the last 12 months475 in the last 12 months · 1055 totalpublished 2022checked 2026-09-04click for Semantic Scholar2022↑ supports a claim here
- The Instruction Hierarchy: Training LLMs to Prioritize Privileged InstructionsThis source: The Instruction Hierarchy: Training LLMs to Prioritize Privileged Inst…How much the field cites it — very heavily cited in the last 12 months285 in the last 12 months · 486 totalpublished 2024checked 2026-09-04click for Semantic Scholar2024↑ supports a claim here · ↑ supports a claim here
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.