prompt-injection · activeFollowing instructions hidden in data
Models treat instructions found in retrieved documents, tool outputs, or web pages as if they came from the user.
Also called: indirect prompt injection, jailbreak via content
Tags: autonomous-agent, coding-agent, customer-support, rag-qa, security
A capable model treats content it reads through tools as data. It follows only instructions from the user and the system, and it flags text that tries to redirect it rather than acting on it.
Claims
- observationsingle paperFor computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.
- mechanismsingle paperDemonstrates that instructions planted in retrieved content can hijack real LLM-integrated applications for data theft and manipulation.
- mechanismsingle paperFor long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.
- mechanismsingle paperFor open-weight agents where attention matrices are accessible, detecting injection by asking which context span actually drove the tool call — aggregating attention over the decision tokens and localizing the guiding span, then checking whether that span's provider has authority for the action — catches unauthorized tool invocations at higher true-positive and lower false-positive rates than static scanners that look for malicious-looking text, because benign-looking injected text is only harmful when it is what the model actually attends to.
- observationsingle paperPrompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.
- mechanismsingle paperSeparating prompt and data with reserved delimiters and fine-tuning on that structure blocks most injections at little cost to utility.
- mechanismsingle paperSimple goal-hijacking and prompt-leaking attacks succeed against production models with short adversarial strings.
- mechanismreplicatedSeparating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.
- mechanismsingle paperFor tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.
- mechanismsingle paperFor tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.
- mechanismsingle paperTraining models to rank system, user, and tool instructions by privilege improves robustness to injections in tool outputs.
Techniques
- Bound what the agent may do, not just what it may notprocessAn explicit allow/ask/deny policy plus rate, cost and time ceilings, enforced by the harness rather than requested of the model.
- Separate instructions from datatrainingMark the boundary between trusted instructions and untrusted content, and train or prompt the model to honor it.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.