long-trusted-instructions-untrusted-retrieved-content-occupy-same-model CLM-0433As long as trusted instructions and untrusted retrieved content occupy the same model context, defenses that only change how untrusted text is marked or weighted (instruction hierarchies, spotlighting, content detectors) reduce indirect-injection success but do not remove it; only defenses that isolate contexts or restrict the action set give guarantees, at a cost in the agent's ability to combine information across sources.
Capability: Following instructions hidden in data
Observed on
LLM agents that read web pages, emails, documents or database records into the same context as their instructions; evidence from static-LLM instruction-hierarchy training and contr.
Sources
- Narrative review, not new measurement: cites indirect injection against deployed applications, a 1,054-case benchmark over 30 tool-integrated agents where a ReAct GPT-4 agent was attacked in roughly one case in four with higher rates under reinforced injections, and AgentDojo. The review notes studies use different agents, attack sets and success criteria, so cross-method comparison is weak and the mechanism claim is partly argued rather than isolated.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For tool-using agents, a safety harness searched separately for each model-and-domain deployment (jointly evolving a natural-language policy and executable pre/post-tool-call code, screened by a fresh-context adversarial reviewer) reaches a better safety-utility trade-off than fixed expert-designed defenses ported unchanged across deployments, because domain semantics determine which safety relations and trajectory state must be checked while the victim model's failure behavior determines whether a check should be semantic or deterministic and where in execution it must run.Following instructions hidden in data · reviewed by AI
- For tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.Following instructions hidden in data · reviewed by AI
- For decoder-only 360M-8B instruct models, adding application-supplied provenance labels as an architectural signal — origin embeddings plus a learned ring-indexed attention bias that suppresses assistant and user queries attending to untrusted-source keys — cuts indirect-injection attack success on unseen task domains and injection styles further than the same two-stage fine-tuning without the architectural components, even though both are equal in-distribution.Following instructions hidden in data · reviewed by AI
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · reviewed by AI
- When automatically rewriting the prompts of a multi-agent LLM system from its failures, locating the responsible agent by injecting a hint into one agent at a time and using that agent's hint-corrected intermediate output as the supervision signal produces prompt updates that generalize better than critique derived only from the system-level output.Improving its own scaffolding from failures · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A marking- or weighting-based defense that keeps trusted instructions and untrusted content in one context and drives injection success to near zero on an adaptive benchmark such as AgentDojo, without loss of task utility. Drafted stance toward structured-separation-needs-training-and-reduces-rather-than-eliminates: supports -- The review concludes that source-marking and hierarchy defenses give partial risk reduction rather than elimination, and that the residual risk follows from shared context.