structured-separation-needs-training-and-reduces-rather-than-eliminates CLM-0161mechanismreplicatedReviewed by AI and Russ Miller
Separating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.
Capability: Following instructions hidden in data · Security, Autonomous agent
Observed on
2024, GPT-3.5 class and open base models.
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- access to train the weights
- Cost:
- high
- Fails when:
- It is used prompt-only, without training, where it has no measured efficacy. Even trained, it reduces rather than eliminates injection, so it is one layer of defence in depth.
Sources
- Structured queries need both a secure front-end and "a specially trained LLM" — a base model converted by structured instruction tuning to ignore instructions in the data channel. Reports significantly improved resistance with little utility cost, not immunity.
- Instruction hierarchy likewise works through generated training data teaching models to ignore lower-privileged instructions, and reports drastically increased robustness — including to unseen attack types — again as a degree, not a guarantee.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Prompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.Following instructions hidden in data · reviewed by AI
- Separating prompt and data with reserved delimiters and fine-tuning on that structure blocks most injections at little cost to utility.Following instructions hidden in data
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · reviewed by AI
- For tool-using agents trained against adaptive indirect prompt injection, attack success measured against a single frozen adaptive attacker understates vulnerability compared with taking the per-case success union over a population of retained attacker checkpoints (0.25% vs 4.97% on the same 1,187 adaptive cases for the trained defender).Following instructions hidden in data · reviewed by AI
- For computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.Following instructions hidden in data · reviewed by AI
Notes
The gap worth being honest about: the technique record describes a prompt-level fallback for people without training access, and neither source evaluates that. So the claim asserts what the papers show (training-based, partial) and marks the untrained variant as unmeasured rather than assuming it inherits the result.