structured-separation-needs-training-and-reduces-rather-than-eliminatesmechanismreplicated
Separating instructions from data raises injection resistance substantially, but both published versions get their strength from fine-tuning the model on the separation, and both report improved robustness rather than elimination. Treat it as one layer of defense in depth; the prompt-only variant, without training, has no measured efficacy behind it here.
Capability: Following instructions hidden in data · Security, Autonomous agent
Observed on
2024, GPT-3.5 class and open base models.
Sources
- Structured queries need both a secure front-end and "a specially trained LLM" — a base model converted by structured instruction tuning to ignore instructions in the data channel. Reports significantly improved resistance with little utility cost, not immunity.
- Instruction hierarchy likewise works through generated training data teaching models to ignore lower-privileged instructions, and reports drastically increased robustness — including to unseen attack types — again as a degree, not a guarantee.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Prompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.Following instructions hidden in data · unreviewed
- Separating prompt and data with reserved delimiters and fine-tuning on that structure blocks most injections at little cost to utility.Following instructions hidden in data
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · unreviewed
- For computer-use agents operating on a real OS, indirect-injection success rates measured with a fixed hand-written injection template understate vulnerability, because an attacker that adapts phrasing from the agent's own refusal trajectories can reach substantial success on models the fixed template never penetrates.Following instructions hidden in data · unreviewed
- Training against an explicit set of principles produces models that are more harmless without becoming evasive.Prioritizing safety under conflicting goals
Notes
The gap worth being honest about: the technique record describes a prompt-level fallback for people without training access, and neither source evaluates that. So the claim asserts what the papers show (training-based, partial) and marks the untrained variant as unmeasured rather than assuming it inherits the result.