open-weight-agents-where-attention-matrices-accessible-detecting-injection-askingFor open-weight agents where attention matrices are accessible, detecting injection by asking which context span actually drove the tool call — aggregating attention over the decision tokens and localizing the guiding span, then checking whether that span's provider has authority for the action — catches unauthorized tool invocations at higher true-positive and lower false-positive rates than static scanners that look for malicious-looking text, because benign-looking injected text is only harmful when it is what the model actually attends to.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Following instructions hidden in data
Observed on
Requires white-box access to attention during inference, plus provenance annotations that map token positions to providers; evaluated on tool-metadata poisoning and indirect prompt.
Sources
- Measured on MCPTox and InjecAgent across ten agent configurations from six model families, against static scanning (LLM-Guard, LLM Detector), behavior auditing (MCIP), and attribution baselines (MindGuard, TracLLM). Reported average AUROC 0.956 and 0.934 TPR at 0.067 FPR; static scanners were near 0.5-0.67 TPR with FPR above 0.36. Detectors are trained per model-dataset pair in the main table, so headline numbers are not a fully unseen-deployment estimate; transfer results are lower. [truncated]
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · unreviewed
- Prompt injection defenses that report near-zero attack success on short-context benchmarks lose most of that protection when the injected instruction sits inside a document of thousands to tens of thousands of tokens: fine-tuned separation defenses and detect-localize-remove pipelines still let a large share of injections through on paper review, resume screening, code review and email threads.Following instructions hidden in data · unreviewed
- For tool-using agents, restricting capabilities at the harness level after untrusted content enters the context — revoking or tightening the parameter envelopes of skills that lie on graph paths to deployer-defined forbidden states — cuts indirect-injection attack success further than prompt-level or per-action authorization defenses, and does so with no extra model calls or tokens, because enforcement reads the transition graph rather than the natural-language content.Following instructions hidden in data · unreviewed
- In open-weight instruction-tuned models, interventions that suppress caving to user pushback (DPO, SFT on chosen responses, or activation steering) also tend to reduce the model's rate of correcting a wrong answer when genuine supporting evidence is supplied, because the two answer-flip behaviors run on overlapping MLP neurons and attention heads with positively aligned steering directions.Telling the user what they want to hear · unreviewed
- For long-horizon tool-calling agents (tasks needing five or more calls), embedding state-transition cues — preconditions, invariants, completion states — into the descriptions of the tools on the intended chain is what actually steers the agent's trajectory; runtime corrective text appended to tool results only patches residual drift, and the plausible user prompt alone (persona, deadlines, format constraints) does not establish the trajectory at all.Following instructions hidden in data · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A study finding that the localized high-attention span does not correspond to the span that changes the tool call under ablation, or that static/content-based scanners match this detection rate at equal false-positive rate on the same benchmarks. Proposed technique, not catalogued: attention-localized behavior-guiding span adjudication.