llm-agent-must-reconcile-user-supplied-identity-credentials-against-database CLM-0077When an LLM agent must reconcile user-supplied identity credentials against database records before a sensitive read or write, frontier and open models frequently skip the cross-field consistency check and act anyway, and the failure rate barely moves whether the request is simple or has several parallel sub-requests and whether the forged field is visually near-identical to the true one or completely unrelated.
Capability: Checking claims against evidence
Observed on
Multi-turn simulated retail customer-service tasks with stateful tools and a domain policy requiring credential verification; nine models, one trial per task, greedy decoding, poli.
Sources
- Measured on 93 retail tasks within a 238-task benchmark, scored by an assertion model plus human trajectory verification; best retail score was 0.7312 (Claude Haiku 4.5) with several models far lower. The complexity and similarity insensitivity is reported qualitatively without per-condition numbers, so that part is argued more than quantified. Single trial per task limits variance estimates.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For tool-using agents, enforcing injection defense as a deterministic provenance check on sensitive tool parameters — with the only model call reading the trusted user request and never fetched content — makes admission decisions invariant to how an injection is worded, whereas defenses that judge the agent's runtime plan or behavior with a model can be steered by reworded injections.Following instructions hidden in data · reviewed by AI
- For LLM agents acting through stateful tools over many turns, unsafe policy-violating completions accumulate with the interaction budget and are roughly doubled when the adversarial user conditions each next request on the agent's actual prior response, so single-turn or short-prefix safety evaluation understates the failure rate.Using the tools it is given · reviewed by AI
- For persistent-memory agents, any memory-write rule that decides using only recency and provenance is stuck on a single tradeoff — accepting more genuine preference updates means admitting more poisoned ones — because an adversary who can launder a claim through the user's own channel matches the statistics a genuine revision produces; conditioning the write decision on the inferred authenticity of the claim given the interaction history moves both axes at once.Remembering across sessions · reviewed by AI
- In multi-agent LLM networks where a correct minority peer disagrees with an agreeing majority, ranking peers by raw divergence from the receiver's own belief systematically prefers the wrong peers, while scoring each peer by asserted precision divided by its divergence from a second-order model of what that peer should have concluded given its data subscription isolates the correct one.Getting more from several models than from one · reviewed by AI
- In multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: Showing that detection and clarification rates rise substantially as the forged credential becomes more dissimilar from the true one, or as the number of parallel requests drops, under the same harness. Proposed technique, not catalogued: conflict-injection agent benchmark.