retrieval-moves-the-trust-boundary-rather-than-removing-itmechanismreplicated
Grounding answers in retrieved documents reduces unsupported generation but transfers the trust problem to the retrieval channel: whatever lands in context is treated as reliable, which shows up both as following instructions planted in retrieved content and as deferring to whichever account is better represented when sources disagree.
Capability: Stating false facts confidently · Knowledge, Retrieval-augmented QA
Sources
- The instruction-following half: text planted in content the system retrieves can hijack real LLM-integrated applications, because the model does not distinguish retrieved data from instructions.
- The conflicting-sources half: models adopt a coherent counter-memory passage readily, and under conflict follow the majority while showing confirmation bias toward their own prior — neither of which is reliability-weighted.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- Grounding generation in retrieved documents improves factual accuracy on knowledge-intensive tasks.Stating false facts confidently
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · unreviewed
- In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Stating false facts confidently · unreviewed
- When the preference needed to answer a query lies in a different concept space from the cues in a user's dialogue history (e.g. inferring an emotion-regulation strategy from personality cues), retrieval-based and profile-summarization personalization methods barely beat a no-history baseline, and dumping the full history into the prompt does worse than no history at all.Remembering across sessions · unreviewed
- Reflect-and-retry raises task success when the feedback in the loop is a genuine external signal — a failing test, a compiler error, an environment outcome — but not when the "feedback" is the model's own unaided critique. The grounding, not the reflection step, is what does the work.Fixing its own mistakes · contested
Notes
Not an argument against retrieval — the reduction in unsupported generation is real and filed separately. This is the scope condition: retrieval is only as trustworthy as the channel feeding it, so the defenses belong at the boundary (see the prompt-injection capability), not in the generation step. Replicated in the sense of two independent papers reaching the same mechanism from different failure modes.