retrieval-moves-the-trust-boundary-rather-than-removing-it CLM-0135mechanismreplicatedReviewed by AI and Russ Miller
Grounding answers in retrieved documents reduces unsupported generation but transfers the trust problem to the retrieval channel: whatever lands in context is treated as reliable, which shows up both as following instructions planted in retrieved content and as deferring to whichever account is better represented when sources disagree.
Capability: Stating false facts confidently · Knowledge, Retrieval-augmented QA
Applies when
- Effect on the capability:
- helps, narrowly
- Needs:
- documents or an index to retrieve from
- Fails when:
- Retrieved content carries planted instructions, or the wrong account is better represented among conflicting sources. Whatever lands in context is treated as reliable.
Sources
- The instruction-following half: text planted in content the system retrieves can hijack real LLM-integrated applications, because the model does not distinguish retrieved data from instructions.
- The conflicting-sources half: models adopt a coherent counter-memory passage readily, and under conflict follow the majority while showing confirmation bias toward their own prior — neither of which is reliability-weighted.
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Grounding generation in retrieved documents improves factual accuracy on knowledge-intensive tasks.Stating false facts confidently
- When a retrieval store holds both the correct answer and a common misconception for the same question, injecting all retrieved memories raises the hallucination rate above a memory-free baseline for weaker models (53.0% vs 23.0% on TruthfulQA with gemma-4-E4B-it), while a stronger backbone (deepseek-v4-flash) instead shows RAG below its own no-memory rate, so conflict-driven amplification is concentrated in models susceptible to memory injection.Stating false facts confidently · reviewed by AI
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · reviewed by AI
- In ESG-report RAG question answering with retrieval held fixed at a single top-1 chunk, open-weight 2B-30B generators vary widely in how well they stay anchored to the retrieved passage (faithfulness 0.607-0.822) yet all land in a narrow low band of factual correctness (0.387-0.449), and the most faithful model is the least factually correct, so groundedness to context and correctness of extracted ESG facts do not track each other.Competence in a specialist domain · reviewed by AI
- In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Stating false facts confidently · reviewed by AI
Notes
Not an argument against retrieval — the reduction in unsupported generation is real and filed separately. This is the scope condition: retrieval is only as trustworthy as the channel feeding it, so the defenses belong at the boundary (see the prompt-injection capability), not in the generation step. Replicated in the sense of two independent papers reaching the same mechanism from different failure modes.