retrieval-augmented-answers-splitting-answer-into-claims-checking-each-against CLM-0134In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.
Capability: Stating false facts confidently
Observed on
RAG answers on the RAGTruth benchmark with hand-annotated unsupported passages; 916 repaired answers judged by three LLM judges from different families; no human usefulness ratings.
Applies when
- Effect on the capability:
- helps
- Needs:
- documents or an index to retrieve from
- Fails when:
- Grounding trades against preservation: deleting unsupported claims removes the most unsupported content but keeps the least original text, and rewriting keeps the most text while removing the least.
Sources
- Measured on one benchmark with three LLM judges agreeing on ordering; retention percentages reported (64.3% deletion, 80.1% rewriting). Does not measure whether repaired answers remain useful to readers, and 83.5% of clean answers were also edited, so precision of the flagging step is a live concern.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- When claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Stating false facts confidently · reviewed by AI
- When retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.Stating false facts confidently · reviewed by AI
- For entity-centric evidence-grounded QA and fact verification, substituting named entities with type-matched synthetic names and checking that the answer transforms correspondingly (back-maps to the original answer) detects ungrounded answers that self-consistency and evidence-sufficiency scores both miss, because synthetic names reduce the parametric activation that keeps a memorised answer stable under perturbation.Saying it does not know · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- On self-contained linguistic Rosetta Stone puzzles where a single context example uniquely carrying a question-required token is deleted, frontier models told to output N/A when context is insufficient almost never abstain, and on some puzzles answer the damaged variant as well as or better than the original.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A repair pipeline where deletion does not reduce judged unsupported content more than rewriting, or where all strategies preserve text equally, would break the trade-off ordering; judges disagreeing on the ordering would too. Drafted stance toward decomposing-long-answers-into-atomic-facts-checking-each: supports -- The paper applies atomic-claim verification and shows the flagged claims can be acted on, extending fact-by-fact checking from detection to repair. Proposed technique, not catalogued: claim-level repair of unsupported content (delete, replace with source, rewrite).