entity-centric-evidence-grounded-qa-fact-verification-substituting-named-entities-type-matched CLM-0565mechanismsingle paperReviewed by AI
For entity-centric evidence-grounded QA and fact verification, substituting named entities with type-matched synthetic names and checking that the answer transforms correspondingly (back-maps to the original answer) detects ungrounded answers that self-consistency and evidence-sufficiency scores both miss, because synthetic names reduce the parametric activation that keeps a memorised answer stable under perturbation.
Capability: Saying it does not know
Observed on
Tasks whose questions and evidence are organised around named entities with reliable NER; not math or procedural reasoning; English only; tested on three backbones (GPT-5.1, LLaMA-.
Sources
- Validated by: Four benchmarks (HotpotQA, MIRAGE, FaithEval, FEVER) x three backbones against seven abstention baselines on F1/Acc/Reliability Score/Abstention Rate; on a 200-item hard subset where self-consistency agreement >=0.8 and sufficiency score >=0.5, abstention rate 0.71 vs 0.09 and 0.11; ablation shows removing back-mapping costs 0.067-0.076 RS and using real Wikipedia entities instead of synthetic ones costs 0.103-0.115 RS; human agreement 83.5%.. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In LLM fact-checking over supplied evidence, high veracity accuracy does not imply the verdict depends on that evidence: when the model's own cited sentences are iteratively removed, off-the-shelf models (GPT-4o-mini, Gemini-2.5-Flash-Lite, Qwen-2.5-32B, Llama-3.1-8B) keep their SUPPORT/REFUTE labels at a non-zero plateau instead of abstaining, and training with paired supervision that requires NOT_ENOUGH_INFO on the evidence-ablated input removes most of that persistence without losing accuracy.Checking claims against evidence · reviewed by AI
- When claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Stating false facts confidently · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- Models judge a syllogism's validity partly by how believable its conclusion is because validity and plausibility are encoded as linearly separable but strongly aligned directions in the residual stream: steering along the plausibility direction causally shifts validity judgements and vice versa, the degree of alignment predicts the size of the content effect across models, and a debiasing vector that separates the two reduces the effect and raises accuracy.Deducing validly from logical structure · reviewed by AI
- In retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Equal or lower abstention than self-consistency/sufficiency on unanswerable items those methods answer confidently, or no loss when synthetic replacement entities are swapped for real ones of the same type, would show the equivariance signal is not doing the attributed work.