closed-box-per-claim-confidence-built-self-verification-cross-sample-consistency-signal-breaks CLM-0554For closed-box per-claim confidence built from self-verification and cross-sample consistency, the signal breaks on adversarial false-premise questions because it faithfully reports the model's own mistaken confidence: on TruthfulQA with GPT-4o claim-level calibration reached ECE 0.412 against 0.118 for response-level temperature scaling, while an active cross-examination protocol (LMvsLM) reached 0.112, so an external probe rather than a confidence estimate is what helps there.
Capability: Saying it does not know
Observed on
Closed-box settings (no logits or fine-tuning) on adversarial misconception/false-premise QA; six models including GPT-4o and DeepSeek-R1; shown on TruthfulQA only..
Sources
- Measured on two datasets and six models with seven baselines, but the adversarial failure is argued from a single dataset (TruthfulQA) and the mechanism (pipeline mirrors the model's own confidence) is inferred rather than isolated; alpha was tuned per model and sometimes defaulted untuned, so the comparison is not fully matched.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- Decomposing a response into atomic claims and applying post-hoc temperature or Platt scaling to per-claim self-verification/log-probability confidence lowers expected calibration error on straightforward factual QA (TriviaQA) in closed-box settings, but on adversarial false-premise questions (TruthfulQA) it is worse than response-level temperature scaling, because the per-claim score faithfully reports the model's own mistaken confidence rather than correcting it.Saying it does not know · reviewed by AI
- When claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Stating false facts confidently · reviewed by AI
- For claim-level hallucination detection on long-document QA, estimating a claim's reliability from how central it is in a graph of sampled model responses cannot separate true claims from confidently repeated false ones, because both get high agreement; adding trusted source documents as anchor nodes with higher-weight edges shifts the basis from internal consensus to external grounding and yields a Grounded pool with few hallucinations (97.6% precision on QASPER), at the cost of conservatively leaving correct-but-novel claims on the boundary.Stating false facts confidently · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- Across 14 frontier and open model routes scored on 3,000 boundary questions (fabricated entities, post-cutoff events, hyper-niche facts, redacted-document values), the rate at which a model restrains itself and calibrates confidence varies by a factor of about 2.6 (composite 0.31 to 0.81) while all models answer a document-grounded restored-span probe at near-ceiling (pooled 99.2%, no detectable between-model variation), so epistemic restraint is not explained by the ability to report information placed in front of the model.Saying it does not know · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A closed-box, confidence-only per-claim signal that lowers ECE below response-level temperature scaling on false-premise benchmarks without any external probe or retrieval. Drafted stance toward decompose-response-into-atomic-claims-applying-post-hoc-temperature-or: supports -- The paper reports the same pattern: per-claim calibration lowers ECE on straightforward factual QA but is worse than response-level temperature scaling on false-premise questions. Automatic check flagged: related_claim_id "decompose-response-into-atomic-claims-applying-post-hoc-temperature-or" does not exist.