decomposing-response-into-atomic-claims-applying-post-hoc-temperature-or CLM-0375mechanismsingle paperReviewed by AI
Decomposing a response into atomic claims and applying post-hoc temperature or Platt scaling to per-claim self-verification/log-probability confidence lowers expected calibration error on straightforward factual QA (TriviaQA) in closed-box settings, but on adversarial false-premise questions (TruthfulQA) it is worse than response-level temperature scaling, because the per-claim score faithfully reports the model's own mistaken confidence rather than correcting it.
Capability: Saying it does not know
Observed on
Closed-box inference-time pipelines with no logits or fine-tuning required for API models; tested on six models (Llama-3.1-8B, Mistral-7B, Qwen2.5-7B, DeepSeek-R1, GPT-4, GPT-4o) o.
Sources
- Validated by: ECE, smECE, Brier, AUROC, AURC against seven baselines (few-shot likelihood, CoT, verbalized confidence, PS, TS, LMvsLM, HalluMeasure) on TriviaQA and TruthfulQA across six models; best on GPT-4o/TriviaQA (ECE 0.006 vs 0.096 for TS), worse than TS on TruthfulQA (0.412 vs 0.118).. Measured on two datasets and six models with seven baselines, but results are noisy: the proposed method loses to PS/TS on several TriviaQA rows (Llama ECE 0.200 vs PS 0.013) and the mechanism claim rests mainly on the GPT-4o and DeepSeek-R1 TruthfulQA comparisons with LMvsLM. [truncated]
Either way, bring evidence: a link, or what you saw.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For closed-box per-claim confidence built from self-verification and cross-sample consistency, the signal breaks on adversarial false-premise questions because it faithfully reports the model's own mistaken confidence: on TruthfulQA with GPT-4o claim-level calibration reached ECE 0.412 against 0.118 for response-level temperature scaling, while an active cross-examination protocol (LMvsLM) reached 0.112, so an external probe rather than a confidence estimate is what helps there.Saying it does not know · reviewed by AI
- When claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Stating false facts confidently · reviewed by AI
- On factual QA (817-item TruthfulQA multiple-choice, plus a 300-item Natural Questions short-answer set), a prompt-only three-stage gate in which the model lists the information units a question needs, scores its support for each, and answers only when the mean score clears a threshold lowers the unconditional wrong-answer rate and raises accuracy on answered items for all eleven models tested, but most of the abstentions it produces fall on questions plain chain-of-thought answered correctly (about 69% at the primary operating point), so the gain is bought with lost coverage rather than with selective self-knowledge.Saying it does not know · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- For response-level hallucination detection in retrieval-augmented settings, having the model first convert the reference documents into a relational SQL database and then check the response against SELECT query results raises precision and macro F1 over zero-shot direct prediction, over atomic-claim decomposition, and over flat JSON key-value extraction, without any task-specific fine-tuning.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A claim-level calibration pipeline using only the model's own confidence signals that matches or beats response-level temperature scaling on false-premise benchmarks across models, without an external probe or retrieval step.