arxiv-2608-22483 SRC-0432 · paperClaim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
Toghrul Abbasli, Kentaroh Toyoda, Yuan Wang, Li Chen
Created: 2026-08-23 · Ingested: 2026-10-17
https://arxiv.org/abs/2608.22483(opens in a new tab)Large Language Models (LLMs) increasingly support decision-making in high-stakes domains, but they often hallucinate and express confidence that is misaligned with factual correctness. Response-level confidence is a coarse signal: a single generation can mix correct and incorrect statements, so a single number is not actionable for users that must accept, reject, or verify individual pieces of information. We study claim-level confidence calibration as a decision-relevant uncertainty signal: each response is decomposed into atomic, verifiable claims, and each claim is assigned a calibrated con
Referenced by
Claims in this catalog that draw on this source, and whether as support or counterpoint.
- supportsFor closed-box per-claim confidence built from self-verification and cross-sample consistency, the signal breaks on adversarial false-premise questions because it faithfully reports the model's own mistaken confidence: on TruthfulQA with GPT-4o claim-level calibration reached ECE 0.412 against 0.118 for response-level temperature scaling, while an active cross-examination protocol (LMvsLM) reached 0.112, so an external probe rather than a confidence estimate is what helps there.
- supportsDecomposing a response into atomic claims and applying post-hoc temperature or Platt scaling to per-claim self-verification/log-probability confidence lowers expected calibration error on straightforward factual QA (TriviaQA) in closed-box settings, but on adversarial false-premise questions (TruthfulQA) it is worse than response-level temperature scaling, because the per-claim score faithfully reports the model's own mistaken confidence rather than correcting it.