hallucination-metric-unsupported-claim-rate-computed-over-whatever-claims-model CLM-0201When a hallucination metric is an unsupported-claim rate computed over whatever claims a model happens to make, models that avoid committing to verifiable claims or directional outputs score best, so the ranking partly measures abstention rather than grounding — on financial chart-to-recommendation tasks the lowest unsupported-claim model made directional recommendations on only 6.4% of queries and ranked near the bottom once coverage was factored in.
Capability: Whether the measurement made the finding
Observed on
Claim-level hallucination rates on open-ended generation where the model may hedge, abstain, or emit no checkable claims; shown on 969 financial chart-reasoning queries across 20 v.
Sources
- Measured on one benchmark (323 HS300 stocks, three input modalities, ten deterministic OHLCV-derived evidence anchors) over 20 models; rank reversal is demonstrated by one clear pair (Gemma-4-E2B-It lowest UCR 39.9% but 6.4% coverage and NDR 5.1%, vs Qwen2.5-VL-3B NDR 32.1% at 82.5% coverage). Oracle-aided verification lowering claims while collapsing coverage is reported but only in aggregate. Claim extraction uses pattern-based detectors, so 'no verifiable claim extracted' may partly be a detector limit rather than genuine abstention; [truncated]
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For claim-level hallucination detection on long-document QA, estimating a claim's reliability from how central it is in a graph of sampled model responses cannot separate true claims from confidently repeated false ones, because both get high agreement; adding trusted source documents as anchor nodes with higher-weight edges shifts the basis from internal consensus to external grounding and yields a Grounded pool with few hallucinations (97.6% precision on QASPER), at the cost of conservatively leaving correct-but-novel claims on the boundary.Stating false facts confidently · reviewed by AI
- An abstention rule built from self-consistency, where the model samples several answers and itself scores how many agree with a candidate, can be thresholded by conformal prediction to bound the hallucination rate at a chosen level on closed-book QA; against log-probability confidence it abstains much less often on long-form answers (Temporal Sequences) for the same error bound and matches it on short answers (TriviaQA).Saying it does not know · reviewed by AI
- When claim-level uncertainty is estimated from a response-claim graph, anchoring the graph with trusted external priors (edges weighted by entailment from verified source sections) and scoring claims by weighted closeness centrality separates ungrounded claims from repeatedly-generated-but-false ones better than internal-consistency centrality alone, and yields retrieval whose answerability tracks its accuracy instead of the near-100% answer rate GraphRAG shows on the same long documents.Stating false facts confidently · reviewed by AI
- When multiple stochastic runs of several mid-tier LLMs each produce an unranked set of candidate security requirements from the same control templates, pooling the runs recovers far more valid items than any single run but also many hallucinations, and ranking the pooled candidates by how many runs produced them separates valid from hallucinated items well enough to beat every individual run's precision-recall and ROC area.Stating false facts confidently · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Show that on such tasks unsupported-claim rate ranks models the same way as a coverage-weighted measure, i.e. low-hallucination models are not systematically the ones that abstain or emit fewer checkable claims. Proposed technique, not catalogued: Coverage-weighted grounding score.