span-level-hallucination-detection-leaderboards-where-absolute-scores-low-bootstrap-resampling CLM-0157On span-level hallucination-detection leaderboards where absolute scores are low, bootstrap resampling of the test set moves top systems across wide rank intervals, so point-estimate ordering does not establish that one detector beats another.
Capability: Whether the measurement made the finding
Observed on
Shared task on character-level hallucination span detection in vision-language outputs, four languages, 27 teams, 600+ submissions; best average scores around 0.58 correlation and .
Sources
- Measured: 25,000-sample bootstrap rank distributions per language and metric. Top English system had mean rank 2.9 with a 95% interval spanning ranks 1-10; intervals up to 15 positions in EN, ~5-6 in ZH. Rank correlations across data-construction partitions were also below 1, lowest for randomly sampled examples. Single task, single dataset; the inverse relation between score level and rank stability is observed, not isolated by design.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For closed-book hallucination detection, training a small supervised classifier over many heterogeneous uncertainty scores beats the best single score, but the gain comes from the diversity of sampling-based black-box signals: ensembles over only single-generation token-probability features add little, while black-box-only ensembles do nearly as well as the full set.Stating false facts confidently · reviewed by AI
- When multiple stochastic runs of several mid-tier LLMs each produce an unranked set of candidate security requirements from the same control templates, pooling the runs recovers far more valid items than any single run but also many hallucinations, and ranking the pooled candidates by how many runs produced them separates valid from hallucinated items well enough to beat every individual run's precision-recall and ROC area.Stating false facts confidently · reviewed by AI
- Used to steer test-time compute, generative LLM judges (7B-70B) rerank candidate responses about as well as outcome reward models, are consistently worse than process reward models at step-level beam search, and their natural-language critiques, the one thing only a judge can produce, did not guide generators to better responses on math, code or instruction following.Judging another's work as well as a careful reviewer would · reviewed by AI
- In a 20-year deterministic football-management simulation with hidden state, adaptive rivals and 340-400 decision stops, final score across 15 frontier models is not ordered by model scale, price, vendor or token spend, and early standings barely predict the final order (rank correlation with the final order about 0.19 at year 5, rising to 0.78 by year 15), so short-horizon measurements would rank a different set of models.Completing tasks that take many steps or many hours · reviewed by AI
- For detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Bootstrap resampling of the same test set yields narrow rank intervals (e.g. top systems confined to one or two positions) despite the low absolute scores. Proposed technique, not catalogued: Bootstrap rank intervals reported alongside leaderboard scores.