span-level-hallucination-detection-leaderboards-where-absolute-scores-low-bootstrap-resamplingOn span-level hallucination-detection leaderboards where absolute scores are low, bootstrap resampling of the test set moves top systems across wide rank intervals, so point-estimate ordering does not establish that one detector beats another.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Whether the measurement made the finding
Observed on
Shared task on character-level hallucination span detection in vision-language outputs, four languages, 27 teams, 600+ submissions; best average scores around 0.58 correlation and .
Sources
- Measured: 25,000-sample bootstrap rank distributions per language and metric. Top English system had mean rank 2.9 with a 95% interval spanning ranks 1-10; intervals up to 15 positions in EN, ~5-6 in ZH. Rank correlations across data-construction partitions were also below 1, lowest for randomly sampled examples. Single task, single dataset; the inverse relation between score level and rank stability is observed, not isolated by design.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For closed-book hallucination detection, training a small supervised classifier over many heterogeneous uncertainty scores beats the best single score, but the gain comes from the diversity of sampling-based black-box signals: ensembles over only single-generation token-probability features add little, while black-box-only ensembles do nearly as well as the full set.Stating false facts confidently · unreviewed
- For detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.Stating false facts confidently · unreviewed
- When research-idea forecasts are scored by whether a later paper matches them under an LLM judge rubric, higher scores partly reflect broader, less specific ideas: the backbone with the higher hit rate also produced measurably more general forecasts and had more retrieved candidate papers pass the gate, and tightening the specificity threshold does not separate the two.Whether the measurement made the finding · unreviewed
- In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.Whether the measurement made the finding · unreviewed
- When comparing a weight-shared recursive model against a depth-matched plain transformer on code generation, the ranking read from a single checkpoint can invert during training, because the larger-parameter arm passes its validation optimum and overfits while the parameter-lean recursive arm keeps improving — so a late-epoch "recursion fits better" reading is a degradation artifact, not a capability difference.Whether the measurement made the finding · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Bootstrap resampling of the same test set yields narrow rank intervals (e.g. top systems confined to one or two positions) despite the low absolute scores. Proposed technique, not catalogued: Bootstrap rank intervals reported alongside leaderboard scores.