2m-parameter-classifier-head-reads-hidden-states-already-computed-duringA ~2M-parameter classifier head that reads hidden states already computed during generation (tapping shallow, middle and deep layers of the generating model) detects response-level hallucination at least as well as specialized hidden-state hallucination detectors, and its detection quality rises with the scale of the base model it is attached to.
Ingested from a paper but not yet reviewed by a human. It is deliberately inert: it does not move any technique’s standing, does not count toward the backtest, and is excluded anywhere a claim would carry weight. Read the source before relying on it.
Capability: Stating false facts confidently
Observed on
Two Ling-3.0 base models (tiny and flash); response-level ROC AUC on six offline hallucination benchmarks and seven free-generation datasets; head trained on curated factuality/hal.
Sources
- supportsSingProbe Technical ReportMeasured AUC, not accuracy at a threshold. Macro-average offline AUC 0.7765 (tiny) and 0.8012 (flash) vs best baseline DRIFT 0.7408/0.8000; online free-generation average 0.6786/0.7271 vs DRIFT 0.6316/0.6546, with labels from an LLM judge. Absolute AUCs are moderate and per-benchmark ranking is mixed (FAVA, RAGTruth, BBH favor baselines). Layer ablation shows hallucination gains more from multi-layer taps than safety does, supporting the mechanism reading. Only one model family, so scale claim rests on two points.
Disagreeing is the most useful thing you can do here. Both sides of every contested claim in this catalog were assembled by the same person, which is its weakest point.
Related claims
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · unreviewed
- In open-weight 7-8B instruct models, a supervised detector built on how hidden states change between adjacent layers separates factually wrong from correct answers better than a probe on any single layer, and shuffling the layer order destroys most of the signal — so the ordering of the depth trajectory, not just the content of one layer, carries the information.Stating false facts confidently · unreviewed
- In open-source vision-language models (LLaVA-1.5, Qwen-VL-Chat, Qwen2.5-VL, LLaVA-NeXT), a training-free decoding intervention that detects large layer-to-layer and step-to-step shifts in hidden states and nudges the diverging states back toward their prior value lowers object-hallucination rates, but the lower rate comes with fewer objects mentioned and shorter outputs.Grounding answers in the image · unreviewed
- For black-box hallucination detection using only sampled responses and token log-probabilities, semantic-entropy signals go to zero whenever the samples collapse into one semantic cluster — which happens for a large share of hallucinated queries, up to nearly all of them on tasks with confusable entities and figures — while token-level uncertainty still separates many of those cases, so the two signals fail on different queries and combining them detects more than either alone.Stating false facts confidently · unreviewed
- For closed-book hallucination detection, training a small supervised classifier over many heterogeneous uncertainty scores beats the best single score, but the gain comes from the diversity of sampling-based black-box signals: ensembles over only single-generation token-probability features add little, while black-box-only ensembles do nearly as well as the full set.Stating false facts confidently · unreviewed
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: A probe trained the same way on reused hidden states scores no better than chance, or falls clearly below external detectors of similar training budget, or its AUC fails to improve when moved from a small to a larger base model. Proposed technique, not catalogued: intrinsic hidden-state probe for hallucination risk during decoding.