open-weight-7-8b-instruct-models-supervised-detector-built-how-hidden CLM-0112In open-weight 7-8B instruct models, a supervised detector built on how hidden states change between adjacent layers separates factually wrong from correct answers better than a probe on any single layer, and shuffling the layer order destroys most of the signal — so the ordering of the depth trajectory, not just the content of one layer, carries the information.
Capability: Stating false facts confidently
Observed on
Requires white-box access to intermediate activations and labelled data to train the scoring head and calibrator; evaluated on English closed-book QA and dialogue (TruthfulQA, Halu.
Sources
- Single paper, single author, author-verified numbers. On Llama-3-8B/TruthfulQA the full method reports 75.5% AUROC vs 66.8% for the best single layer, 66.4% for a matched final-layer MLP, and 58.3% with shuffled layer order; baselines include logit entropy, perplexity, spectral geometry, semantic entropy and an NLI verifier. Transfer to other backbones and datasets loses a few points. No independent replication, and the detector's advantage over the multi-sample baselines is small enough to be within plausible noise given no error bars.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- A ~2M-parameter classifier head that reads hidden states already computed during generation (tapping shallow, middle and deep layers of the generating model) detects response-level hallucination at least as well as specialized hidden-state hallucination detectors, and its detection quality rises with the scale of the base model it is attached to.Stating false facts confidently · reviewed by AI
- In 7-8B open-weight instruct models (Mistral-7B, Qwen2.5-7B, Llama-3-8B) answering closed-book factual questions, a small set of feed-forward neurons (top 1% or fewer) selected on one dataset by class-conditional separation of their residual-space contribution at the final prompt token transfers to other factual QA datasets, where probes on those neuron activations detect hallucination about as well as probes on full residual-stream or MLP-output states.Stating false facts confidently · reviewed by AI
- For small open-weight models (1.5-4B) generating answers in a closed-domain RAG setup, hallucinated generations show a broken hidden-state proximity graph between the retrieved context entity and the generated entity — connectivity below 20% and path lengths above 7 hops, versus near-full connectivity at about 3 hops for grounded answers — giving a training-free detection signal (AUROC 0.89 on Qwen2.5-1.5B).Stating false facts confidently · reviewed by AI
- A linear probe on a tool-calling model's hidden state at the last generated token detects incorrect tool-calls — including wrong-but-well-typed argument values that parsers and logs do not catch — and works better on larger models and at middle-to-late layers than at the final layer.Using the tools it is given · reviewed by AI
- In frozen large vision-language models, token-level hallucination evidence is distributed across the relational, spatial and sequential structure of the model's internal trace rather than contained in any single token's hidden state, so a detector that reads the attention graph over image patches, query and response tokens with interleaved GNN/ViT/GRU modules localizes hallucinated spans better than flat probes on the same activations.Grounding answers in the image · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A replication where a probe on the best single layer, or a matched-size head on the final layer, matches the cross-layer transition detector's AUROC, or where shuffling layer order leaves accuracy roughly unchanged. Proposed technique, not catalogued: Inter-layer transition-divergence hallucination probe.