hallucination · activeStating false facts confidently
Models generate plausible but false statements, citations, and details, with no signal that they are unsure.
Also called: confabulation, fabrication, factual error
Tags: chat-assistant, general, knowledge, rag-qa
A capable model states only what it has grounds for, marks uncertainty, distinguishes what it recalls from what it infers, and declines to invent specifics such as names, numbers, quotes, or citations.
What counts as this capability
Scope boundary used when deciding whether a paper is really about this capability, rather than merely mentioning it.
Confidently stating things that are not true: invented citations, URLs, numbers, entities, or events, in the absence of a retrieval failure. In scope when the paper measures, explains, or mitigates unsupported generation. NOT in scope when "hallucination" is used loosely for any error, or for perception errors in vision models unless the paper treats them as the same phenomenon.
Claims
- mechanismsingle paperA ~2M-parameter classifier head that reads hidden states already computed during generation (tapping shallow, middle and deep layers of the generating model) detects response-level hallucination at least as well as specialized hidden-state hallucination detectors, and its detection quality rises with the scale of the base model it is attached to.
- mechanismsingle paperAccuracy on a fact scales with how many pretraining documents mention it, so long-tail facts are systematically unreliable.
- mechanismsingle paperFor black-box hallucination detection using only sampled responses and token log-probabilities, semantic-entropy signals go to zero whenever the samples collapse into one semantic cluster — which happens for a large share of hallucinated queries, up to nearly all of them on tasks with confusable entities and figures — while token-level uncertainty still separates many of those cases, so the two signals fail on different queries and combining them detects more than either alone.
- mechanismreplicatedFor closed-book hallucination detection, training a small supervised classifier over many heterogeneous uncertainty scores beats the best single score, but the gain comes from the diversity of sampling-based black-box signals: ensembles over only single-generation token-probability features add little, while black-box-only ensembles do nearly as well as the full set.
- observationsingle paperOn code prompts that are unsatisfiable by design, open-weight code and reasoning models refuse based on how suspicious the request looks rather than how deeply impossible it is: they refuse famous theoretical impossibilities far more often than requests for plausible-sounding nonexistent packages, and the stronger models in the set close the gap on theory while barely improving on fabricated ecosystem entities.
- mechanismsingle paperDecomposing long answers into atomic facts and checking each with search catches most errors and beats human annotators on cost and agreement.
- mechanismsingle paperFor detecting hallucinated text spans in vision-language model outputs, span localization and confidence calibration come from different architectures: a fine-tuned generative VLM proposes better span boundaries but scores them poorly, while a discriminative token tagger trained with a direct regression loss on human-consensus confidence gives better-correlated probabilities but conservative spans, so using the generative spans as proposals and the tagger's probabilities as scores beats either alone.
- mechanismsingle paperFor filtering unsupported LLM-generated vulnerability reports in security triage, statistical machine-text detectors are the wrong instrument: they score provenance, not correctness, and human-written security reports are template-driven and low-perplexity, so they trip the same signal that flags machine text.
- mechanismsingle paperWhen fine-tuning a base model on instruction data, removing from the training targets any factual claim the base model cannot consistently recall raises the supported-claim rate of later long-form generations, but the gain comes largely from a more conservative response policy — more refusals and fewer supported claims per answer — not from generating more correct facts at equal coverage.
- observationsingle paperGeneral-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.
- mechanismsingle paperGrounding generation in retrieved documents improves factual accuracy on knowledge-intensive tasks.
- mechanismsingle paperIn LLM-based ASR, steering decoding toward acoustic evidence only helps if candidates are first restricted to tokens within a small likelihood margin of the greedy token: unconstrained selection by an acoustic compatibility score removes hallucinated outputs but destroys transcription accuracy through unbounded insertions, while the constrained version removes a large share of hallucinations at near-zero WER/CER cost.
- mechanismsingle paperLong-form biographies from strong models contain a substantial share of unsupported atomic facts, rising for less well-known subjects.
- mechanismsingle paperModels reproduce common human misconceptions, and larger models were not more truthful on this benchmark.
- mechanismsingle paperIn multi-agent LLM systems where one agent starts with an erroneous shared belief, broadcasting every agent's full context at every step raises the rate of false assertions above doing no synchronization at all, because the error propagates to agents that were previously correct — and the harm appears only in tasks where one wrong fact cascades across semantically linked dimensions (destination to airport to weather), not where agent contexts are largely orthogonal.
- mechanismsingle paperIn open-weight 7-8B instruct models, a supervised detector built on how hidden states change between adjacent layers separates factually wrong from correct answers better than a probe on any single layer, and shuffling the layer order destroys most of the signal — so the ordering of the depth trajectory, not just the content of one layer, carries the information.
- mechanismsingle paperOn questions whose premises are false or internally contradictory, prompting mid-size instruct models (Qwen2.5-7B, LLaMA-3.1-8B, Gemma3-12B) to reason step by step can lower accuracy below plain answering, because the reasoning chain elaborates from the flawed premise instead of challenging it.
- mechanismsingle paperIn retrieval-augmented answers, splitting an answer into claims, checking each against the retrieved source, and then acting on the flagged claims reduces the share of answers judged to contain unsupported content — but the strategies trade grounding against preservation: deleting unsupported claims reduces unsupported content most while retaining the least original text, and rewriting retains the most while reducing least.
- mechanismreplicatedGrounding answers in retrieved documents reduces unsupported generation but transfers the trust problem to the retrieval channel: whatever lands in context is treated as reliable, which shows up both as following instructions planted in retrieved content and as deferring to whichever account is better represented when sources disagree.
- mechanismsingle paperWhen retrieval for a summarization task pulls definitions by surface term match from a general encyclopedia, ambiguous terms fetch the wrong sense and the model incorporates that unsupported background into its output, lowering claim-level factuality below the no-retrieval baseline even when the prompt tells it to ignore unrelated retrieved content.
- mechanismsingle paperSurveys causes, kinds, and detection of hallucination across data, training, and inference, and finds no complete solution.
- mechanismsingle paperTraining a language model with differential privacy — whether DP-SGD pre-training or DP LoRA fine-tuning — makes it generate more false claims about facts in its training data than a non-private counterpart, and the effect grows as the privacy budget tightens, because per-example gradient clipping plus noise prevents low-frequency facts from being acquired at all while flattening the next-token distribution onto incorrect alternatives.
Techniques
- Check long answers fact by facttoolingSplit a long answer into atomic claims and verify each against search or a corpus.
- Ground answers in retrieved documentsretrievalRetrieve relevant documents and generate from them, with the model deciding when retrieval is needed.
Capabilities are a way of carving up the subject, and carvings are arguable. Say so if this one is wrong — especially a proposed one, which a pipeline added because several papers used the same framing, not because anyone decided it was right.