synthetic-hallucination-benchmarks-where-faithful-items-human-written-hallucinated-items CLM-0165In synthetic hallucination benchmarks where faithful items are human-written and hallucinated items are LLM-rewrites, a detector's binary score partly measures human-vs-machine style rather than factuality: feeding faithful LLM rewrites of the same truthful articles makes some fine-tuned encoders flag nearly all of them as hallucinated.
Capability: Whether the measurement made the finding
Observed on
Fine-tuned multilingual encoders (ReMBERT, mDeBERTa) trained on an LLM-generated news hallucination corpus; provenance analysis run on Russian original text only..
Sources
- Measured on 290 truthful human articles, 290 factcheck.kz human-written fakes, 290 faithful Gemini rewrites, and synthetic hallucinations. ReMBERT flagged 0.352 of human truthful vs 1.000 of faithful LLM rewrites; mDeBERTa 0.221 vs 0.586, so the effect is real but model-dependent. Single language, single domain, two encoders; register differences between real fakes and newswire are acknowledged but not fully controlled.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- For encoder-based hallucination/claim-faithfulness detectors moved from general-domain data to biomedical claim verification, which corpus the encoder was pre-trained on matters more than which data it is fine-tuned on: a PubMed-pretrained encoder fine-tuned on a few hundred SciFact pairs beat both a general encoder fine-tuned on the same pairs and a general encoder pre-fine-tuned on a large general hallucination set.Competence in a specialist domain · reviewed by AI
- On adversarial video hallucination pairs scored only when a method both accepts the factual statement and rejects the counterfactual one, purpose-built video hallucination detectors (embedding-similarity, learned-entailment, atomic-fact and attention-based) top out around 34.63% overall accuracy, well below a general frontier video-language model prompted to judge the same items (about 83.63%).Stating false facts confidently · reviewed by AI
- For training-free inference-time hallucination mitigation in 7B vision- language models, reported gains on object-hallucination benchmarks are largely inseparable from reduced informativeness: hallucination rate and object recall/coverage move together, so a lower score can mean the model mentioned fewer visual entities rather than grounded better.Whether the measurement made the finding · reviewed by AI
- In claim-level hallucination detection pipelines over image, video and audio inputs (both comprehension and generation directions), decomposing an output into atomic claims contributes more to detection accuracy than ensembling several verifier models: removing decomposition costs about 6.6-7.9 macro-F1 across six cross-modal tasks while removing multi-expert majority voting costs about 4.5-5.3.Stating false facts confidently · reviewed by AI
- For response-level hallucination detection in retrieval-augmented settings, having the model first convert the reference documents into a relational SQL database and then check the response against SELECT query results raises precision and macro F1 over zero-shot direct prediction, over atomic-claim decomposition, and over flat JSON key-value extraction, without any task-specific fine-tuning.Stating false facts confidently · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: medium. Falsifier as drafted: Detectors trained on such synthetic splits show near-baseline flag rates on faithful LLM rewrites of truthful articles, i.e. their scores track veracity independently of text provenance. Proposed technique, not catalogued: Provenance-controlled evaluation split.