deception-narrative-text-analysed-instantiating-fixed-argumentation-schemes-llm CLM-0372When deception in narrative text is analysed by instantiating fixed argumentation schemes with an LLM (premise extraction, then critical-question evaluation), the scheme assessments are stable across ten different LLM backbones and repeated runs for clear-cut deception cases but grow inconsistent on ambiguous intelligence-style narratives.
Capability: Strategic deception and detecting it
Observed on
Narrative/PDF text analysed with a fixed ten-scheme library; reliability measured as cross-model and repeated-run consistency, not classification accuracy against ground truth (no .
Sources
- Validated by: Not measured against ground truth; validated only by consistency of premise and conclusion assessments across ten LLMs and repeated runs, plus an illustrative case analysis.. Argued plus a reliability study reported in summary form: consistency across ten contemporary LLMs and repeated runs on illustrative narratives. No accuracy baseline, no ground-truth corpus, no stated agreement statistics in the available text, so effect size and the definition of 'ambiguous' are unverified.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In four experiments with 3,996 US participants discussing a conspiracy they were unsure about, frontier models argued people into as well as out of belief with no consistent truth advantage: debunking cut belief by about 12 points on a 100-point scale and bunking raised it comparably, and participants rated the bunking model as more informative and trustworthy; the one asymmetry was that debunking produced more large (40+ point) shifts (16% vs 7%), and constraining the bunking model to true statements retained only 57-64% of its effect.Changing what a person believes or decides · reviewed by AI
- On questions whose premises are false or internally contradictory, prompting mid-size instruct models (Qwen2.5-7B, LLaMA-3.1-8B, Gemma3-12B) to reason step by step can lower accuracy below plain answering, because the reasoning chain elaborates from the flawed premise instead of challenging it.Stating false facts confidently · reviewed by AI
- General-purpose factuality verifiers (entailment-, consistency-, and RAG-trained) and zero-shot frontier LLM judges perform near chance at spotting unsupported claims in scientific peer reviews that must be checked against the full submitted paper, because they cannot separate paper-unsupported assertions from legitimate evaluative critique; small models fine-tuned on in-domain examples beat them by a wide margin.Stating false facts confidently · reviewed by AI
- When an 8B model verifies natural-language claims against knowledge-graph triples, scoring the evidence twice under opposing prompts (argue it supports, argue it refutes) and adjudicating between the two arguments avoids the bias toward a "Supported" verdict that a single-perspective assessment shows, and is the largest single contributor to accuracy in that pipeline.Checking claims against evidence · reviewed by AI
- When a prompt contains several legitimate but conflicting evidence passages, models internally register the conflict — linear probes on mid-to-late-layer hidden states separate conflicting from consistent inputs at high AUC — while still resolving the conflict badly at the output, so the failure is in evidence integration rather than in conflict detection.Checking claims against evidence · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A replication showing premise and conclusion assessments from the same schemes disagree substantially across models or runs even on clear-cut deception cases, or that ambiguity of the narrative does not predict inconsistency.