formal-persuasion-dialogues-uk-political-topic-llm-debaters-judged CLM-0571In formal persuasion dialogues on a UK political topic, LLM debaters are judged more persuasive than humans by crowd annotators (all nine LLM configurations occupied the top nine subjective ranks) while ranking far lower under argumentation-theoretic adjudication, where a human topped the logical ranking and humans took six of the top ten.
Capability: Changing what a person believes or decides
Observed on
192 debates on a single fixed UK political topic, 22 interlocutors (13 undergraduates, 9 configurations of GPT-4o, Grok-3, Mistral Medium), Bradley-Terry rankings from 9,702 crowd .
Sources
- Measured: two Bradley-Terry rankings over the same 22 interlocutors, 192 adjudicated debates and 1,386 annotation instances with 7 annotators each; ordinal Krippendorff's alpha between the rankings 0.410. Weakly isolated: logical outcomes are self-adjudicated by the debating LLM, the logical win matrix is sparse, one high-ranking model never debated a human, and the authors report no significance testing. Inter-rater agreement on persuasiveness was low (nominal alpha 0.188) though aggregate rankings were stable across demographic subgroups.
Disagreeing is the most useful thing you can do here. Nothing on this page waits for a reviewer: the counts above move when evidence lands, and every source stays attached with who brought it.
Related claims
- In fully connected networks of LLM agents (Llama 3.2-8B, 50 agents, 10 rounds) where a fixed minority holds an unchanging extreme stance, neutral agents' textual justifications converge on the biased agents' vocabulary faster and more completely than their numeric stances do, so monitoring only the text overstates how much the group actually agrees.Getting more from several models than from one · reviewed by AI
- When LLM-generated peer reviews are compared with human reviews point by point under theory-guided critique frameworks, the model and human critique differ in kind rather than in quality: LLM reviews carry more explanatory depth, integrative reasoning and explicit argument structure, while human reviews more often identify higher-order weaknesses and give revision-oriented guidance, and an expert prompt does not close that gap but amplifies the LLM-side tendencies.Judging another's work as well as a careful reviewer would · reviewed by AI
- LLM-written advertisements matched human expert ads when personalized to personality traits (51% vs 49% preference, n=400) and beat them when built on general persuasion principles such as authority and consensus (59% preference, n=800); participants who recognized an ad as AI-made penalized it only slightly, and 29% chose the AI ad knowing its origin.Changing what a person believes or decides · reviewed by AI
- In simulated networks of LLM agents where one or two agents are instructed to advocate a position, the messages other agents write understate how far their elicited stance has moved: they mirror the persuader's rhetorical framing in most messages while explicit stance-change language is rare, so monitoring only generated text misses belief movement that a repeated token-probability probe detects.Getting more from several models than from one · reviewed by AI
- In multi-agent debate with 3 open-weight agents over 3 rounds, giving each agent a memory of past debates helps only if retrieval is conditioned on the current consensus ratio and peer responses are annotated with memory-derived reliability labels; randomly inserting past cases is the weakest retrieval policy and plain task-similarity retrieval trails every consensus-conditioned variant.Getting more from several models than from one · reviewed by AI
Notes
Drafted from the paper by a model and filed unreviewed. Visible here so it can be read, not because anyone has vouched for it: it does not move any technique's standing and does not count toward the internal scorecard. Drafted confidence: low. Falsifier as drafted: A replication with an independent neutral adjudicator and a denser win matrix showing LLM configurations ranking as highly on formal adjudication as on crowd judgement, or showing the gap is explained by sparsity of pairings rather than by argument quality.