arxiv-2403-14380 SRC-0063 · paper

On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial

Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti, Robert West

Created: 2024-03-21 · Ingested: 2026-09-18

The development and popularization of large language models (LLMs) have raised concerns that they will be used to create tailor-made, convincing arguments to push false or misleading narratives online. Early work has found that language models can generate content perceived as at least on par and often more persuasive than human-written messages. However, there is still limited knowledge about LLMs' persuasive capabilities in direct conversations with human counterparts and how personalization can improve their performance. In this pre-registered study, we analyze the effect of AI-driven persuasion in a controlled, harmless setting. We create a web-based platform where participants engage in short, multiple-round debates with a live opponent. Each participant is randomly assigned to one of four treatment conditions, corresponding to a two-by-two factorial design: (1) Games are either…

In brief

GPT-4 given 6 basic demographic facts about its opponent out-persuades human debaters, and out-persuades humans given the same facts by a wider margin still.

Participants recruited on Prolific were randomly paired with either a human or gpt-4-0613 for a timed 10-minute, 3-stage debate (opening, rebuttal, conclusion) on 1 of 30 pre-screened propositions, with a randomly assigned PRO or CON side. A 2x2 design crossed opponent type with whether one side received the opponent's anonymized survey data (gender, age, ethnicity, education, employment, political affiliation). Agreement with the proposition was measured on a 1-5 Likert scale before and after.

Personalized GPT-4 raised the odds of shifting toward the opponent by 81.7% ([+26.3%, +161.4%], p < 0.01) versus human-human; unpersonalized GPT-4 gave +21.3% ([-16.7%, +76.6%], p = 0.31). Personalized humans did worse than the baseline (-17.4%, p = 0.38). Debates mostly backfired overall: mean agreement shift was -0.22 for human-human, and only personalized GPT-4 was positive at 0.14.

Pre-registered, randomized, 150 debates per condition (110 for human-human personalized), N = 820, ordinal partial proportional odds model with cluster-robust errors, and demographic controls that left treatment effects unchanged. Only the personalized-AI arm is significant; the plain AI advantage is not.

Boundaries: 1 model, 1 prompt, US participants, forced sides and rigid timing, and participants identified the AI in about 3 out of 4 AI debates.

Treat microtargeting, not raw argument quality, as the load-bearing variable in LLM persuasion.

Written from the abstract by claude-opus-5 on 2026-09-18, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.

No claims cite this yet.