arxiv-2602-18092 SRC-0347 · paper

Perceived Political Bias in LLMs Reduces Persuasive Abilities

Created: 2026-02-20 · Ingested: 2026-09-19

https://arxiv.org/abs/2602.18092(opens in a new tab)

Conversational AI has been proposed as a scalable way to correct public misconceptions and spread misinformation. Yet its effectiveness may depend on perceptions of its political neutrality. As LLMs enter partisan conflict, elites increasingly portray them as ideologically aligned. We test whether these credibility attacks reduce LLM-based persuasion. In a preregistered U.S. survey experiment (N=2144), participants completed a three-round conversation with ChatGPT about a personally held economic policy misconception. Compared to a neutral control, a short message indicating that the LLM was b

In brief

Telling a partisan that an LLM is biased against their side cuts how much the model can correct their beliefs, without eliminating persuasion. This puts source-credibility effects, long documented for human messengers, squarely onto conversational AI.

In a preregistered 4-arm US survey experiment (N=2144, Prolific, December 2025-January 2026), respondents held at least 1 of 6 economic misconceptions (household-budget analogy, rent control, immigration and jobs, trade deficits, tax cuts, "Buy American") and then had a 3-round conversation with gpt-4.1-2025-04-14 prompted to persuade them toward the academic-economist consensus while staying truthful. Arms: blank control, an undirected "AI may be biased" warning, and 2 out-party bias warnings, the heavier adding Sam Altman donation facts and a real photo with an out-party politician.

Mean pre-post change was -1.20 (SE=0.06) on a 0-4 scale in control, -0.93 (SE=0.06) light and -0.86 (SE=0.06) heavy: a 28% reduction in persuasion (16-40%) and 23% (9-35%). Full opinion reversals fell from 34.4% to 22.1%. Heavy treatment also lowered rated persuasiveness (d = -0.31) and trust in AI (d = -0.10).

Effect is measured, randomized, with a manipulation check (76%-80% recalled the cue correctly) and an undirected-bias arm separating directional from generic distrust. Transcript scaling via LLM-as-judge pairwise comparisons showed more words and more argumentativeness, not disengagement, favoring motivated reasoning.

Boundaries: 1 short interaction, 1 US online sample, 1 model family, 6 economic topics; topic-by-treatment interactions were not statistically significant.

Treat existing LLM persuasion estimates as upper bounds for politicized contexts.

Written from the abstract by claude-opus-5 on 2026-10-17, with every figure checked against it. Not a substitute for the paper.

Referenced by

Claims in this catalog that draw on this source, and whether as support or counterpoint.