The Political Ideology of Large Language Models: Measurement, Inconsistency, and Persuasive Influence
Large Language Models (LLMs) are a transformational technology, fundamentally changing how people obtain information and interact with the world. As people become increasingly reliant on them for an enormous variety of tasks, a body of academic research has developed to examine these models for inherent biases, especially political biases, often finding them small. We challenge this prevailing wisdom. First, by comparing 43 LLMs to legislators, judges, and a nationally representative sample of U.S. voters, we show that LLMs' apparently moderate overall partisan positioning is the net result of offsetting strongly partisan expressed positions on specific topics, much like moderate voters. Second, in a pre-registered randomized experiment, we show that LLMs can exert persuasive influence on political attitudes. Voters randomized to discuss a policy issue with an LLM shift toward that model's measured ideological position by 3.5 percentage points on average, an effect at least as large as those produced by professional campaign advertising. Explicitly prompting a model to argue one side of the issue shifts attitudes by more than 10 percentage points relative to unsteered conversations, and this steering accounts for the pooled effect. When the same models converse naturally, without steering, we detect no persuasive effect, and our confidence interval rules out effects as small as the pre-registered smallest effect of interest. Contrary to expectations, these persuasive effects are not moderated by familiarity with LLMs, news consumption, or interest in politics. LLMs, especially those controlled by private companies or governments, may become a powerful and targeted vector for political influence.