What is the Causal Effect of a Conversation? Estimands and Inference in AI Mediated Conversations

Abstract

Political scientists increasingly use conversations as treatments. Sometimes these conversations are conducted by humans, but often and increasingly they will be conducted by generative artificial intelligence (AI). AI makes it possible to scale treatments that are responsive, and rich in real-world relevance. But this same interactivity creates specific challenges for causal inference. A respondent may be randomly assigned to a conversational condition, but the conversation that follows is not merely received by the respondent. It is generated jointly by the respondent and the conversational agent and is thus endogenous to who the respondent is. Consequently, when the theoretical object of interest is the conversation itself -- or particular messages, features, or other attributes of that conversation -- randomization of assignment does not necessarily identify the causal quantity the researcher seeks to estimate. This paper develops a potential outcomes framework for causal inference with conversations, with broad application but particular relevance to AI-mediated interaction. We distinguish among several causal objects: assignment to a conversational condition, assignment to a conversational policy, opening messages, messages within a conversation, realized conversational features, and the full realized conversation. Each corresponds to a distinct estimand and a different set of identifying assumptions. While some of these quantities are identified by standard randomized designs, others require additional assumptions or research designs, including sequential assumptions, representations of conversational histories, or explicit message-level interventions. The framework clarifies these distinctions and provides a common language for defining, interpreting, and designing conversational experiments as conversations increasingly become objects of causal inquiry.