A Formal Theory of Survey Experiment Generalizability: Attention and Salience

Abstract

Survey experiments are widely used to identify causal effects in political science and the social sciences. Yet researchers are typically interested in more than the internal validity of an experimentally induced contrast. They also want to know whether the estimated effect corresponds to the effect in the real world. We develop a formal theory of survey experiment generalizability grounded in behavioral microfoundations. The theory highlights two mechanisms. First, the survey environment shapes attention: it determines which considerations enter the respondent's active consideration set. Second, it shapes salience: conditional on consideration, it influences the relative weight assigned to those considerations. This framework yields two main results. Consideration-set compression generates amplification: survey-experimental effects can be larger in magnitude than their real-world counterparts, even for the same individuals, treatment content, and outcome. Context-dependent salience generates sign instability: the direction of the survey effect need not coincide with the direction of the corresponding real-world effect. The theory clarifies what survey experiments identify, when those effects are likely to generalize, and how survey designs can be modified to improve decision-environment transportability.