Transporting causal effects from a randomized trial without "transportability:" a case study of political advertising during U.S. elections

Abstract

During the 2020 U.S. presidential election, Aggarwal et al. (2023) conducted a large-scale randomized experiment to evaluate a digital ad campaign against Trump in five battleground states. While the study found no effect on voter turnout, it's unclear whether this null result generalizes to other battleground states, notably Georgia, which played a unique role in the 2020 election and differs from the battleground states. Inspired by the study, we present a transfer learning framework to estimate treatment effects in a target population (e.g., Georgia) based on a randomized experiment from a source population (e.g., the five battleground states). Our framework is based on a sensitivity analysis that allows for violation of transportability, a popular yet impractical assumption which requires all differences between the source and target populations to be characterized by observed variables. Under our framework, we propose two estimators of the target treatment effect: a simple regression estimator with bootstrap, which we recommend for practitioners in this field, and an estimator based on the efficient influence function. Importantly, both estimators allow for covariates to differ between the target and the source populations, another common scenario in practice. We also propose a new, sample splitting approach to calibrate the sensitivity parameter. We apply our framework to estimate the effect of the ad campaign on voter turnout in Georgia during the 2020 election. Our findings indicate that small departures from transportability can lead to dramatically different ad effects across counties of Georgia. The direction of the effects is largely driven by racial composition: counties with higher White and lower Black percents tend to show positive effects, while counties with higher Latinx percents tend to show negative effects.