arXiv subjects

Apoorva Lal

Publications and source records attributed to Apoorva Lal.

11 recordsLinked to original sources

arXiv2026

AI-Assisted Variance Reduction in Randomized Experiments

Generative AI and large language models can produce realistic predictions of human behavior from rich, unstructured inputs with little to no task-specific training data. Recent work uses these ``digital twin'' predictions to supplement human responses in surveys and experiments. We study the special case of using AI-generated predictions to reduce variance in randomized experiments. We argue that doing so requires no new estimators and that researchers can simply include AI predictions as covariates in standard regression adjustment, analogous to adjusting for a prognostic score. A benefit of this approach is a ``do no harm'' property whereby the adjusted estimator reverts to the unadjusted difference in means when predictions are uninformative. Other methods, such as variants of prediction-powered inference, do not have this guarantee. We provide implementation guidance, including how to obtain continuous scores from discrete LLM outputs and how to use LLMs to featurize unstructured inputs as auxiliary covariates. We demonstrate these ideas in simulations and three empirical applications: a survey mega-study, an email marketing A/B test, and a large-scale technology platform experiment. Overall, efficiency gains are real if modest, with greater benefits in studies that contain substantial text and other unstructured data. We also confirm the do no harm property empirically. Given these gains and limited costs, we recommend adjusting for AI-generated predictions as a regular empirical practice.

econ.EM↗

arXiv2026

Long-Term Causal Inference with Many Noisy Proxies

We propose a method for estimating long-term treatment effects with many short-term proxy outcomes: a central challenge when experimenting on digital platforms. We formalize this challenge as a latent variable problem where observed proxies are noisy measures of a low-dimensional set of unobserved surrogates that mediate treatment effects. Through theoretical analysis and simulations, we demonstrate that regularized regression methods substantially outperform naive proxy selection. We show in particular that the bias of Ridge regression decreases as more proxies are added, with closed-form expressions for the bias-variance tradeoff. We illustrate our method with an empirical application to the California GAIN experiment.

econ.EM↗

arXiv2025

Estimating Representative Causal Effects with Double Machine Learning

Double Machine Learning is widely used to estimate treatment effects from non-experimental data. The "residuals-on-residuals" regression (RORR) is especially popular for its simplicity and computational tractability. However, with heterogeneous treatment effects, the proper interpretation of RORR may not be well understood. We show that, for non-binary treatments with continuous dose-response functions, RORR estimates a conditional variance-weighted average of derivatives evaluated at treatment values not in the observed dataset. This estimand does not equal the Average Causal Derivative (ACD) in general. Hence, even if all units share the same dose-response function, RORR does not estimate an average treatment effect in the population represented by the sample. We propose an alternative estimator for the ACD that is well suited to the large datasets found in applied data science settings. We demonstrate the pitfalls of RORR and the favorable properties of the proposed estimator through an illustrative numerical example and with real-world data from Netflix. Our methodology is used by default in Netflix's observational causal inference platform, where it regularly powers causal research and decision-making at scale.

econ.EM↗

arXiv2025

When can we get away with using the two-way fixed effects regression?

The use of the two-way fixed effects regression in empirical social science was historically motivated by folk wisdom that it uncovers the Average Treatment effect on the Treated (ATT) as in the canonical two-period two-group case. This belief has come under scrutiny recently due to recent results in applied econometrics showing that it fails to uncover meaningful averages of heterogeneous treatment effects in the presence of effect heterogeneity over time and across adoption cohorts, and several heterogeneity-robust alternatives have been proposed. However, these estimators often have higher variance and are therefore under-powered for many applications, which poses a bias-variance tradeoff that is challenging for researchers to navigate. In this paper, we propose simple tests of linear restrictions that can be used to test for differences in dynamic treatment effects over cohorts, which allows us to test for when the two-way fixed effects regression is likely to yield biased estimates of the ATT. These tests are implemented as methods in the pyfixest python library.

econ.EM↗

arXiv2024

Optimizing Returns from Experimentation Programs

Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem (Azevedo et al., 2020) as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning $p$-value thresholds. Furthermore, we develop several extensions of the A/B Testing Problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing $p$-value thresholds away from $p = 0.05$.

stat.ME↗

arXiv2024

Does Regression Produce Representative Causal Rankings?

We examine the challenges in ranking multiple treatments based on their estimated effects when using linear regression or its popular double-machine-learning variant, the Partially Linear Model (PLM), in the presence of treatment effect heterogeneity. We demonstrate by example that overlap-weighting performed by linear models like PLM can produce Weighted Average Treatment Effects (WATE) that have rankings that are inconsistent with the rankings of the underlying Average Treatment Effects (ATE). We define this as ranking reversals and derive a necessary and sufficient condition for ranking reversals under the PLM. We conclude with several simulation studies conditions under which ranking reversals occur.

econ.EM↗

arXiv2024

Large Scale Longitudinal Experiments: Estimation and Inference

Large-scale randomized experiments are seldom analyzed using panel regression methods because of computational challenges arising from the presence of millions of nuisance parameters. We leverage Mundlak's insight that unit intercepts can be eliminated by using carefully chosen averages of the regressors to rewrite several common estimators in a form that is amenable to weighted-least squares estimation with frequency weights. This renders regressions involving arbitrary strata intercepts tractable with very large datasets, optionally with the key compression step computed out-of-memory in SQL. We demonstrate that these methods yield more precise estimates than other commonly used estimators, and also find that the compression strategy greatly increases computational efficiency. We provide in-memory (pyfixest) and out-of-memory (duckreg) python libraries to implement these estimators.

econ.EM↗

arXiv2024

Nonparametric Jackknife Instrumental Variable Estimation and Confounding Robust Surrogate Indices

Jackknife instrumental variable estimation (JIVE) is a classic method to leverage many weak instrumental variables (IVs) to estimate linear structural models, overcoming the bias of standard methods like two-stage least squares. In this paper, we extend the jackknife approach to nonparametric IV (NPIV) models with many weak IVs. Since NPIV characterizes the structural regression as having residuals projected onto the IV being zero, existing approaches minimize an estimate of the average squared projected residuals, but their estimates are biased under many weak IVs. We introduce an IV splitting device inspired by JIVE to remove this bias, and by carefully studying this split-IV empirical process we establish learning rates that depend on generic complexity measures of the nonparametric hypothesis class. We then turn to leveraging this for semiparametric inference on average treatment effects (ATEs) on unobserved long-term outcomes predicted from short-term surrogates, using historical experiments as IVs to learn this nonparametric predictive relationship even in the presence of confounding between short- and long-term observations. Using split-IV estimates of a debiasing nuisance, we develop asymptotically normal estimates for predicted ATEs, enabling inference.

math.ST↗

arXiv2023

How Much Should We Trust Instrumental Variable Estimates in Political Science? Practical Advice Based on Over 60 Replicated Studies

Instrumental variable (IV) strategies are widely used in political science to establish causal relationships. However, the identifying assumptions required by an IV design are demanding, and it remains challenging for researchers to assess their validity. In this paper, we replicate 67 papers published in three top journals in political science during 2010-2022 and identify several troubling patterns. First, researchers often overestimate the strength of their IVs due to non-i.i.d. errors, such as a clustering structure. Second, the most commonly used t-test for the two-stage-least-squares (2SLS) estimates often severely underestimates uncertainty. Using more robust inferential methods, we find that around 19-30% of the 2SLS estimates in our sample are underpowered. Third, in the majority of the replicated studies, the 2SLS estimates are much larger than the ordinary-least-squares estimates, and their ratio is negatively correlated with the strength of the IVs in studies where the IVs are not experimentally generated, suggesting potential violations of unconfoundedness or the exclusion restriction. To help researchers avoid these pitfalls, we provide a checklist for better practice.

econ.EM↗

arXiv2023

Did Private Election Administration Funding Advantage Democrats in 2020?

Private donors contributed more than $350 million to local election officials to support the administration of the 2020 election. Supporters argue these grants were neutral and necessary to maintain normal election operations during the pandemic, while critics worry these grants mostly went to Democratic strongholds and tilted election outcomes. These concerns have led twenty-four states to restrict private election grants. How much did these grants shape the 2020 presidential election? To answer this question, we collect administrative data on private election administration grants and election outcomes. We then use new advances in synthetic control methods to compare presidential election results and turnout in counties that received grants to counties with identical average presidential election results and turnout before 2020. While counties that favor Democrats were much more likely to apply for a grant, we find that the grants did not have a noticeable effect on the presidential election. Our estimates of the average effect of receiving a grant on Democratic vote share range from 0.02 percentage points to 0.36 percentage points. Our estimates of the average effect of receiving a grant on turnout range from -0.03 percentage points to 0.13 percentage points. Across specifications, our 95% confidence intervals typically include negative effects, and our confidence intervals from all specifications fail to include effects on Democratic vote share larger than 0.58 percentage points and effects on turnout larger than 0.40 percentage points. We characterize the magnitude of our effects by asking how large they are compared to the margin by which Biden won the 2020 election. In simple bench-marking exercises, we find that the effects of the grants were likely too small to have changed the outcome of the 2020 presidential election.

econ.GN↗

arXiv2023

A Framework for Generalization and Transportation of Causal Estimates Under Covariate Shift

Randomized experiments are an excellent tool for estimating internally valid causal effects with the sample at hand, but their external validity is frequently debated. While classical results on the estimation of Population Average Treatment Effects (PATE) implicitly assume random selection into experiments, this is typically far from true in many medical, social-scientific, and industry experiments. When the experimental sample is different from the target sample along observable or unobservable dimensions, experimental estimates may be of limited use for policy decisions. We begin by decomposing the extrapolation bias from estimating the Target Average Treatment Effect (TATE) using the Sample Average Treatment Effect (SATE) into covariate shift, overlap, and effect modification components, which researchers can reason about in order to diagnose the severity of extrapolation bias. Next, We cast covariate shift as a sample selection problem and propose estimators that re-weight the doubly-robust scores from experimental subjects to estimate treatment effects in the overall sample (=: generalization) or in an alternate target sample (=: transportation). We implement these estimators in the open-source R package causalTransportR and illustrate its performance in a simulation study and discuss diagnostics to evaluate its performance.

stat.ME↗