CUPED on Steroids: Multivariate Covariate Adjustment for Switchback Experiments

Abstract

Controlled-experiment Using Pre-Experiment Data (CUPED) reduces the variance of the treatment effect estimator in online experiments by adjusting the in-experiment outcome metric using its lagged pre-experiment value. This method can be strengthened by enriching its covariate set while keeping it automatable and guarding against overfitting and covariate leakage. In switchback and related clustered experiment designs, this enrichment may be especially fruitful since randomization-unit-level variation in the outcome amplifies the variance of the treatment effect estimator, so covariates that explain it yield disproportionately large efficiency gains. We develop an extended CUPED framework for switchback and related clustered designs that introduces sizeable improvement in variance reduction efficiency relative to conventional CUPED while remaining lightweight and free of assumptions beyond those already implicit in CUPED. The framework combines multiple historical lags with cyclic hour-of-day encodings and cluster- and hour-level fixed-effect dummies in the pre-period prediction model---covariates that target randomization-unit-level outcome variation---and uses cross-fitting so that realized variance reduction is not inflated by in-sample overfitting. In simulation, we analyze how each component affects validity and efficiency of this method and show that the full framework yields large gains in statistical power. We further validate the approach on publicly available trip records from the New York City Taxi and Limousine Commission. Finally, we derive a closed-form ceiling on the variance reduction attainable by any covariate adjustment and connect our empirical findings to the composition-dependent limits this ceiling imposes.