When Does Interference Matter? Decision-Making in Platform Experiments
This paper investigates decision-making in A/B experiments for online platforms and marketplaces. In such settings, due to constraints on inventory, A/B experiments typically lead to biased estimators because of *interference* between treatment and control groups; this phenomenon has been well studied in recent literature. By contrast, there has been relatively little discussion of the impact of interference on decision-making. In this paper, we analyze a benchmark Markovian model of an inventory-constrained platform, where arriving customers book listings that are limited in supply. We focus on the commonly used frequentist hypothesis testing approach for making launch decisions based on data from customer-randomized experiments, and we study the impact of interference on (1) false positive probability and (2) statistical power. We obtain three main findings. First, we show that for *sign-consistent* treatments -- i.e., those where the treatment changes booking probabilities in the same direction relative to control for all states of inventory availability -- the false positive probability of a test statistic using the standard difference-in-means estimator with a corresponding na\"ive variance estimator is correctly controlled. Second, we demonstrate that for sign-consistent treatments in realistic settings, the statistical power of this na\"ive approach is higher than that of any similar pipeline using a debiased estimator. Taken together, these two findings suggest that platforms may be better off *not* debiasing when treatments are sign-consistent. Third, using numerics, we investigate false positive probability and statistical power when treatments are sign-inconsistent, and we show that in principle, the performance of the na\"ive approach can be arbitrarily worse in such cases.