arXiv subjects

Dominic Coey

Publications and source records attributed to Dominic Coey.

3 recordsLinked to original sources

arXiv2024

Experimentation on Endogenous Graphs

We study experimentation under endogenous network interference. Interference patterns are mediated by an endogenous graph, where edges can be formed or eliminated as a result of treatment. We show that conventional estimators are biased in these circumstances, and present a class of unbiased, consistent and asymptotically normal estimators of total treatment effects in the presence of such interference. We show via simulation that our estimator outperforms existing estimators in the literature. Our results apply both to bipartite experimentation, in which the units of analysis and measurement differ, and the standard network experimentation case, in which they are the same.

stat.ME↗

arXiv2022

Empirical Bayes Selection for Value Maximization

We study the problem of selecting the best $m$ units from a set of $n$ as $m / n \to \alpha \in (0, 1)$, where noisy, heteroskedastic measurements of the units' true values are available and the decision-maker wishes to maximize the aggregate true value of the units selected. Given a parametric prior distribution, the empirical Bayes decision rule incurs $O_p(n^{-1})$ regret relative to the Bayesian oracle that knows the true prior. More generally, if the error in the estimated prior is of order $O_p(r_n)$, regret is $O_p(r_n^2)$. In this sense \emph{selection} of the best units is fundamentally easier than \emph{estimation} of their values. We show this regret bound is sharp in the parametric case, by giving an example in which it is attained. Using priors calibrated from a dataset of over four thousand internet experiments, we confirm that empirical Bayes methods perform well in detecting the best treatments with only a modest number of experiments.

stat.ME↗

arXiv2022

Machine Learning for Variance Reduction in Online Experiments

We consider the problem of variance reduction in randomized controlled trials, through the use of covariates correlated with the outcome but independent of the treatment. We propose a machine learning regression-adjusted treatment effect estimator, which we call MLRATE. MLRATE uses machine learning predictors of the outcome to reduce estimator variance. It employs cross-fitting to avoid overfitting biases, and we prove consistency and asymptotic normality under general conditions. MLRATE is robust to poor predictions from the machine learning step: if the predictions are uncorrelated with the outcomes, the estimator performs asymptotically no worse than the standard difference-in-means estimator, while if predictions are highly correlated with outcomes, the efficiency gains are large. In A/A tests, for a set of 48 outcome metrics commonly monitored in Facebook experiments the estimator has over 70% lower variance than the simple difference-in-means estimator, and about 19% lower variance than the common univariate procedure which adjusts only for pre-experiment values of the outcome.

stat.ML↗