arXiv subjects

Mattias Nordin

Publications and source records attributed to Mattias Nordin.

3 recordsLinked to original sources

arXiv2024

Precision-based designs for sequential randomized experiments

In this paper, we consider an experimental setting where units enter the experiment sequentially. Our goal is to form stopping rules which lead to estimators of treatment effects with a given precision. We propose a fixed-width confidence interval design (FWCID) where the experiment terminates once a pre-specified confidence interval width is achieved. We show that under this design, the difference-in-means estimator is a consistent estimator of the average treatment effect and standard confidence intervals have asymptotic guarantees of coverage and efficiency for several versions of the design. In addition, we propose a version of the design that we call fixed power design (FPD) where a given power is asymptotically guaranteed for a given treatment effect, without the need to specify the variances of the outcomes under treatment or control. In addition, this design also gives a consistent difference-in-means estimator with correct coverage of the corresponding standard confidence interval. We complement our theoretical findings with Monte Carlo simulations where we compare our proposed designs with standard designs in the sequential experiments literature, showing that our designs outperform these designs in several important aspects. We believe our results to be relevant for many experimental settings where units enter sequentially, such as in clinical trials, as well as in online A/B tests used by the tech and e-commerce industry.

math.ST↗

arXiv2021

Properties of restricted randomization with implications for experimental design

Recently, there as been an increasing interest in the use of heavily restricted randomization designs which enforces balance on observed covariates in randomized controlled trials. However, when restrictions are strict, there is a risk that the treatment effect estimator will have a very high mean squared error. In this paper, we formalize this risk and propose a novel combinatoric-based approach to describe and address this issue. First, we validate our new approach by re-proving some known properties of complete randomization and restricted randomization. Second, we propose a novel diagnostic measure for restricted designs that only use the information embedded in the combinatorics of the design. Third, we show that the variance of the mean squared error of the difference-in-means estimator in a randomized experiment is a linear function of this diagnostic measure. Finally, we identify situations in which restricted designs can lead to an increased risk of getting a high mean squared error and discuss how our diagnostic measure can be used to detect such designs. Our results have implications for any restricted randomization design and can be used to evaluate the trade-off between enforcing balance on observed covariates and avoiding too restrictive designs.

stat.ME↗

arXiv2021

Inference in experiments conditional on observed imbalances in covariates

Double blind randomized controlled trials are traditionally seen as the gold standard for causal inferences as the difference-in-means estimator is an unbiased estimator of the average treatment effect in the experiment. The fact that this estimator is unbiased over all possible randomizations does not, however, mean that any given estimate is close to the true treatment effect. Similarly, while pre-determined covariates will be balanced between treatment and control groups on average, large imbalances may be observed in a given experiment and the researcher may therefore want to condition on such covariates using linear regression. This paper studies the theoretical properties of both the difference-in-means and OLS estimators \emph{conditional} on observed differences in covariates. By deriving the statistical properties of the conditional estimators, we can establish guidance for how to deal with covariate imbalances. We study both inference with OLS, as well as with a new version of Fisher's exact test, where the randomization distribution comes from a small subset of all possible assignment vectors.

stat.ME↗