Using Prior Studies to Design Experiments: An Empirical Bayes Approach

Abstract

We develop an empirical Bayes framework for experimental design that leverages information from prior related studies. When a researcher has access to estimates from previous studies on similar parameters, they can use empirical Bayes to estimate an informative prior over the parameter of interest in the new study. We show how this prior can be incorporated into a decision-theoretic experimental design framework to choose optimal design. The approach is illustrated via propensity score designs in stratified randomized experiments. Our theoretical results show that the empirical Bayes design achieves oracle-optimal performance as the number of prior studies grows, and characterize the rate at which regret vanishes. To illustrate the approach, we present two empirical applications--oncology drug trials and the Tennessee Project STAR experiment. Our framework connects the Bayesian meta-analysis literature to experimental design and provides practical guidance for researchers seeking to design more efficient experiments.