arXiv subjects

Drew Dimmery

Publications and source records attributed to Drew Dimmery.

13 recordsLinked to original sources

arXiv2025

Enough?

We respond to Aronow et al. (2025)'s paper arguing that randomized controlled trials (RCTs) are "enough," while nonparametric identification in observational studies is not. We agree with their position with respect to experimental versus observational research, but question what it would mean to extend this logic to the scientific enterprise more broadly. We first investigate what is meant by "enough," arguing that this is fundamentally a sociological claim about the relationship between statistical work and larger social and institutional processes, rather than something that can be decided from within the logic of statistics. For a more complete conception of "enough," we outline all that would need to be known -- not just knowledge of propensity scores, but knowledge of many other spatial and temporal characteristics of the social world. Even granting the logic of the critique in Aronow et al. (2025), its practical importance is a question of the contexts under study. We argue that we should not be satisfied by appeals to intuition about the complexity of "naturally occurring" propensity score functions. Instead, we call for more empirical metascience to begin to characterize this complexity. We apply this logic to the example of recommender systems developed by Aronow et al. (2025) as a demonstration of the weakness of allowing statisticians' intuitions to serve in place of metascientific data. Rather than implicitly deciding what is "enough" based on statistical applications the social world has determined to be most profitable, we argue that practicing statisticians should explicitly engage with questions like "for what?" and "for whom?" in order to adequately answer the question of "enough?"

stat.ME↗

arXiv2023

Subjective visualization experiences: impact of visual design and experimental design

In contrast to objectively measurable aspects (such as accuracy, reading speed, or memorability), the subjective experience of visualizations has only recently gained importance, and we have less experience how to measure it. We explore how subjective experience is affected by chart design using multiple experimental methods. We measure the effects of changes in color, orientation, and source annotation on the perceived readability and trustworthiness of simple bar charts. Three different experimental designs (single image rating, forced choice comparison, and semi-structured interviews) provide similar but different results. We find that these subjective experiences are different from what prior work on objective dimensions would predict. Seemingly inconsequential choices, like orientation, have large effects for some methods, indicating that study design alters decision-making strategies. Next to insights into the effect of chart design, we provide methodological insights, such as a suggested need to carefully isolate individual elements in charts to study subjective experiences.

cs.HC↗

arXiv2022

Interpretable Personalized Experimentation

Black-box heterogeneous treatment effect (HTE) models are increasingly being used to create personalized policies that assign individuals to their optimal treatments. However, they are difficult to understand, and can be burdensome to maintain in a production environment. In this paper, we present a scalable, interpretable personalized experimentation system, implemented and deployed in production at Meta. The system works in a multiple treatment, multiple outcome setting typical at Meta to: (1) learn explanations for black-box HTE models; (2) generate interpretable personalized policies. We evaluate the methods used in the system on publicly available data and Meta use cases, and discuss lessons learnt during the development of the system.

cs.LG↗

arXiv2022

Online Balanced Experimental Design

e consider the experimental design problem in an online environment, an important practical task for reducing the variance of estimates in randomized experiments which allows for greater precision, and in turn, improved decision making. In this work, we present algorithms that build on recent advances in online discrepancy minimization which accommodate both arbitrary treatment probabilities and multiple treatments. The proposed algorithms are computational efficient, minimize covariate imbalance, and include randomization which enables robustness to misspecification. We provide worst case bounds on the expected mean squared error of the causal estimate and show that the proposed estimator is no worse than an implicit ridge regression, which are within a logarithmic factor of the best known results for offline experimental design. We conclude with a detailed simulation study showing favorable results relative to complete randomization as well as to offline methods for experimental design with time complexities exceeding our algorithm.

stat.ME↗

arXiv2021

Designing Transportable Experiments

We consider the problem of designing a randomized experiment on a source population to estimate the Average Treatment Effect (ATE) on a target population. We propose a novel approach which explicitly considers the target when designing the experiment on the source. Under the covariate shift assumption, we design an unbiased importance-weighted estimator for the target population's ATE. To reduce the variance of our estimator, we design a covariate balance condition (Target Balance) between the treatment and control groups based on the target population. We show that Target Balance achieves a higher variance reduction asymptotically than methods that do not consider the target population during the design phase. Our experiments illustrate that Target Balance reduces the variance even for small sample sizes.

stat.ME↗

arXiv2021

Online Discrepancy Minimization via Persistent Self-Balancing Walks

We study the online discrepancy minimization problem for vectors in $\mathbb{R}^d$ in the oblivious setting where an adversary is allowed fix the vectors $x_1, x_2, \ldots, x_n$ in arbitrary order ahead of time. We give an algorithm that maintains $O(\sqrt{\log(nd/δ)})$ discrepancy with probability $1-δ$, matching the lower bound given in [Bansal et al. 2020] up to an $O(\sqrt{\log \log n})$ factor in the high-probability regime. We also provide results for the weighted and multi-color versions of the problem.

cs.DS↗

arXiv2021

Designing Multi-Arm Experiments for Global Average Treatment Effects Under Partial Interference

When interference is present, a unit's outcome depends on others' assignments and there is generally no single, design-free average treatment effect. We study settings where the decision problem is to choose among several alternative uniform policies---e.g., rolling out one platform configuration to all users, or one policy to all constituents---so the estimand of interest is the Global Average Treatment Effect (GATE), the average outcome difference under two uniform (global) policies. Under partial interference with clusters, cluster-level randomization identifies the GATE but can be statistically inefficient, while unit-level randomization can be highly precise yet biased for the GATE. We propose a continuum of implementable two-stage randomization designs that smoothly interpolate between these extremes by tuning within-cluster treatment correlation. For multi-arm experiments, we operationalize this continuum via a Dirichlet--multinomial design and give a pilot- and model-assisted procedure for selecting the design parameter using estimated finite-sample RMSE for the GATE. We also show that, even under a correctly specified linear interference model, difference-in-means estimators can have lower RMSE than least squares regression for GATE targets.Simulations and a large-scale Facebook video-player configuration experiment (43 million user sessions) illustrate the practical trade-off: intermediate designs can substantially reduce RMSE for estimating global rollout effects while remaining straightforward to deploy at scale.

stat.ME↗

arXiv2020

Efficient Balanced Treatment Assignments for Experimentation

In this work, we reframe the problem of balanced treatment assignment as optimization of a two-sample test between test and control units. Using this lens we provide an assignment algorithm that is optimal with respect to the minimum spanning tree test of Friedman and Rafsky (1979). This assignment to treatment groups may be performed exactly in polynomial time. We provide a probabilistic interpretation of this process in terms of the most probable element of designs drawn from a determinantal point process which admits a probabilistic interpretation of the design. We provide a novel formulation of estimation as transductive inference and show how the tree structures used in design can also be used in an adjustment estimator. We conclude with a simulation study demonstrating the improved efficacy of our method.

stat.ME↗

arXiv2020

Real-world Video Adaptation with Reinforcement Learning

Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). We evaluate recently proposed RL-based ABR methods in Facebook's web-based video streaming platform. Real-world ABR contains several challenges that requires customized designs beyond off-the-shelf RL algorithms -- we implement a scalable neural network architecture that supports videos with arbitrary bitrate encodings; we design a training method to cope with the variance resulting from the stochasticity in network conditions; and we leverage constrained Bayesian optimization for reward shaping in order to optimize the conflicting QoE objectives. In a week-long worldwide deployment with more than 30 million video streaming sessions, our RL approach outperforms the existing human-engineered ABR algorithms.

cs.NI↗

arXiv2020

Permutation Weighting

In observational causal inference, in order to emulate a randomized experiment, weights are used to render treatments independent of observed covariates. This property is known as balance; in its absence, estimated causal effects may be arbitrarily biased. In this work we introduce permutation weighting, a method for estimating balancing weights using a standard binary classifier (regardless of cardinality of treatment). A large class of probabilistic classifiers may be used in this method; the choice of loss for the classifier implies the particular definition of balance. We bound bias and variance in terms of the excess risk of the classifier, show that these disappear asymptotically, and demonstrate that our classification problem directly minimizes imbalance. A wide variety of existing balancing weights may be estimated through this regime, allowing for direct comparison between methods based on classifier loss, as well as hyper-parameter tuning using cross-validation. Empirical evaluations indicate that permutation weighting provides favorable performance in comparison to existing methods.

stat.ME↗

arXiv2020

Balanced off-policy evaluation in general action spaces

Estimation of importance sampling weights for off-policy evaluation of contextual bandits often results in imbalance - a mismatch between the desired and the actual distribution of state-action pairs after weighting. In this work we present balanced off-policy evaluation (B-OPE), a generic method for estimating weights which minimize this imbalance. Estimation of these weights reduces to a binary classification problem regardless of action type. We show that minimizing the risk of the classifier implies minimization of imbalance to the desired counterfactual distribution of state-action pairs. The classifier loss is tied to the error of the off-policy estimate, allowing for easy tuning of hyperparameters. We provide experimental evidence that B-OPE improves weighting-based approaches for offline policy evaluation in both discrete and continuous action spaces.

cs.LG↗

arXiv2019

Thompson Sampling for Contextual Bandit Problems with Auxiliary Safety Constraints

Recent advances in contextual bandit optimization and reinforcement learning have garnered interest in applying these methods to real-world sequential decision making problems. Real-world applications frequently have constraints with respect to a currently deployed policy. Many of the existing constraint-aware algorithms consider problems with a single objective (the reward) and a constraint on the reward with respect to a baseline policy. However, many important applications involve multiple competing objectives and auxiliary constraints. In this paper, we propose a novel Thompson sampling algorithm for multi-outcome contextual bandit problems with auxiliary constraints. We empirically evaluate our algorithm on a synthetic problem. Lastly, we apply our method to a real world video transcoding problem and provide a practical way for navigating the trade-off between safety and performance using Bayesian optimization.

cs.LG↗

arXiv2019

Shrinkage Estimators in Online Experiments

We develop and analyze empirical Bayes Stein-type estimators for use in the estimation of causal effects in large-scale online experiments. While online experiments are generally thought to be distinguished by their large sample size, we focus on the multiplicity of treatment groups. The typical analysis practice is to use simple differences-in-means (perhaps with covariate adjustment) as if all treatment arms were independent. In this work we develop consistent, small bias, shrinkage estimators for this setting. In addition to achieving lower mean squared error these estimators retain important frequentist properties such as coverage under most reasonable scenarios. Modern sequential methods of experimentation and optimization such as multi-armed bandit optimization (where treatment allocations adapt over time to prior responses) benefit from the use of our shrinkage estimators. Exploration under empirical Bayes focuses more efficiently on near-optimal arms, improving the resulting decisions made under uncertainty. We demonstrate these properties by examining seventeen large-scale experiments conducted on Facebook from April to June 2017.

stat.ME↗