arXiv subjects

Martin Tingley

Publications and source records attributed to Martin Tingley.

6 recordsLinked to original sources

arXiv2026

Evaluating for the long term: Learnings from industry

Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and share industry knowledge on how to make decisions from short-term experiments that are better aligned with long-term outcomes. Based on a daylong workshop with 26 experts from 15 online platforms and 4 universities, we formulate a series of propositions that reflect current industry knowledge. Participants largely agreed that reversals of sign from short-run to long-run treatment effects are rare, with reversals concentrating in specific cases such as treatments involving content quality signals, hyper-monetization, and pricing. Although the magnitude of treatment effects can shift over time, a "univariate autosurrogate", corresponding to the short-run treatment effect on the long-run metric of interest, is often hard to beat. A recurring theme was the importance of surrogates that are not only (or even primarily) unbiased for true long-run outcomes, but that improve decision-making. Thus, participants generally agreed that simple, interpretable surrogates were generally preferable to elaborate but hard-to-explain surrogate indices. Participants also agreed that, due to concerns about confounding and transportability, experimentally-learned surrogates are generally preferable to observationally-learned surrogates. However, the drawback is that learning good surrogates from experiments typically requires a large, representative portfolio of long-run experiments that few platforms possess. We conclude that there is no substitute for a well-run long-term experiment, whether for learning surrogates or validating them, and we highlight open challenges including evolving treatments, persistent treatments not fully mediated by short-term proxies, and mismatch between experimental samples and the target population.

stat.AP↗

arXiv2024

Optimizing Returns from Experimentation Programs

Experimentation in online digital platforms is used to inform decision making. Specifically, the goal of many experiments is to optimize a metric of interest. Null hypothesis statistical testing can be ill-suited to this task, as it is indifferent to the magnitude of effect sizes and opportunity costs. Given access to a pool of related past experiments, we discuss how experimentation practice should change when the goal is optimization. We survey the literature on empirical Bayes analyses of A/B test portfolios, and single out the A/B Testing Problem (Azevedo et al., 2020) as a starting point, which treats experimentation as a constrained optimization problem. We show that the framework can be solved with dynamic programming and implemented by appropriately tuning $p$-value thresholds. Furthermore, we develop several extensions of the A/B Testing Problem and discuss the implications of these results on experimentation programs in industry. For example, under no-cost assumptions, firms should be testing many more ideas, reducing test allocation sizes, and relaxing $p$-value thresholds away from $p = 0.05$.

stat.ME↗

arXiv2023

Design-Based Inference for Multi-arm Bandits

Multi-arm bandits are gaining popularity as they enable real-world sequential decision-making across application areas, including clinical trials, recommender systems, and online decision-making. Consequently, there is an increased desire to use the available adaptively collected datasets to distinguish whether one arm was more effective than the other, e.g., which product or treatment was more effective. Unfortunately, existing tools fail to provide valid inference when data is collected adaptively or require many untestable and technical assumptions, e.g., stationarity, iid rewards, bounded random variables, etc. Our paper introduces the design-based approach to inference for multi-arm bandits, where we condition the full set of potential outcomes and perform inference on the obtained sample. Our paper constructs valid confidence intervals for both the reward mean of any arm and the mean reward difference between any arms in an assumption-light manner, allowing the rewards to be arbitrarily distributed, non-iid, and from non-stationary distributions. In addition to confidence intervals, we also provide valid design-based confidence sequences, sequences of confidence intervals that have uniform type-1 error guarantees over time. Confidence sequences allow the agent to perform a hypothesis test as the data arrives sequentially and stop the experiment as soon as the agent is satisfied with the inference, e.g., the mean reward of an arm is statistically significantly higher than a desired threshold.

stat.ME↗

arXiv2022

Anytime-Valid Linear Models and Regression Adjusted Causal Inference in Randomized Experiments

Linear models are foundational tools in statistics and ubiquitous across the applied sciences. However, conventional statistical inference -- such as $t$-tests and $F$-tests -- are only valid at fixed sample sizes, making them unsuitable for sequential settings such as online A/B testing. We develop an anytime-valid theory of inference for the linear model, introducing sequential analogues of classical tests and confidence sets that provide Type-I error control and coverage guarantees uniformly over all sample sizes. Our construction is based on likelihood ratios of invariantly sufficient statistics, yielding simple closed-form expressions of ordinary least squares estimators and standard errors. The resulting tests are optimal in the GROW/REGROW sense for both frequentist and Bayesian alternative hypotheses. We then relax the linear model assumptions to provide heteroskedasticity-robust asymptotic sequential tests and confidence sequences, which enable sequential regression-adjusted inference for causal estimands in randomized controlled experiments. This formally allows experiments to be continuously monitored for significance, stopped early, and safeguards against statistical malpractices in data collection. We demonstrate the practical utility of our approach through simulations and applications to real A/B test data from Netflix.

stat.ME↗

arXiv2022

Design-Based Confidence Sequences: A General Approach to Risk Mitigation in Panel Experiments

Randomized experiments have become the standard method for companies to evaluate the performance of new products or services. Beyond aiding managerial decision-making, experiments mitigate risk by limiting the proportion of customers exposed to innovations. Since many experiments are conducted sequentially over time, an emerging strategy to further derisk the process is to allow managers to ``peek'' at the results as new data become available and stop the test if the results are statistically significant. The class of statistical methods that allow managers to peek and still provide valid inference are often called anytime-valid since they maintain proper uniform type-1 error guarantees. In this paper, we extend existing anytime-valid approaches to accommodate the more complex yet standard settings in time series, switchback, and panel experiments. To achieve this, we leverage the design-based approach to focus on assumption-light and managerial relevant finite-sample estimands defined on the study participants as a direct measure of the risks incurred by companies. As a special case, our (asymptotic) results also provide a robust method for achieving always-valid inference in A/B tests. We further provide a variance reduction technique incorporating modeling assumptions and covariates. Finally, we demonstrate the effectiveness of our proposed approach through a simulation study and three real-world applications from Netflix. Our results show that using our confidence sequence, harmful experiments could be stopped after only observing a handful of units; for instance, our method would have stopped a 30,000 person Netflix experiment after the first 100 people.

stat.ME↗

arXiv2020

Success Stories from a Democratized Experimentation Platform

We demonstrate the effectiveness of democratization and efficient computation as key concepts of our experimentation platform (XP) by presenting four new models supported by the platform: 1) Weighted least squares, 2) Quantile bootstrapping, 3) Bayesian shrinkage, and 4) Dynamic treatment effects. Each model is motivated by a specific business problem but is generalizable and extensible. The modular structure of our platform allows independent innovation on statistical and computational methods. In practice, a technical symbiosis is created where increasingly advanced user contributions inspire innovations to the software that in turn enable further methodological improvements. This cycle adds further value to how the XP contributes to business solutions.

stat.ME↗