arXiv subjects

Ryan Kessler

Publications and source records attributed to Ryan Kessler.

3 recordsLinked to original sources

arXiv2026

Robust Estimation of Prior Parameters in Normal-Normal Empirical Bayes

Many large-scale A/B testing systems leverage normal-normal empirical Bayes models to improve estimates of treatment impacts and launch decisions. We show that in these models maximum likelihood estimates of the variance of true impacts can be sensitive to a small number of extreme observations. We propose a bounded influence maximum likelihood estimator (BIMLE) that downweights such observations in proportion to their inconsistency with the model, reducing sensitivity while retaining the full sample in estimation. Absent extreme observations, BIMLE coincides with the standard MLE. In an application to A/B tests run at Amazon, BIMLE reduces instability in the estimated prior variance by over 50 percent while downweighting less than 0.1 percent of observations.

stat.ME↗

arXiv2026

Evaluating A/B Testing Methodologies via Sample Splitting: Theory and Practice

We develop a theoretical framework for sample splitting in A/B testing environments, where data for each test are partitioned into two splits to measure methodological performance when the true impacts of tests are unobserved. We show that sample-split estimators are generally biased for full-sample performance but consistently estimate sample-split analogues of it. We derive their asymptotic distributions, construct valid confidence intervals, and characterize the bias-variance trade-offs underlying sample-split design choices. We validate our theoretical results through simulations and provide implementation guidance for A/B testing products seeking to evaluate new estimators and decision rules.

econ.EM↗

arXiv2026

Rewarding Noise: Incentives in Decentralized A/B Testing

Many firms that run A/B tests struggle with low statistical power, increasing the risk that launch decisions are driven by noise rather than true impact. We argue that in decentralized organizations, low power can reflect misaligned incentives: product teams choose what and how to test on a customer-facing experience to maximize their private reward, while the experience owner sets evaluation rules and launch criteria to optimize the experience. We formalize this interaction via a principal-agent model and show that evaluating tests using raw impact estimates creates a reward to sampling noise. Correctly calibrated posterior-mean evaluation removes this reward and creates a local incentive to reallocate toward fewer, more precise tests of more ambitious ideas. Using data on A/B tests run at Amazon, we document patterns consistent with these model predictions. After an Amazon experience shifted to posterior-mean-based evaluation in 2024, test volume declined while both precision and the dispersion of true impacts increased, more than doubling signal-to-noise ratios. Our results highlight that A/B testing methodologies can shape not only measurement and launch decisions, but which tests are pursued in the first place.

econ.TH↗