arXiv subjects

Ilya Gorbachev

Publications and source records attributed to Ilya Gorbachev.

5 recordsLinked to original sources

arXiv2026

Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale

A/B testing underpins product decisions by providing trustworthy, reproducible quantification of proposed changes' effects. Growing large language model (LLM) capabilities have motivated efforts to automate experimentation workflows. Yet LLMs are nondeterministic and prone to hallucination, properties that conflict with the trust experimentation systems must provide. Experiments are also operationally messy: diagnosing anomalies, debugging configurations, and conducting bespoke follow-up analyses require flexible reasoning and manual effort. These tasks let agents accelerate experimentation without replacing its deterministic statistical foundations. We report lessons from building and operating such an agent. Our central finding is that this tension is resolved not by making the model deterministic, but by bounding where its nondeterminism can reach; the model selects and sequences procedures, while deterministic code computes every quantity it reports. Three consequences surprised us: withholding information from the model improved its answers, demand concentrated on mundane retrieval rather than the autonomous decisions we anticipated, and the agent's eagerness to help threatened statistical validity.

cs.AI↗

arXiv2026

Outlier Impact: Detection by Consequences

We introduce a novel outlier detection method that utilizes a concrete measure of the statistical impact of an outlier rather than a vague notion of "unusualness". This allows practitioners to only flag points that materially affect the inferences made from the data while also ensuring these points are anomalous. This is of particular interest in experimentation, where outliers affect the validity of the causal inferences. In particular, we consider a potential outlier's impact on the mean and the False Positive Rate (FPR) of a hypothesis test.

stat.ME↗

arXiv2026

Estimating the True Effect Size Distribution with SIMEX

Large-scale online experimentation produces noisy effect estimates, which can overstate gains and complicate decisions about launches and testing policies. We propose a nonparametric method based on SIMulation-EXtrapolation (SIMEX) to estimate the latent distribution of true effects from estimated average treatment effects with known variances. The method evaluates quantiles after adding progressively more simulated measurement noise and extrapolates the resulting inverse cumulative distribution function to the zero-noise setting while enforcing monotonicity of the quantiles. In a synthetic example with normally distributed true effects and measurement error, the method recovers the underlying effect distribution and performs nearly as well as a parametric empirical Bayes normal means approach. This provides a flexible way to characterize effect-size distributions without fully specifying a parametric model.

stat.ME↗

arXiv2026

Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit upwardly biased effect estimates and invalid confidence intervals. This selection bias leads to over-optimistic impact projections and undermines decision-making, particularly in low-power regimes. We propose Bayesian Hybrid Shrinkage (BHS), an empirical Bayes (EB) framework that leverages data-driven priors to mitigate selection bias and provides accurate uncertainty quantification. Unlike traditional EB methods that apply uniform shrinkage, BHS introduces an experiment-specific "local" shrinkage factor that incorporates individual experiment characteristics, improving robustness against prior misspecification. We also derive a closed-form inference strategy designed for high-throughput production environments. Extensive simulations and real-world evaluations at Meta Platforms demonstrate that BHS outperforms existing methods in terms of bias reduction and interval coverage, even under substantial violations of modeling assumptions.

stat.ME↗

arXiv2025

Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings, classical difference-in-means estimators of treatment effects are upwardly biased and conventional confidence intervals are rendered invalid. The bias scales with the magnitude of sampling variability and the selection threshold, and inversely with the treatment's true effect size. We propose a new Bayesian approach that incorporates experiment-specific 'local shrinkage' factors that mitigate sensitivity to the choice of prior and improve robustness to assumption violations. We demonstrate how the associated posterior distribution can be estimated without numerical integration techniques, making it a practical choice for at-scale deployment. Through simulation, we evaluate the performance of our approach under various scenarios and find that it performs well even when assumptions about the sampling and selection processes are violated. In an empirical evaluation, our approach demonstrated superior performance over alternative methods, providing more accurate estimates with well-calibrated uncertainty quantification.

stat.ME↗