arXiv subjects

Abbas Zaidi

Publications and source records attributed to Abbas Zaidi.

13 recordsLinked to original sources

arXiv2026

Let Them Peek: Group Sequential Designs for A/B Testing

Randomized controlled trials, commonly known as A/B tests, are the standard mechanism for inferring causal effects of product innovations. Standard platforms evaluate these tests with fixed-horizon procedures that control the false positive rate at a predetermined time, forcing teams to wait until the end even when early data strongly indicates success or failure. Unfortunately, continuous monitoring ("peeking") invalidates these guarantees, while always-valid inference is overly conservative under a strict deadline, and group sequential designs (GSD) have received limited traction in large-scale experimentation due to operational barriers: terminal-rule preservation, autocorrelation, latent effect-size distributions, calendar-duration optimization. We fill in this gap by proposing a Predictive Probability of Success (PPoS) group sequential framework that is autocorrelation-aware and duration-optimal while preserving the platform's fixed-horizon terminal rule. Our key insight is that PPoS thresholds map to closed-form lift boundaries that converge to the standard test at the horizon by construction, admit an effective-sample-size correction estimated from historical A/A tests, and suffice to attain the likelihood-ratio efficiency floor. Experimentally on online A/A and A/B experiments, we show that the proposed policy reduces expected duration by 43.1% (7.97 days vs. 14-day baseline), increasing effective platform capacity by 76%, while retaining the guarantees on FPR and power.

stat.AP↗

arXiv2026

Halve-Then-Greedy: Adaptive Sampling for Posterior Success with Near-Optimal Instance Dependence

We study fixed-budget adaptive experimentation aimed at maximizing the probability of obtaining sufficient posterior evidence that a recommended design's mean performance exceeds a target. Unlike best-arm identification, this objective does not require finding the best design. For Gaussian bandits with a Gaussian prior, we propose \emph{Halve-Then-Greedy} (HTG), which combines a large initial balanced allocation, successive halving, and posterior-greedy sampling. We prove instance-dependent, finite-budget exponential bounds on posterior-failure probability that capture both separation from unfavorable arms and the presence of multiple favorable arms. On one-positive-arm families, we establish an asymptotic sample-complexity lower bound that HTG matches up to a logarithmic factor. We further analyze uniform sampling and sequential halving, revealing regimes where they require more samples than HTG by linear and logarithmic factors, respectively. Simulations illustrate HTG's sample-efficiency advantages and their dependence on instance structure.

cs.LG↗

arXiv2026

Neutral Is Not Free: Evaluating Downside Risk in Neutral Launches

Evaluating "neutral launches" (e.g., infrastructure upgrades) using traditional confidence interval overlap is flawed: it is dangerously permissive with scarce data and excessively restrictive with abundant data. To resolve this, this paper introduces Expected Bayesian Loss (EBL), a continuous metric that quantifies both the probability and expected severity of metric degradation. Computable directly from standard frequentist estimates, EBL explicitly penalizes empirical noise and high-variance experiments. Validated against expert decisions, EBL provides experimentation platforms with a rigorous, tunable guardrail that aligns statistical safety with institutional risk appetite.

stat.AP↗

arXiv2026

On the Benefit of Blocking for Online Experiments with Skewed Data

This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.

stat.AP↗

arXiv2026

Incentive Alignment in Online Experimentation

Evaluating the causal effect of new features is a central goal for online platforms. While recent literature addresses limited testing traffic via centralized portfolio optimization, this perspective abstracts away a critical institutional reality: experimentation is operationally decentralized. The experimenters who develop new features also dictate which hypotheses to test, and they are typically rewarded based on empirical average treatment effects that are prone to upward bias. Left unchecked, this principal-agent conflict can severely erode platform value, a structural failure that conventional centralized levers, such as significance thresholds and traffic budgets, cannot resolve. By reframing experimentation as an incentive design problem, we demonstrate that two practical mechanisms, sample splitting and shrinkage, can effectively bridge this gap. Sample splitting aligns incentives perfectly at a bounded traffic cost, while shrinkage consumes no additional traffic and guarantees that interventions with negative expected effects are strictly unprofitable to field.

cs.GT↗

arXiv2026

Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

The widespread adoption of randomized controlled trials (A/B Tests) for decision-making has introduced a pervasive "Winner's Curse": experiments selected for launch often exhibit upwardly biased effect estimates and invalid confidence intervals. This selection bias leads to over-optimistic impact projections and undermines decision-making, particularly in low-power regimes. We propose Bayesian Hybrid Shrinkage (BHS), an empirical Bayes (EB) framework that leverages data-driven priors to mitigate selection bias and provides accurate uncertainty quantification. Unlike traditional EB methods that apply uniform shrinkage, BHS introduces an experiment-specific "local" shrinkage factor that incorporates individual experiment characteristics, improving robustness against prior misspecification. We also derive a closed-form inference strategy designed for high-throughput production environments. Extensive simulations and real-world evaluations at Meta Platforms demonstrate that BHS outperforms existing methods in terms of bias reduction and interval coverage, even under substantial violations of modeling assumptions.

stat.ME↗

arXiv2025

Breaking the Winner's Curse with Bayesian Hybrid Shrinkage

A 'Winner's Curse' arises in large-scale online experimentation platforms when the same experiments are used to both select treatments and evaluate their effects. In these settings, classical difference-in-means estimators of treatment effects are upwardly biased and conventional confidence intervals are rendered invalid. The bias scales with the magnitude of sampling variability and the selection threshold, and inversely with the treatment's true effect size. We propose a new Bayesian approach that incorporates experiment-specific 'local shrinkage' factors that mitigate sensitivity to the choice of prior and improve robustness to assumption violations. We demonstrate how the associated posterior distribution can be estimated without numerical integration techniques, making it a practical choice for at-scale deployment. Through simulation, we evaluate the performance of our approach under various scenarios and find that it performs well even when assumptions about the sampling and selection processes are violated. In an empirical evaluation, our approach demonstrated superior performance over alternative methods, providing more accurate estimates with well-calibrated uncertainty quantification.

stat.ME↗

arXiv2025

Bayesian Predictive Probabilities for Online Experimentation

The widespread adoption of online randomized controlled experiments (A/B Tests) for decision-making has created ongoing capacity constraints which necessitate interim analyses. As a consequence, platform users are increasingly motivated to use ad-hoc means of optimizing limited resources via peeking. Such processes, however, are error prone and often misaligned with end-of-experiment outcomes (e.g., inflated type-I error). We introduce a system based on Bayesian Predictive Probabilities that enable us to perform interim analyses without compromising fidelity of the experiment; This idea has been widely utilized in applications outside of the technology domain to more efficiently make decisions in experiments. Motivated by at-scale deployment within an experimentation platform, we demonstrate how predictive probabilities can be estimated without numerical integration techniques and recommend systems to study its properties at scale as an ongoing health check, along with system design recommendations - all on experiment data from Instagram - to demonstrate practical benefits that it enables.

stat.AP↗

arXiv2025

Treatment Effect Learning Under Sequential Randomization

Sequential treatment assignments in online experiments lead to complex dependency structures, often rendering identification, estimation and inference over treatments a challenge. Treatments in one session (e.g., a user logging on) can have an effect that persists into subsequent sessions, leading to cumulative effects on outcomes measured at a later stage. This can render standard methods for identification and inference trivially misspecified. We propose T-Learners layered into the G-Formula for this setting, building on literature from causal machine learning and identification in sequential settings. In a simple simulation, this approach prevents decaying accuracy in the presence of carry-over effects, highlighting the importance of identification and inference strategies tailored to the nature of systems often seen in the tech domain.

stat.AP↗

arXiv2021

Bayesian Sensitivity Analysis for Missing Data Using the E-value

Sensitivity Analysis is a framework to assess how conclusions drawn from missing outcome data may be vulnerable to departures from untestable underlying assumptions. We extend the E-value, a popular metric for quantifying robustness of causal conclusions, to the setting of missing outcomes. With motivating examples from partially-observed Facebook conversion events, we present methodology for conducting Sensitivity Analysis at scale with three contributions. First, we develop a method for the Bayesian estimation of sensitivity parameters leveraging noisy benchmarks(e.g., aggregated reports for protecting unit-level privacy); both empirically derived subjective and objective priors are explored. Second, utilizing the Bayesian estimation of the sensitivity parameters we propose a mechanism for posterior inference of the E-value via simulation. Finally, closed form distributions of the E-value are constructed to make direct inference possible when posterior simulation is infeasible due to computational constraints. We demonstrate gains in performance over asymptotic inference of the E-value using data-based simulations, supplemented by a case-study of Facebook conversion events.

stat.ME↗

arXiv2018

Gaussian Process Mixtures for Estimating Heterogeneous Treatment Effects

We develop a Gaussian-process mixture model for heterogeneous treatment effect estimation that leverages the use of transformed outcomes. The approach we will present attempts to improve point estimation and uncertainty quantification relative to past work that has used transformed variable related methods as well as traditional outcome modeling. Earlier work on modeling treatment effect heterogeneity using transformed outcomes has relied on tree based methods such as single regression trees and random forests. Under the umbrella of non-parametric models, outcome modeling has been performed using Bayesian additive regression trees and various flavors of weighted single trees. These approaches work well when large samples are available, but suffer in smaller samples where results are more sensitive to model misspecification - our method attempts to garner improvements in inference quality via a correctly specified model rooted in Bayesian non-parametrics. Furthermore, while we begin with a model that assumes that the treatment assignment mechanism is known, an extension where it is learnt from the data is presented for applications to observational studies. Our approach is applied to simulated and real data to demonstrate our theorized improvements in inference with respect to two causal estimands: the conditional average treatment effect and the average treatment effect. By leveraging our correctly specified model, we are able to more accurately estimate the treatment effects while reducing their variance.

stat.ME↗

arXiv2016

Flexible Models for Microclustering with Application to Entity Resolution

Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points. Finite mixture models, Dirichlet process mixture models, and Pitman--Yor process mixture models make this assumption, as do all other infinitely exchangeable clustering models. However, for some applications, this assumption is inappropriate. For example, when performing entity resolution, the size of each cluster should be unrelated to the size of the data set, and each cluster should contain a negligible fraction of the total number of data points. These applications require models that yield clusters whose sizes grow sublinearly with the size of the data set. We address this requirement by defining the microclustering property and introducing a new class of models that can exhibit this property. We compare models within this class to two commonly used clustering models using four entity-resolution data sets.

stat.ME↗

arXiv2015

Microclustering: When the Cluster Sizes Grow Sublinearly with the Size of the Data Set

Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points. Finite mixture models, Dirichlet process mixture models, and Pitman--Yor process mixture models make this assumption, as do all other infinitely exchangeable clustering models. However, for some tasks, this assumption is undesirable. For example, when performing entity resolution, the size of each cluster is often unrelated to the size of the data set. Consequently, each cluster contains a negligible fraction of the total number of data points. Such tasks therefore require models that yield clusters whose sizes grow sublinearly with the size of the data set. We address this requirement by defining the \emph{microclustering property} and introducing a new model that exhibits this property. We compare this model to several commonly used clustering models by checking model fit using real and simulated data sets.

stat.ME↗