Searcharxiv⌕ Search

arXiv subjects

Vamsi Manthena

Publications and source records attributed to Vamsi Manthena.

2 recordsLinked to original sources

Making Experiment Results Comparable through Sitewide Impact Estimation

Roblox spans many product surfaces-discovery, the virtual economy, creator tools-that differ sharply in reach and in how much the affected population contributes to any sitewide metric. A randomized experiment shows whether a treatment works for the users in its sample, but this experiment lift is not comparable across surfaces, because surfaces differ in reach and in their contribution to sitewide metrics. The quantity a strategic decision needs is a counterfactual-how much an aggregated sitewide metric (e.g., total platform play time) would move if the treatment reached 100% of its target population, which may be only part of the sitewide population, during the measurement period. We call this sitewide impact. Its scaling factor, which we call the coverage factor, combines the reach and the target population's contribution to the sitewide metric under no treatment. Hence a low-reach surface whose users carry much of the metric could outrank a high-reach surface whose users carry little. We develop two estimators under this framing: one for difference-in-means metrics and one for variant-level ratio metrics (a metric defined as a ratio of two population aggregates). Empirically, experiment lift is a poor proxy for sitewide impact-across 139 experiments that are statistically significant on time spent, the Spearman rank correlation between the two is 0.20, and 5 of the 10 largest experiment lifts are among the 10 largest sitewide impacts. We validate the estimator with A/A tests and against a long-term holdout, and apply it to cross-domain prioritization and attributing year-over-year growth to shipped innovations.

stat.AP↗

From Viral Exposure to Engagement: A Causal Mediation Framework for Digital Platforms

At Roblox, viral hits can change how exposed players engage with the viral game and with the rest of the platform. We develop a causal mediation framework to decompose the effect of viral-game playtime on subsequent player engagement into a direct effect and indirect effects through measured player interaction and discovery behaviors. The framework uses a causal DAG to state the assumed causal structure and a time-ordered design that measures covariates, exposure, mediators, and outcomes in non-overlapping time windows. Under stated identification assumptions and a model linear in the exposure and each mediator at fixed covariates, we estimate every path as a function of player covariates with cross-fitted R-learners, which use only prediction models and therefore scale to platform-size data. Averaging the fitted surfaces inherits their regularization bias, so we add a per-player bias correction before averaging (Chernozhukov et al., 2018). In simulation, the correction restored nominal coverage of 90% to 95%, against at most 39% for intervals built by averaging the surfaces. Counterintuitively, interval coverage worsened with more data on a difficult mediator-outcome path, bias shrank more slowly than the standard error, reducing corrected coverage from 93.6% to 57.3% as n increased from 2k to 200k. We applied the same design to two de-identified viral games with different lifecycle histories, comparing effects across outcome scopes, player subgroups, and a placebo window without a viral event. The results indicate that subsequent engagement is not explained by viral exposure alone, player interaction behaviors mediate 46% engagement within the viral game, whereas new game discovery mediates 52% engagement in other games. Finally, we compare the estimator with its alternatives and generalize the study design into a reusable framework for similar mediation studies across product domains.

stat.AP↗