Searcharxiv⌕ Search

arXiv subjects

Wenjing Zheng

Publications and source records attributed to Wenjing Zheng.

11 recordsLinked to original sources

Making Experiment Results Comparable through Sitewide Impact Estimation

Roblox spans many product surfaces-discovery, the virtual economy, creator tools-that differ sharply in reach and in how much the affected population contributes to any sitewide metric. A randomized experiment shows whether a treatment works for the users in its sample, but this experiment lift is not comparable across surfaces, because surfaces differ in reach and in their contribution to sitewide metrics. The quantity a strategic decision needs is a counterfactual-how much an aggregated sitewide metric (e.g., total platform play time) would move if the treatment reached 100% of its target population, which may be only part of the sitewide population, during the measurement period. We call this sitewide impact. Its scaling factor, which we call the coverage factor, combines the reach and the target population's contribution to the sitewide metric under no treatment. Hence a low-reach surface whose users carry much of the metric could outrank a high-reach surface whose users carry little. We develop two estimators under this framing: one for difference-in-means metrics and one for variant-level ratio metrics (a metric defined as a ratio of two population aggregates). Empirically, experiment lift is a poor proxy for sitewide impact-across 139 experiments that are statistically significant on time spent, the Spearman rank correlation between the two is 0.20, and 5 of the 10 largest experiment lifts are among the 10 largest sitewide impacts. We validate the estimator with A/A tests and against a long-term holdout, and apply it to cross-domain prioritization and attributing year-over-year growth to shipped innovations.

stat.AP↗

From Viral Exposure to Engagement: A Causal Mediation Framework for Digital Platforms

At Roblox, viral hits can change how exposed players engage with the viral game and with the rest of the platform. We develop a causal mediation framework to decompose the effect of viral-game playtime on subsequent player engagement into a direct effect and indirect effects through measured player interaction and discovery behaviors. The framework uses a causal DAG to state the assumed causal structure and a time-ordered design that measures covariates, exposure, mediators, and outcomes in non-overlapping time windows. Under stated identification assumptions and a model linear in the exposure and each mediator at fixed covariates, we estimate every path as a function of player covariates with cross-fitted R-learners, which use only prediction models and therefore scale to platform-size data. Averaging the fitted surfaces inherits their regularization bias, so we add a per-player bias correction before averaging (Chernozhukov et al., 2018). In simulation, the correction restored nominal coverage of 90% to 95%, against at most 39% for intervals built by averaging the surfaces. Counterintuitively, interval coverage worsened with more data on a difficult mediator-outcome path, bias shrank more slowly than the standard error, reducing corrected coverage from 93.6% to 57.3% as n increased from 2k to 200k. We applied the same design to two de-identified viral games with different lifecycle histories, comparing effects across outcome scopes, player subgroups, and a placebo window without a viral event. The results indicate that subsequent engagement is not explained by viral exposure alone, player interaction behaviors mediate 46% engagement within the viral game, whereas new game discovery mediates 52% engagement in other games. Finally, we compare the estimator with its alternatives and generalize the study design into a reusable framework for similar mediation studies across product domains.

stat.AP↗

From More to Better: Causal Inference and Digital Experimentation in the Age of AI

Artificial intelligence (AI) is quickly expanding the capacity of organizations to produce and analyze data. We argue that this expansion shifts the leverage of data scientists from performing individual analyses to assessing the credibility and importance of findings and determining when evidence generalizes across contexts. Drawing on a roundtable of academic researchers and industry practitioners, we examine three ways in which data scientists who work on causal inference and experimentation can help organizations turn more analytical capacity into better decisions: preserving and accumulating the learning value of experimentation beyond individual tests; choosing and combining experimental, observational, and synthetic evidence; and ensuring evaluation keeps pace with AI as its capabilities and uses evolve. In each area, we distinguish established practice from open questions. We conclude with implications for the training and practice of data science and for organizations.

cs.AI↗

Evaluating for the long term: Learnings from industry

Online platforms prioritize long-term business outcomes, yet typical experiments are far too short to measure these outcomes directly. Our goal in this paper is to collect and share industry knowledge on how to make decisions from short-term experiments that are better aligned with long-term outcomes. Based on a daylong workshop with 26 experts from 15 online platforms and 4 universities, we formulate a series of propositions that reflect current industry knowledge. Participants largely agreed that reversals of sign from short-run to long-run treatment effects are rare, with reversals concentrating in specific cases such as treatments involving content quality signals, hyper-monetization, and pricing. Although the magnitude of treatment effects can shift over time, a "univariate autosurrogate", corresponding to the short-run treatment effect on the long-run metric of interest, is often hard to beat. A recurring theme was the importance of surrogates that are not only (or even primarily) unbiased for true long-run outcomes, but that improve decision-making. Thus, participants generally agreed that simple, interpretable surrogates were generally preferable to elaborate but hard-to-explain surrogate indices. Participants also agreed that, due to concerns about confounding and transportability, experimentally-learned surrogates are generally preferable to observationally-learned surrogates. However, the drawback is that learning good surrogates from experiments typically requires a large, representative portfolio of long-run experiments that few platforms possess. We conclude that there is no substitute for a well-run long-term experiment, whether for learning surrogates or validating them, and we highlight open challenges including evolving treatments, persistent treatments not fully mediated by short-term proxies, and mismatch between experimental samples and the target population.

stat.AP↗

Tests of CP symmetry in entangled hyperon anti-hyperon pairs at BESIII

Decays of charmonium into hyperon and antihyperon pairs provide a pristine laboratory for exploring hyperon properties, such as their polarization and decay parameters, and for conducting tests of fundamental symmetries. This brief review highlights the significant progress made in precise tests of CP symmetry at BESIII using entangled hyperon-antihyperon pairs, including $Λ\barΛ$, $Σ\barΣ$, $Ξ\barΞ$ and $Λ\barΣ$, selected from the high statistics of $J/ψ$ and $ψ(3686)$ events produced in $e^+e^-$ annihilations. These recent findings have sparked renewed interest in both theoretical and experimental aspects of hyperon physics, but there is still much room for improvement to reach the Standard Model expectations. To address this challenge, the prospects for future investigations on CP asymmetry at next-generation experiments are discussed.

hep-ex↗

A Framework for Generalization and Transportation of Causal Estimates Under Covariate Shift

Randomized experiments are an excellent tool for estimating internally valid causal effects with the sample at hand, but their external validity is frequently debated. While classical results on the estimation of Population Average Treatment Effects (PATE) implicitly assume random selection into experiments, this is typically far from true in many medical, social-scientific, and industry experiments. When the experimental sample is different from the target sample along observable or unobservable dimensions, experimental estimates may be of limited use for policy decisions. We begin by decomposing the extrapolation bias from estimating the Target Average Treatment Effect (TATE) using the Sample Average Treatment Effect (SATE) into covariate shift, overlap, and effect modification components, which researchers can reason about in order to diagnose the severity of extrapolation bias. Next, We cast covariate shift as a sample selection problem and propose estimators that re-weight the doubly-robust scores from experimental subjects to estimate treatment effects in the overall sample (=: generalization) or in an alternate target sample (=: transportation). We implement these estimators in the open-source R package causalTransportR and illustrate its performance in a simulation study and discuss diagnostics to evaluate its performance.

stat.ME↗

A framework for causal segmentation analysis with machine learning in large-scale digital experiments

We present an end-to-end methodological framework for causal segment discovery that aims to uncover differential impacts of treatments across subgroups of users in large-scale digital experiments. Building on recent developments in causal inference and non/semi-parametric statistics, our approach unifies two objectives: (1) the discovery of user segments that stand to benefit from a candidate treatment based on subgroup-specific treatment effects, and (2) the evaluation of causal impacts of dynamically assigning units to a study's treatment arm based on their predicted segment-specific benefit or harm. Our proposal is model-agnostic, capable of incorporating state-of-the-art machine learning algorithms into the estimation procedure, and is applicable in randomized A/B tests and quasi-experiments. An open source R package implementation, sherlock, is introduced.

stat.ME↗

Semi-Supervised Record Linkage for Construction of Large-Scale Sociocentric Networks in Resource-limited Settings: An application to the SEARCH Study in Rural Uganda and Kenya

This paper presents a novel semi-supervised algorithmic approach to creating large scale sociocentric networks in rural East Africa. We describe the construction of 32 large-scale sociocentric social networks in rural Sub-Saharan Africa. Networks were constructed by applying a semi-supervised record-linkage algorithm to data from census-enumerated residents of the 32 communities included in the SEARCH study (NCT01864603), a community-cluster randomized HIV prevention trial in Uganda and Kenya. Contacts were solicited using a five question name generator in the domains of emotional support, food sharing, free time, health issues and money issues. The fully constructed networks include 170; 028 nodes and 362; 965 edges aggregated across communities (ranging from 4449 to 6829 nodes and from 2349 to 31,779 edges per community). Our algorithm matched on average 30% of named contacts in Kenyan communities and 50% of named contacts in Ugandan communities to residents named in census enumeration. Assortative mixing measures for eight different covariates reveal that residents in the network have a very strong tendency to associate with others who are similar to them in age, sex, and especially village. The networks in the SEARCH Study will provide a platform for improved understanding of health outcomes in rural East Africa. The network construction algorithm we present may facilitate future social network research in resource-limited settings.

stat.AP↗

Robust and Flexible Estimation of Stochastic Mediation Effects: A Proposed Method and Example in a Randomized Trial Setting

Causal mediation analysis can improve understanding of the mechanisms underlying epidemiologic associations. However, the utility of natural direct and indirect effect estimation has been limited by the assumption of no confounder of the mediator-outcome relationship that is affected by prior exposure---an assumption frequently violated in practice. We build on recent work that identified alternative estimands that do not require this assumption and propose a flexible and double robust semiparametric targeted minimum loss-based estimator for data-dependent stochastic direct and indirect effects. The proposed method treats the intermediate confounder affected by prior exposure as a time-varying confounder and intervenes stochastically on the mediator using a distribution which conditions on baseline covariates and marginalizes over the intermediate confounder. In addition, we assume the stochastic intervention is given, conditional on observed data, which results in a simpler estimator and weaker identification assumptions. We demonstrate the estimator's finite sample and robustness properties in a simple simulation study. We apply the method to an example from the Moving to Opportunity experiment. In this application, randomization to receive a housing voucher is the treatment/instrument that influenced moving to a low-poverty neighborhood, which is the intermediate confounder. We estimate the data-dependent stochastic direct effect of randomization to the voucher group on adolescent marijuana use not mediated by change in school district and the stochastic indirect effect mediated by change in school district. We find no evidence of mediation. Our estimator is easy to implement in standard statistical software, and we provide annotated R code to further lower implementation barriers.

stat.AP↗

A new approach to hierarchical data analysis: Targeted maximum likelihood estimation for the causal effect of a cluster-level exposure

We often seek to estimate the impact of an exposure naturally occurring or randomly assigned at the cluster-level. For example, the literature on neighborhood determinants of health continues to grow. Likewise, community randomized trials are applied to learn about real-world implementation, sustainability, and population effects of interventions with proven individual-level efficacy. In these settings, individual-level outcomes are correlated due to shared cluster-level factors, including the exposure, as well as social or biological interactions between individuals. To flexibly and efficiently estimate the effect of a cluster-level exposure, we present two targeted maximum likelihood estimators (TMLEs). The first TMLE is developed under a non-parametric causal model, which allows for arbitrary interactions between individuals within a cluster. These interactions include direct transmission of the outcome (i.e. contagion) and influence of one individual's covariates on another's outcome (i.e. covariate interference). The second TMLE is developed under a causal sub-model assuming the cluster-level and individual-specific covariates are sufficient to control for confounding. Simulations compare the alternative estimators and illustrate the potential gains from pairing individual-level risk factors and outcomes during estimation, while avoiding unwarranted assumptions. Our results suggest that estimation under the sub-model can result in bias and misleading inference in an observational setting. Incorporating working assumptions during estimation is more robust than assuming they hold in the underlying causal model. We illustrate our approach with an application to HIV prevention and treatment.

stat.ME↗

Sampling Lissajous and Fourier knots

A Lissajous knot is one that can be parameterized by a single cosine function in each coordinate. Lissajous knots are highly symmetric, and for this reason, not all knots are Lissajous. We prove several theorems which allow us to place bounds on the number of Lissajous knot types with given frequencies and to efficiently sample all possible Lissajous knots with a given set of frequencies. In particular, we systematically tabulate all Lissajous knots with small frequencies and as a result substantially enlarge the tables of known Lissajous knots. A Fourier (i, j, k) knot is similar to a Lissajous knot except that each coordinate is now described by a finite sum of i, j, and k cosine functions respectively. According to Lamm, every knot is a Fourier-(1,1,k) knot for some k. By randomly searching the set of Fourier-(1,1,2) knots we find that all 2-bridge knots up to 14 crossings are either Lissajous or Fourier-(1,1,2) knots. We show that all twist knots are Fourier-(1,1,2) knots and give evidence suggesting that all torus knots are Fourier-(1,1,2) knots. As a result of our computer search, several knots with relatively small crossing numbers are identified as potential counterexamples to interesting conjectures.

math.GT↗