arXiv subjects

Johann A. Gagnon-Bartsch

Publications and source records attributed to Johann A. Gagnon-Bartsch.

9 recordsLinked to original sources

arXiv2026

A General Framework for Design-Based Treatment Effect Estimation in Paired Cluster-Randomized Experiments

Paired cluster-randomized experiments (pCRTs) are common in education program impact evaluation trials. Although common, there is surprisingly no clear consensus regarding how to analyze this randomization design to estimate average treatment effects. Variance estimation is also complicated due to the dependency created through pairing clusters. Therefore, we aim to provide an intuitive and practical comparison between different estimation strategies for pCRTs to inform practitioners' choice of strategy. To this end, we present a general framework for design-based estimation of an average individual effect in pCRTs. This framework offers a novel and intuitive view on the bias-variance trade-off between point estimators and emphasizes the benefits of covariate adjustment for estimation with pCRTs. In addition to providing a general framework for estimation with pCRTs, the point and variance estimators we present support fixed-sample unbiased estimation with similar precision to a common regression model and conservative variance estimation. Through simulation studies based on an educational efficacy trial, we compare the performance of the point and variance estimators reviewed. Our analysis and simulation studies inform the choice of point and variance estimators for analyzing pCRTs in practice.

stat.ME↗

arXiv2026

Leveraging Large Language Models to Improve Precision in Randomized Controlled Trials

Large language models (LLMs) are increasingly used in statistical research and applications. However,they are also notorious for unreliable or biased information. Here, we explore whether LLMs can be used to improve the precision of randomized controlled trials (RCTs) in a safe and rigorous way. Following similar work on leveraging observational data, we incorporate LLM predictions into an RCT analysis. While incorporating external predictions to improve precision is not new, the value of using LLM predictions in this manner is an open question. We develop a pipeline for best leveraging LLM predictions in this context and apply it to three different case studies. We find that these predictions can safely improve precision, particularly when the RCT lacks predictive covariates or contains covariates, such as text data, that are well-suited to LLMs.

stat.AP↗

arXiv2025

Building A Theoretical Foundation for Combining Negative Controls and Replicates

Studies using assays to quantify the expression of thousands of genes on tens to thousands of cell samples have been carried out for over 20 years. Such assays are based on microarrays, DNA sequencing or other molecular technologies. All such studies involve unwanted variation, often called batch effects, associated with the cell samples and the assay process. Removing this unwanted variation is essential before the measurements can be used to address the questions that motivated the studies. Combining the results of replicate assays with measurements on negative control genes to estimate the unwanted variation and remove it has proved to be effective at this task. The main goal of this paper is to present asymptotic theory that explains this effectiveness. The approach can be widened by using pseudo-replicate sets of pseudo-samples, for use with studies having no replicate assays. Theory covering this case is also presented. The established theory is supported by results of empirical investigations, including simulation studies and a real-data example.

math.ST↗

arXiv2024

Combining observational and experimental data for causal inference considering data privacy

Combining observational and experimental data for causal inference can improve treatment effect estimation. However, many observational data sets cannot be released due to data privacy considerations, so one researcher may not have access to both experimental and observational data. Nonetheless, a small amount of risk of disclosing sensitive information might be tolerable to organizations that house confidential data. In these cases, organizations can employ data privacy techniques, which decrease disclosure risk, potentially at the expense of data utility. In this paper, we explore disclosure limiting transformations of observational data, which can be combined with experimental data to estimate the sample and population average treatment effects. We consider leveraging observational data to improve generalizability of treatment effect estimates when a randomized experiment (RCT) is not representative of the population of interest, and to increase precision of treatment effect estimates. Through simulation studies, we illustrate the trade-off between privacy and utility when employing different disclosure limiting transformations. We find that leveraging transformed observational data in treatment effect estimation can still improve estimation over only using data from an RCT.

stat.ME↗

arXiv2023

Using Auxiliary Data to Boost Precision in the Analysis of A/B Tests on an Online Educational Platform: New Data and New Results

Randomized A/B tests within online learning platforms represent an exciting direction in learning sciences. With minimal assumptions, they allow causal effect estimation without confounding bias and exact statistical inference even in small samples. However, often experimental samples and/or treatment effects are small, A/B tests are underpowered, and effect estimates are overly imprecise. Recent methodological advances have shown that power and statistical precision can be substantially boosted by coupling design-based causal estimation to machine-learning models of rich log data from historical users who were not in the experiment. Estimates using these techniques remain unbiased and inference remains exact without any additional assumptions. This paper reviews those methods and applies them to a new dataset including over 250 randomized A/B comparisons conducted within ASSISTments, an online learning platform. We compare results across experiments using four novel deep-learning models of auxiliary data and show that incorporating auxiliary data into causal estimates is roughly equivalent to increasing the sample size by 20\% on average, or as much as 50-80\% in some cases, relative to t-tests, and by about 10\% on average, or as much as 30-50\%, compared to cutting-edge machine learning unbiased estimates that use only data from the experiments. We show that the gains can be even larger for estimating subgroup effects, hold even when the remnant is unrepresentative of the A/B test sample, and extend to post-stratification population effects estimators.

stat.ME↗

arXiv2023

Precise Unbiased Estimation in Randomized Experiments using Auxiliary Observational Data

Randomized controlled trials (RCTs) are increasingly prevalent in education research, and are often regarded as a gold standard of causal inference. Two main virtues of randomized experiments are that they (1) do not suffer from confounding, thereby allowing for an unbiased estimate of an intervention's causal impact, and (2) allow for design-based inference, meaning that the physical act of randomization largely justifies the statistical assumptions made. However, RCT sample sizes are often small, leading to low precision; in many cases RCT estimates may be too imprecise to guide policy or inform science. Observational studies, by contrast, have strengths and weaknesses complementary to those of RCTs. Observational studies typically offer much larger sample sizes, but may suffer confounding. In many contexts, experimental and observational data exist side by side, allowing the possibility of integrating "big observational data" with "small but high-quality experimental data" to get the best of both. Such approaches hold particular promise in the field of education, where RCT sample sizes are often small due to cost constraints, but automatic collection of observational data, such as in computerized educational technology applications, or in state longitudinal data systems (SLDS) with administrative data on hundreds of thousand of students, has made rich, high-dimensional observational data widely available. We outline an approach that allows one to employ machine learning algorithms to learn from the observational data, and use the resulting models to improve precision in randomized experiments. Importantly, there is no requirement that the machine learning models are "correct" in any sense, and the final experimental results are guaranteed to be exactly unbiased. Thus, there is no danger of confounding biases in the observational data leaking into the experiment.

stat.AP↗

arXiv2021

Separating and reintegrating latent variables to improve classification of genomic data

Genomic datasets contain the effects of various unobserved biological variables in addition to the variable of primary interest. These latent variables often affect a large number of features (e.g., genes) and thus give rise to dense latent variation, which presents both challenges and opportunities for classification. Some of these latent variables may be partially correlated with the phenotype of interest and therefore helpful, while others may be uncorrelated and thus merely contribute additional noise. Moreover, whether potentially helpful or not, these latent variables may obscure weaker effects that impact only a small number of features but more directly capture the signal of primary interest. We propose the cross-residualization classifier to better account for the latent variables in genomic data. Through an adjustment and ensemble procedure, the cross-residualization classifier essentially estimates the latent variables and residualizes out their effects, trains a classifier on the residuals, and then re-integrates the the latent variables in a final ensemble classifier. Thus, the latent variables are accounted for without discarding any potentially predictive information that they may contribute. We apply the method to simulated data as well as a variety of genomic datasets from multiple platforms. In general, we find that the cross-residualization classifier performs well relative to existing classifiers and sometimes offers substantial gains.

stat.AP↗

arXiv2021

A Review of Containerization for Interactive and Reproducible Analysis

In recent decades the analysis of data has become increasingly computational. Correspondingly, this has changed how scientific and statistical work is shared. For example, it is now commonplace for underlying analysis code and data to be proffered alongside journal publications and conference talks. Unfortunately, sharing code faces several challenges. First, it is often difficult to take code from one computer and run it on another. Code configuration, version, and dependency issues often make this challenging. Secondly, even if the code runs, it is often hard to understand or interact with the analysis. This makes it difficult to assess the code and its findings, for example, in a peer review process. In this review we describe the combination of two computing technologies that help make analyses shareable, interactive, and completely reproducible. These technologies are (1) analysis containerization, which leverages virtualization to fully encapsulate analysis, data, code and dependencies into an interactive and shareable format, and (2) code notebooks, a literate programming format for interacting with analyses. The fusion of these two technologies offers significant advantages over using either individually. This review surveys how the combination enhances the accessibility and reproducibility of code, analyses, and ideas.

stat.OT↗

arXiv2019

The P-LOOP Estimator: Covariate Adjustment for Paired Experiments

In paired experiments, participants are grouped into pairs with similar characteristics, and one observation from each pair is randomly assigned to treatment. Because of both the pairing and the randomization, the treatment and control groups should be well balanced; however, there may still be small chance imbalances. It may be possible to improve the precision of the treatment effect estimate by adjusting for these imbalances. Building on related work for completely randomized experiments, we propose the P-LOOP (paired leave-one-out potential outcomes) estimator for paired experiments. We leave out each pair and then impute its potential outcomes using any prediction algorithm. The imputation method is flexible; for example, we could use lasso or random forests. While similar methods exist for completely randomized experiments, covariate adjustment methods in paired experiments are relatively understudied. A unique trade-off exists for paired experiments, where it can be unclear whether to factor in pair assignments when making adjustments. We address this issue in the P-LOOP estimator by automatically deciding whether to account for the pairing when imputing the potential outcomes. By addressing this trade-off, the method has the potential to improve precision over existing methods.

stat.AP↗