arXiv subjects

Nicole E. Pashley

Publications and source records attributed to Nicole E. Pashley.

14 recordsLinked to original sources

arXiv2026

Disentangling Causal Mechanisms in Conjoint Experiments Using Mediation

Conjoint experiments provide an attractive way to assess the role of multiple attributes simultaneously on decision-making. However, the randomization of multiple attributes prevents understanding the causal mechanisms that, critically, depend on the relationship between attributes -- e.g., how one attribute affects the respondent's belief as to another attribute. This is because conjoint experiments recover controlled effects whereas a substantively important estimand may be the total or indirect effect of one attribute. Unfortunately, existing experimental designs for conjoint experiments cannot estimate these effects. We provide an alternative framework that requires one additional, simple experiment to learn the relationship between attributes among respondents alongside the standard assumptions for causal mediation. Estimation of the relevant effects can be done in a doubly robust fashion using machine learning methods. We illustrate this by conducting a pre-registered experiment on candidate choice and disentangle the effect of different attributes by understanding their mediation through the candidate's party.

stat.ME↗

arXiv2026

Stratified Sampling for Model-Assisted Estimation with Surrogate Outcomes

In many randomized trials, outcomes such as essays or open-ended responses must be manually scored as a preliminary step to impact analysis, a process that is costly and limiting. Model-assisted estimation offers a way to combine surrogate outcomes generated by machine learning or large language models with a human-coded subset, yet typical implementations use simple random sampling and therefore overlook systematic variation in surrogate prediction error. We extend this framework by incorporating stratified sampling to more efficiently allocate human coding effort. We derive the exact variance of the stratified model-assisted estimator, characterize conditions under which stratification improves precision, and identify a Neyman-type optimal allocation rule that oversamples strata with larger residual variance. We evaluate our methods through a comprehensive simulation study to assess finite-sample performance. Overall, we find stratification consistently improves efficiency when surrogate prediction errors exhibit structured bias or heteroskedasticity. We also present two empirical applications, one using data from an education RCT and one using a large observational corpus, to illustrate how these methods can be implemented in practice using ChatGPT-generated surrogate outcomes. Overall, this framework provides a practical design-based approach for leveraging surrogate outcomes and strategically allocating human coding effort to obtain unbiased estimates with greater efficiency. While motivated by text-as-data applications, the methodology applies broadly to any setting where outcome measurement is costly or prohibitive, and can be applied to comparisons across groups or estimating the mean of a single group.

stat.ME↗

arXiv2025

Design-based Causal Inference for Incomplete Block Designs

Researchers often turn to block randomization to increase the precision of their inference or due to practical considerations, such as in multisite trials. However, if the number of treatments under consideration is large it might not be feasible or practical to assign all treatments within each block. We develop novel inference results under the finite-population design-based framework for natural alternatives to the complete block design that do not require reducing the number of treatment arms, the incomplete block design (IBD) and the balanced incomplete block design. This includes deriving the properties of two design-based estimators, developing a finite-population central limit theorem, and proposing conservative variance estimators. Comparisons of the design-based estimators are made to linear model-based estimators. Simulations and a data illustration further demonstrate performance of IBD estimators. This work highlights IBDs as practical and currently underutilized designs.

stat.ME↗

arXiv2025

Bounds on causal effects in $2^{K}$ factorial experiments with non-compliance

Factorial experiments are ubiquitous in the social and biomedical sciences, but when units fail to comply with each assigned factors, identification and estimation of the average treatment effects become impossible without strong assumptions. Leveraging an instrumental variables approach, previous studies have shown how to identify and estimate the causal effect of treatment uptake among respondents who comply with treatment. A major caveat is that these identification results rely on strong assumptions on the effect of randomization on treatment uptake. This paper shows how to bound these complier average treatment effects for bounded outcomes under more mild assumptions on non-compliance.

stat.ME↗

arXiv2025

Estimating Heterogeneous Causal Effects of High-Dimensional Treatments: Application to Conjoint Analysis

Estimation of heterogeneous treatment effects is an active area of research. Most of the existing methods, however, focus on estimating the conditional average treatment effects of a single, binary treatment given a set of pre-treatment covariates. In this paper, we propose a method to estimate the heterogeneous causal effects of high-dimensional treatments, which poses unique challenges in terms of estimation and interpretation. The proposed approach finds maximally heterogeneous groups and uses a Bayesian mixture of regularized logistic regressions to identify groups of units who exhibit similar patterns of treatment effects. By directly modeling group membership with covariates, the proposed methodology allows one to explore the unit characteristics that are associated with different patterns of treatment effects. Our motivating application is conjoint analysis, which is a popular type of survey experiment in social science and marketing research and is based on a high-dimensional factorial design. We apply the proposed methodology to the conjoint data, where survey respondents are asked to select one of two immigrant profiles with randomly selected attributes. We find that a group of respondents with a relatively high degree of prejudice appears to discriminate against immigrants from non-European countries like Iraq. An open-source software package is available for implementing the proposed methodology.

stat.ME↗

arXiv2024

Improving instrumental variable estimators with post-stratification

Experiments studying get-out-the-vote (GOTV) efforts estimate the causal effect of various mobilization efforts on voter turnout. However, there is often substantial noncompliance in these studies. A usual approach is to use an instrumental variable (IV) analysis to estimate impacts for compliers, here being those actually contacted by the investigators. Unfortunately, popular IV estimators can be unstable in studies with a small fraction of compliers. We explore post-stratifying the data (e.g., taking a weighted average of IV estimates within each stratum) using variables that predict complier status (and, potentially, the outcome) to mitigate this. We present the benefits of post-stratification in terms of bias, variance, and improved standard error estimates, and provide a finite-sample asymptotic variance formula. We also compare the performance of different IV approaches and discuss the advantages of our design-based post-stratification approach over incorporating compliance-predictive covariates into the two-stage least squares estimator. In the end, we show that covariates predictive of compliance can increase precision, but only if one is willing to make a bias-variance trade-off by down-weighting or dropping strata with few compliers. By contrast, standard approaches such as two-stage least squares fail to use such information. We finally examine the benefits of our approach in two GOTV applications.

stat.ME↗

arXiv2023

Optimal allocation of sample size for randomization-based inference from $2^K$ factorial designs

Optimizing the allocation of units into treatment groups can help researchers improve the precision of causal estimators and decrease costs when running factorial experiments. However, existing optimal allocation results typically assume a super-population model and that the outcome data comes from a known family of distributions. Instead, we focus on randomization-based causal inference for the finite-population setting, which does not require model specifications for the data or sampling assumptions. We propose exact theoretical solutions for optimal allocation in $2^K$ factorial experiments under complete randomization with A-, D- and E-optimality criteria. We then extend this work to factorial designs with block randomization. We also derive results for optimal allocations when using cost-based constraints. To connect our theory to practice, we provide convenient integer-constrained programming solutions using a greedy optimization approach to find integer optimal allocation solutions for both complete and block randomization. The proposed methods are demonstrated using two real-life factorial experiments conducted by social scientists.

stat.ME↗

arXiv2022

Note on the Delta Method for Finite Population Inference with Applications to Causal Inference

The delta method creates more general inference results when coupled with central limit theorem results for the finite population. This opens up a range of new estimators for which we can find finite population asymptotic properties. We focus on the use of this method to derive asymptotic distributional results and variance expressions for causal estimators. We illustrate the use of the method by obtaining a finite population asymptotic distribution for a causal ratio estimator.

stat.ME↗

arXiv2022

Causal inference from treatment-control studies having an additional factor with unknown assignment mechanism

Consider a situation with two treatments, the first of which is randomized but the second is not, and the multifactor version of this. Interest is in treatment effects, defined using standard factorial notation. We define estimators for the treatment effects and explore their properties when there is information about the nonrandomized treatment assignment and when there is no information on the assignment of the nonrandomized treatment. We show when and how hidden treatments can bias estimators and inflate their sampling variances.

stat.ME↗

arXiv2022

Causal Inference for Multiple Treatments using Fractional Factorial Designs

We consider the design and analysis of multi-factor experiments using fractional factorial and incomplete designs within the potential outcome framework. These designs are particularly useful when limited resources make running a full factorial design infeasible. We connect our design-based methods to standard regression methods. We further motivate the usefulness of these designs in multi-factor observational studies, where certain treatment combinations may be so rare that there are no measured outcomes in the observed data corresponding to them. Therefore, conceptualizing a hypothetical fractional factorial experiment instead of a full factorial experiment allows for appropriate analysis in those settings. We illustrate our approach using biomedical data from the 2003-2004 cycle of the National Health and Nutrition Examination Survey to examine the effects of four common pesticides on body mass index.

stat.ME↗

arXiv2021

Conditional As-If Analyses in Randomized Experiments

The injunction to `analyze the way you randomize' is well-known to statisticians since Fisher advocated for randomization as the basis of inference. Yet even those convinced by the merits of randomization-based inference seldom follow this injunction to the letter. Bernoulli randomized experiments are often analyzed as completely randomized experiments, and completely randomized experiments are analyzed as if they had been stratified; more generally, it is not uncommon to analyze an experiment as if it had been randomized differently. This paper examines the theoretical foundation behind this practice within a randomization-based framework. Specifically, we ask when is it legitimate to analyze an experiment randomized according to one design as if it had been randomized according to some other design. We show that a sufficient condition for this type of analysis to be valid is that the design used for analysis be derived from the original design by an appropriate form of conditioning. We use our theory to justify certain existing methods, question others, and finally suggest new methodological insights such as conditioning on approximate covariate balance.

stat.ME↗

arXiv2021

Block what you can, except when you shouldn't

Several branches of the potential outcome causal inference literature have discussed the merits of blocking versus complete randomization. Some have concluded it can never hurt the precision of estimates, and some have concluded it can hurt. In this paper, we reconcile these apparently conflicting views, give a more thorough discussion of what guarantees no harm, and discuss how other aspects of a blocked design can cost, all in terms of precision. We discuss how the different findings are due to different sampling models and assumptions of how the blocks were formed. We also connect these ideas to common misconceptions, for instance showing that analyzing a blocked experiment as if it were completely randomized, a seemingly conservative method, can actually backfire in some cases. Overall, we find that blocking can have a price, but that this price is usually small and the potential for gain can be large. It is hard to go too far wrong with blocking.

stat.ME↗

arXiv2021

Design-Based Ratio Estimators and Central Limit Theorems for Clustered, Blocked RCTs

This article develops design-based ratio estimators for clustered, blocked randomized controlled trials (RCTs), with an application to a federally funded, school-based RCT testing the effects of behavioral health interventions. We consider finite population weighted least squares estimators for average treatment effects (ATEs), allowing for general weighting schemes and covariates. We consider models with block-by-treatment status interactions as well as restricted models with block indicators only. We prove new finite population central limit theorems for each block specification. We also discuss simple variance estimators that share features with commonly used cluster-robust standard error estimators. Simulations show that the design-based ATE estimator yields nominal rejection rates with standard errors near true ones, even with few clusters.

stat.ME↗

arXiv2020

Insights on Variance Estimation for Blocked and Matched Pairs Designs

Evaluating blocked randomized experiments from a potential outcomes perspective has two primary branches of work. The first focuses on larger blocks, with multiple treatment and control units in each block. The second focuses on matched pairs, with a single treatment and control unit in each block. These literatures not only provide different estimators for the standard errors of the estimated average impact, but they are also built on different sets of assumptions. Neither literature handles cases with blocks of varying size that contain singleton treatment or control units, a case which can occur in a variety of contexts, such as with different forms of matching or post-stratification. In this paper, we reconcile the literatures by carefully examining the performance of variance estimators under several different frameworks. We then use these insights to derive novel variance estimators for experiments containing blocks of different sizes.

stat.ME↗