arXiv subjects

Wendy Chan

Publications and source records attributed to Wendy Chan.

4 recordsLinked to original sources

arXiv2022

Redefining Populations of Inference for Generalizations from Small Studies

With the growth in experimental studies in education, policymakers and practitioners are interested in understanding not only what works, but for whom an intervention works. This interest in the generalizability of a study's findings has benefited from advances in statistical methods that aim to improve generalizations, particularly when the original study sample is not randomly selected. A challenge, however, is that generalizations are frequently based on small study samples. Limited data affects both the precision and bias of treatment impact estimates, calling into question the validity of generalizations. This study explores the extent to which redefining the inference population is a useful tool to improve generalizations from small studies. We discuss two main frameworks for redefining populations and apply the methods to an empirical example based on a completed cluster randomized trial in education. We discuss the implications of various methods to redefine the population and conclude with guidance and some recommendations for practitioners interested in using redefinition.

stat.ME↗

arXiv2021

The Role of Distributional Overlap on the Precision Gain of Bounds for Generalization

Over the past ten years, propensity score methods have made an important contribution to improving generalizations from studies that do not select samples randomly from a population of inference. However, these methods require assumptions and recent work has considered the role of bounding approaches that provide a range of treatment impact estimates that are consistent with the observable data. An important limitation to bound estimates is that they can be uninformatively wide. This has motivated research on the use of propensity score stratification to narrow bounds. This article assesses the role of distributional overlap in propensity scores on the effectiveness of stratification to tighten bounds. Using the results of two simulation studies and two case studies, I evaluate the relationship between distributional overlap and precision gain and discuss the implications when propensity score stratification is used as a method to improve precision in the bounding framework.

stat.ME↗

arXiv2019

An Evaluation of Bounding Approaches for Generalization

Statisticians have recently developed propensity score methods to improve generalizations from randomized experiments that do not employ random sampling. However, these methods typically rely on assumptions whose plausibility may be questionable in practice. In this article, we introduce and discuss bounding, an approach that is based on alternative assumptions that may be more plausible in a given study. The bounding framework nonparametrically estimates population parameters using a range of plausible values that are consistent with the observed characteristics of the data. We illustrate how the bounds can be tightened using three approaches: imposing an alternative assumption based on monotonicity, redefining the population of inference, and using propensity score stratification. Using the results from two simulation studies, we examine the conditions under which bounds for the population parameter are tightened. We conclude with an application of bounding to SimCalc, a cluster randomized trial that evaluated the effectiveness of a technology aid on mathematics achievement.

stat.ME↗

arXiv2017

Partial Identification of Treatment Effects for Generalizability

Recent methods to improve generalizations from nonrandom samples typically invoke assumptions such as the strong ignorability of sample selection that are often controversial in practice to derive point estimates. Rather than focus on the point estimate based inferences, this article considers inferences on partially identified estimates from fewer and weaker assumptions. We extend partial identification methods to causal generalization with nonrandom samples by using a cluster randomized trial in education. Bounds on the population average treatment effect are derived under four cases, two under no assumptions on the data, and two that assume bounded sample variation and monotonicity of response. This approach is amenable to incorporating population data frames to tighten bounds on the population average treatment effect. Under the assumptions of bounded sample variation and monotonicity, the interval estimates of the average treatment effect provide sufficiently informative bounds to rule out large treatment effects, which are consistent with the point estimates from the experimental study. This illustrates that partial identification methods can provide an alternative perspective to causal generalization in the absence of strong ignorability of sample selection.

stat.AP↗