arXiv subjects

Gary Hettinger

Publications and source records attributed to Gary Hettinger.

7 recordsLinked to original sources

arXiv2026

Small Samples and Short Panels: Evaluating Policy Evaluation Methods with Realistic Data

Methods for estimating causal effects in longitudinal, quasi-experimental settings are widely used in economics, public health, political science, and other fields. However, studies evaluating effects of health policies often rely on limited sample sizes, both in terms of study units and time periods analyzed. Synthetic difference-in-differences (SDiD) and the augmented synthetic control method (ASCM) are recently developed methods for evaluating the effects of policy interventions. Although SDiD and ASCM generally rely on weaker assumptions than both DiD and SCM, they lack theoretical performance guarantees with respect to bias or coverage in relevant settings with small sample sizes or short panel lengths. To evaluate the performance of SDiD and ASCM in realistic, small-sample settings, we employ a calibrated simulation strategy that allows the injection of a known treatment effect into existing data in a setting of interest. Drawing on findings from these empirical investigations, we offer practical guidance for researchers and policymakers on when SDiD and ASCM are likely to yield reliable estimates and inferences under realistic scenarios.

stat.ME↗

arXiv2025

Bayesian Sensitivity Analyses for Policy Evaluation with Difference-in-Differences under Violations of Parallel Trends

Violations of the parallel trends assumption pose significant challenges for causal inference in difference-in-differences (DiD) studies, especially in policy evaluations where pre-treatment dynamics and external shocks may bias estimates. In this work, we propose a Bayesian DiD framework to allow us to estimate the effect of policies when parallel trends is violated. To address potential deviations from the parallel trends assumption, we introduce a formal sensitivity parameter representing the extent of the violation, specify an autoregressive AR(1) prior on this term to robustly model temporal correlation, and explore a range of prior specifications - including fixed, fully Bayesian, and empirical Bayes (EB) approaches calibrated from pre-treatment data. By systematically comparing posterior treatment effect estimates across prior configurations when evaluating Philadelphia's sweetened beverage tax using Baltimore as a control, we show how Bayesian sensitivity analyses support robust and interpretable policy conclusions under violations of parallel trends.

stat.ME↗

arXiv2024

A Causal Framework for Evaluating Drivers of Policy Effect Heterogeneity Using Difference-in-Differences

Policymakers and researchers often seek to understand how a policy differentially affects a population and the pathways driving this heterogeneity. For example, when studying an excise tax on sweetened beverages, researchers might assess the roles of cross-border shopping, economic competition, and store-level price changes on beverage sales trends. However, traditional policy evaluation tools, like the difference-in-differences (DiD) approach, primarily target average effects of the observed intervention rather than the underlying drivers of effect heterogeneity. Common approaches to evaluate sources of heterogeneity often lack a causal framework, making it difficult to determine whether observed outcome differences are truly driven by the proposed source of heterogeneity or by other confounding factors. In this paper, we present a framework for evaluating such policy drivers by representing questions of effect heterogeneity under hypothetical interventions and use it to evaluate drivers of the Philadelphia sweetened beverage tax policy effects. Building on recent advancements in estimating causal effect curves under DiD designs, we provide tools to assess policy effect heterogeneity while addressing practical challenges including confounding and neighborhood dynamics.

stat.ME↗

arXiv2024

Multiply Robust Difference-in-Differences Estimation of Causal Effect Curves for Continuous Exposures

Researchers commonly use difference-in-differences (DiD) designs to evaluate public policy interventions. While methods exist for estimating effects in the context of binary interventions, policies often result in varied exposures across regions implementing the policy. Yet, existing approaches for incorporating continuous exposures face substantial limitations in addressing confounding variables associated with intervention status, exposure levels, and outcome trends. These limitations significantly constrain policymakers' ability to fully comprehend policy impacts and design future interventions. In this work, we propose new estimators for causal effect curves within the DiD framework, accounting for multiple sources of confounding. Our approach accommodates misspecification of a subset of treatment, exposure, and outcome models while avoiding any parametric assumptions on the effect curve. We present the statistical properties of the proposed methods and illustrate their application through simulations and a study investigating the heterogeneous effects of a nutritional excise tax under different levels of accessibility to cross-border shopping.

stat.ME↗

arXiv2023

Policy effect evaluation under counterfactual neighborhood interventions in the presence of spillover

Policy interventions can spill over to units of a population that are not directly exposed to the policy but are geographically close to the units receiving the intervention. In recent work, investigations of spillover effects on neighboring regions have focused on estimating the average treatment effect of a particular policy in an observed setting. Our research question broadens this scope by asking what policy consequences would the treated units have experienced under hypothetical exposure settings. When we only observe treated unit(s) surrounded by controls -- as is common when a policy intervention is implemented in a single city or state -- this effect inquires about the policy effects under a counterfactual neighborhood policy status that we do not, in actuality, observe. In this work, we extend difference-in-differences (DiD) approaches to spillover settings and develop identification conditions required to evaluate policy effects in counterfactual treatment scenarios. These causal quantities are policy-relevant for designing effective policies for populations subject to various neighborhood statuses. We develop doubly robust estimators and use extensive numerical experiments to examine their performance under heterogeneous spillover effects. We apply our proposed method to investigate the effect of the Philadelphia beverage tax on unit sales.

stat.ME↗

arXiv2023

Estimating the Instantaneous Reproduction Number With Imperfect Data: A Method to Account for Case-Reporting Variation and Serial Interval Uncertainty

During an infectious disease outbreak, public health decision-makers require real-time monitoring of disease transmission to respond quickly and intelligently. In these settings, a key measure of transmission is the instantaneous time-varying reproduction number, $R_t$. Estimation of this number using a Time-Since-Infection model relies on case-notification data and the distribution of the serial interval on the target population. However, in practice, case-notification data may contain measurement error due to variation in case reporting while available serial interval estimates may come from studies on non-representative populations. We propose a new data-driven method that accounts for particular forms of case-reporting measurement error and can incorporate multiple partially representative serial interval estimates into the transmission estimation process. In addition, we provide practical tools for automatically identifying measurement error patterns and determining when measurement error may not be adequately accounted for. We illustrate the potential bias undertaken by methods that ignore these practical concerns through a variety of simulated outbreaks. We then demonstrate the use of our method on data from the COVID-19 pandemic to estimate transmission and explore the relationships between social distancing, temperature, and transmission.

stat.ME↗

arXiv2023

Estimation of Policy-Relevant Causal Effects in the Presence of Interference with an Application to the Philadelphia Beverage Tax

To comprehensively evaluate a public policy intervention, researchers must consider the effects of the policy not just on the implementing region, but also nearby, indirectly-affected regions. For example, an excise tax on sweetened beverages in Philadelphia was shown to not only be associated with a decrease in volume sales of taxed beverages in Philadelphia, but also an increase in sales in bordering counties not subject to the tax. The latter association may be explained by cross-border shopping behaviors of Philadelphia residents and indicate a causal effect of the tax on nearby regions, which may offset the total effect of the intervention. To estimate causal effects in this setting, we extend difference-in-differences methodology to account for such interference between regions and adjust for potential confounding present in quasi-experimental evaluations. Our doubly robust estimators for the average treatment effect on the treated and neighboring control relax standard assumptions on interference and model specification. We apply these methods to evaluate the change in volume sales of taxed beverages in 231 Philadelphia and bordering county stores due to the Philadelphia beverage tax. We also use our methods to explore the heterogeneity of effects across geographic features.

stat.ME↗