arXiv subjects

Yuhao Deng

Publications and source records attributed to Yuhao Deng.

18 recordsLinked to original sources

arXiv2026

Estimating Pathway Treatment Effects in the Presence of Intermediate Events with Multi-State Data

During clinical trials evaluating a drug's effect on a survival endpoint, intermediate events often occur in addition to the primary event. The treatment can exert its effect on the primary endpoint along multiple pathways through intermediate events. Assumptions for identifying mediation effects, such as sequential ignorability in natural effects or the dismissible components condition in separable effects, fail because intermediate events act as treatment-induced confounding. To understand the effect along each pathway, we consider hypothetical interventions in transitions between event statuses to mimic the treatment mechanism. The hypothetical interventions adjust for effects through intermediate events and marginalize over unobserved treatment-induced confounding, if any. Based on the derived efficient influence functions for the counterfactual cumulative incidences under hypothetical interventions, we construct multiply robust and semiparametrically efficient estimators for pathway treatment effects. Our proposed framework enables the examination of treatment effects through each transition, on each event, and along each path. By analyzing data from the LEADER Trial, we find that liraglutide significantly reduces the risk of cardiovascular and microvascular events. The reduction in all-cause mortality is primarily mediated by its effects on expanded major adverse cardiovascular events.

stat.ME↗

arXiv2026

EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries

The diverse formats of CSV and Parquet files in data lakes pose a significant challenge to traditional ETL, which relies on data engineers to pre-define a target database schema and build a complex pipeline for data integration. Moreover, with this approach, the integrated data often cannot support various analytical needs, as the predefined schema does not necessarily satisfy the table format or join relationships required to answer unforeseen queries. To address this, we propose EcoTable, the first natural language-based data integration framework. Given a set of user-specified natural language queries, EcoTable automatically integrates the tables into a form that adequately supports the corresponding SQL queries. EcoTable achieves this by leveraging the semantic understanding and complex reasoning capabilities of LLMs. Moreover, EcoTable addresses the scalability and cost issues introduced by expensive LLM inferences with a set of novel ideas. First, EcoTable introduces a graph to represent the overall search space, where nodes represent tables and edges carry weights indicating join likelihood produced by a lightweight deep learning model. On top of this graph data structure, EcoTable designs three components to achieve our goal: (1) the table identification layer aims to identify relevant tables via a two-stage schema linking based on user queries; (2) the graph-based validation layer aims to discover significant join paths, including necessary data transformations and bridging tables, by modeling the problem as Steiner tree searches; and (3) the table transformation layer generates transformation code to implement the joins using LLMs. We construct 4 real-world benchmark datasets with more than 200 queries. Extensive experiments demonstrate that EcoTable outperforms the state-of-the-art baselines, increasing accuracy by more than 30% and cutting LLM invocation costs by 5 times.

cs.DB↗

arXiv2026

Difference-in-differences with a mediator

Causal mediation analysis is a powerful tool for disentangling the total effect of a treatment into its direct effect on the outcome and its indirect effect mediated through an intermediate variable. However, in observational studies, confounding between treatment and potential outcomes typically renders the total and natural effects non-identifiable. In this work, we advance mediation analysis within the difference-in-differences framework. Under a mediator-adjusted parallel trends assumption and additional conditions, we demonstrate that natural indirect, direct, and total effects are identifiable in the treated group. We further derive efficient influence functions for these estimands, enabling the construction of multiply robust and nonparametrically efficient estimators. We establish the asymptotic properties of these estimators. Applying our methodology to data from the Job Corps Study, we find that job training significantly increases both short-term and long-term earnings, after controlling for the indirect effect through the proportion of weeks employed.

econ.EM↗

arXiv2026

Doubly Robust Estimation of Treatment Effects in Staggered Difference-in-Differences with Time-Varying Covariates

The difference-in-differences (DiD) design is a quasi-experimental method for estimating treatment effects. In staggered DiD with multiple treatment groups and periods, estimation based on the two-way fixed effects model yields negative weights when averaging heterogeneous group-period treatment effects into an overall effect. To address this issue, we first define group-period average treatment effects on the treated (ATT), and then define groupwise, periodwise, dynamic, and overall ATTs nonparametrically, so that the estimands are model-free. We propose doubly robust estimators for these types of ATTs in the form of augmented inverse variance weighting (AIVW). The proposed framework allows time-varying covariates that partially explain the time trends in outcomes. Even if part of the working models is misspecified, the proposed estimators still consistently estimate the parameter of interest. The asymptotic variance can be explicitly computed from influence functions. Under a homoskedastic working model, the AIVW estimator is simplified to an augmented inverse probability weighting (AIPW) estimator. We demonstrate the desirable properties of the proposed estimators through simulation and an application that compares the effects of a parallel admission mechanism with immediate admission on the China National College Entrance Examination.

stat.ME↗

arXiv2026

Semiparametric Analysis of Interval-Censored Data Subject to Inaccurate Diagnoses with A Terminal Event

Interval-censoring frequently occurs in studies of chronic diseases where disease status is inferred from intermittently collected biomarkers. Although many methods have been developed to analyze such data, they typically assume perfect disease diagnosis, which often does not hold in practice due to the inherent imperfect clinical diagnosis of cognitive functions or measurement errors of biomarkers such as cerebrospinal fluid. In this work, we introduce a semiparametric modeling framework using the Cox proportional hazards model to address interval-censored data in the presence of inaccurate disease diagnosis. Our model incorporates sensitivity and specificity of the diagnosis to account for uncertainty in whether the interval truly contains the disease onset. Furthermore, the framework accommodates scenarios involving a terminal event and when diagnosis is accurate, such as through postmortem analysis. We propose a nonparametric maximum likelihood estimation method for inference and develop an efficient EM algorithm to ensure computational feasibility. The regression coefficient estimators are shown to be asymptotically normal, achieving semiparametric efficiency bounds. We further validate our approach through extensive simulation studies and an application assessing Alzheimer's disease (AD) risk. We find that amyloid-beta is significantly associated with AD, but Tau is predictive of both AD and mortality.

stat.ME↗

arXiv2025

Debiased Inference for High-Dimensional Regression Models Based on Profile M-Estimation

Debiased inference for high-dimensional regression models has received substantial recent attention to ensure regularized estimators have valid inference. Many existing methods focus on achieving Neyman orthogonality through explicitly constructing projections onto the space of nuisance parameters, which is infeasible when an explicit form of the projection is unavailable. We introduce a general debiasing framework, Debiased Profile $M$-Estimation (DPME), which applies to a broad class of models and does not require model-specific Neyman orthogonalization or projection derivations as in existing methods. Our approach begins with obtaining an initial estimator of the parameters by optimizing a penalized objective function. To correct for the bias introduced by penalization, we construct a one-step estimator using the Newton--Raphson update, applied to the gradient of a profile function defined as the optimal objective function with the parameter of interest held fixed. We use numerical differentiation without requiring explicit calculation of the gradients. The resulting DPME estimator is shown to be asymptotically linear and normally distributed. Through extensive simulations, we demonstrate that the proposed method achieves better coverage rates than existing alternatives with largely reduced computational cost. Finally, we illustrate the utility of our method by applying it to estimate a treatment rule for multiple myeloma.

stat.ME↗

arXiv2025

Unstructured Data Analysis using LLMs: A Comprehensive Benchmark

Nowadays, the explosion of unstructured data presents immense analytical value. Leveraging the remarkable capability of large language models (LLMs) in extracting attributes of structured tables from unstructured data, researchers are developing LLM-powered data systems for users to analyze unstructured documents as working with a database. These unstructured data analysis (UDA) systems differ significantly in all aspects, including query interfaces, query optimization strategies, and operator implementations, making it unclear which performs best in which scenario. Unfortunately, there does not exist a comprehensive benchmark that offers high-quality, large-volume, and diverse datasets as well as rich query workload to thoroughly evaluate such systems. To fill this gap, we present UDA-Bench, the first benchmark for unstructured data analysis that meets all the above requirements. Specifically, we organize a team with 30 graduate students that spends over in total 10,000 hours on curating 5 datasets from various domains and constructing a relational database view from these datasets by manual annotation. These relational databases can be used as ground truth to evaluate any of these UDA systems despite their differences in programming interfaces. Moreover, we design diverse queries to analyze the attributes defined in the database schema, covering different types of analytical operators with varying selectivities and complexities. We conduct in-depth analysis of the key building blocks of existing UDA systems: query interface, query optimization, operator design, and data processing. We run exhaustive experiments over the benchmark to fully evaluate these systems and different techniques w.r.t. the above building blocks.

cs.DB↗

arXiv2025

CSTEapp: An interactive R-Shiny application of the covariate-specific treatment effect curve for visualizing individualized treatment rule

In precision medicine, deriving the individualized treatment rule (ITR) is crucial for recommending the optimal treatment based on patients' baseline covariates. The covariate-specific treatment effect (CSTE) curve presents a graphical method to visualize an ITR within a causal inference framework. Recent advancements have enhanced the causal interpretation of the CSTE curves and provided methods for deriving simultaneous confidence bands for various study types. To facilitate the implementation of these methods and make ITR estimation more accessible, we developed CSTEapp, a web-based application built on the R Shiny framework. CSTEapp allows users to upload data and create CSTE curves through simple point and click operations, making it the first application for estimating the ITRs. CSTEapp simplifies the analytical process by providing interactive graphical user interfaces with dynamic results, enabling users to easily report optimal treatments for individual patients based on their covariates information. Currently, CSTEapp is applicable to studies with binary and time-to-event outcomes, and we continually expand its capabilities to accommodate other outcome types as new methods emerge. We demonstrate the utility of CSTEapp using real-world examples and simulation datasets. By making advanced statistical methods more accessible, CSTEapp empowers researchers and practitioners across various fields to advance precision medicine and improve patient outcomes.

stat.CO↗

arXiv2025

Not All Documents Are What You Need for Extracting Instruction Tuning Data

Instruction tuning improves the performance of large language models (LLMs), but it heavily relies on high-quality training data. Recently, LLMs have been used to synthesize instruction data using seed question-answer (QA) pairs. However, these synthesized instructions often lack diversity and tend to be similar to the input seeds, limiting their applicability in real-world scenarios. To address this, we propose extracting instruction tuning data from web corpora that contain rich and diverse knowledge. A naive solution is to retrieve domain-specific documents and extract all QA pairs from them, but this faces two key challenges: (1) extracting all QA pairs using LLMs is prohibitively expensive, and (2) many extracted QA pairs may be irrelevant to the downstream tasks, potentially degrading model performance. To tackle these issues, we introduce EQUAL, an effective and scalable data extraction framework that iteratively alternates between document selection and high-quality QA pair extraction to enhance instruction tuning. EQUAL first clusters the document corpus based on embeddings derived from contrastive learning, then uses a multi-armed bandit strategy to efficiently identify clusters that are likely to contain valuable QA pairs. This iterative approach significantly reduces computational cost while boosting model performance. Experiments on AutoMathText and StackOverflow across four downstream tasks show that EQUAL reduces computational costs by 5-10x and improves accuracy by 2.5 percent on LLaMA-3.1-8B and Mistral-7B

cs.CL↗

arXiv2024

Randomized interventional effects in semicompeting risks, with application to a hematopoietic cell transplantation study

In clinical studies, the risk of the primary (terminal) event may be modified by intermediate events, resulting in semicompeting risks. To study the treatment effect on the terminal event mediated by the intermediate event, researchers wish to decompose the total effect into direct and indirect effects. In this article, we extend the randomized interventional approach to time-to-event outcomes, where both intermediate and terminal events are subject to right censoring. We envision a random draw for the intermediate event process from a reference distribution, either marginally over time-varying confounders or conditionally given the observed history. We present the identification formula for interventional effects. We also discuss some variants of the identification assumptions. We estimate the treatment effects using nonparametric maximum likelihood estimation and propose a sensitivity analysis that incorporates a latent frailty. As an illustration, we study the effect of matched unrelated donor versus haploidentical donor on death mediated by relapse in a hematopoietic cell transplantation study with graft-versus-host disease (GVHD) as the time-varying confounder. We find that matched unrelated donor transplantation is preferable in terms of survival rates under the use of post-transplant PTCy GVHD prophylaxis for lymphoma patients.

stat.ME↗

arXiv2024

Adjusted Nelson--Aalen estimators by inverse treatment probability weighting with an estimated propensity score

Inverse probability of treatment weighting (IPW) has been well applied in causal inference to estimate population-level estimands from observational studies. For time-to-event outcomes, the failure time distribution can be estimated by estimating the cumulative hazard in the presence of random right censoring. IPW can be performed by weighting the event counting process and at-risk process by the inverse treatment probability, resulting in an adjusted Nelson--Aalen estimator for the population-level counterfactual cumulative incidence function. We consider the adjusted Nelson--Aalen estimator with an estimated propensity score in the competing risks setting. When the estimated propensity score is regular and asymptotically linear, we derive the influence functions for the counterfactual cumulative hazard and cumulative incidence. Then we establish the asymptotic properties for the estimators. We show that the uncertainty in the estimated propensity score contributes to an additional variation in the estimators. However, through simulation and real-data application, we find that such an additional variation is usually small.

stat.ME↗

arXiv2024

Analysis of Broken Randomized Experiments by Principal Stratification

Although randomized controlled trials have long been regarded as the ``gold standard'' for evaluating treatment effects, there is no natural prevention from post-treatment events. For example, non-compliance makes the actual treatment different from the assigned treatment, truncation-by-death renders the outcome undefined or ill-defined, and missingness prevents the outcomes from being measured. In this paper, we develop a statistical analysis framework using principal stratification to investigate the treatment effect in broken randomized experiments. The average treatment effect in compliers and always-survivors is adopted as the target causal estimand. We establish the asymptotic property for the estimator. To relax the identification assumptions, we also propose an interventionist estimand defined in compliers by adjusting for baseline covariates. We apply the framework to study the effect of training on earnings in the Job Corps study and find that the training program improves employment and earnings in the long term.

stat.ME↗

arXiv2024

Inference for Cumulative Incidences and Treatment Effects in Randomized Controlled Trials with Time-to-Event Outcomes under ICH E9 (R1)

In randomized controlled trials (RCTs) that focus on time-to-event outcomes, intercurrent events can arise in two ways: as semi-competing events, which modify the hazard of the primary outcome events, or as competing events, which make the definition of the primary outcome events unclear. Although five strategies have been proposed in the ICH E9 (R1) addendum to address intercurrent events in RCTs, these strategies are not easily applicable to time-to-event outcomes when aiming for causal interpretations. In this study, we show how to define, estimate, and make inferences concerning objectives that have causal interpretations within these contexts. Specifically, we derive the mathematical formulations of the causal estimands corresponding to the five strategies and clarify the data structure needed to identify these causal estimands. Furthermore, we introduce nonparametric methods for estimating and making inferences about these causal estimands, including the asymptotic variance of estimators and hypothesis tests. Finally, we illustrate our methods using data from the LEADER Trial, which aims to investigate the effect of liraglutide on cardiovascular outcomes.

stat.ME↗

arXiv2023

Direct and Indirect Treatment Effects in the Presence of Semi-Competing Risks

Semi-competing risks refer to the phenomenon that the terminal event (such as death) can censor the non-terminal event (such as disease progression) but not vice versa. The treatment effect on the terminal event can be delivered either directly following the treatment or indirectly through the non-terminal event. We consider two strategies to decompose the total effect into a direct effect and an indirect effect under the framework of mediation analysis in completely randomized experiments by adjusting the prevalence and hazard of non-terminal events, respectively. They require slightly different assumptions on cross-world quantities to achieve identifiability. We establish asymptotic properties for the estimated counterfactual cumulative incidences and decomposed treatment effects. We illustrate the subtle difference between these two decompositions through simulation studies and two real-data applications.

stat.ME↗

arXiv2023

Separable pathway effects of semi-competing risks using multi-state models

Semi-competing risks refer to the phenomenon where a primary event (such as mortality) can ``censor'' an intermediate event (such as relapse of a disease), but not vice versa. Under the multi-state model, the primary event consists of two specific types: the direct outcome event and an indirect outcome event developed from intermediate events. Within this framework, we show that the total treatment effect on the cumulative incidence of the primary event can be decomposed into three separable pathway effects, capturing treatment effects on population-level transition rates between states. We next propose two estimators for the counterfactual cumulative incidences of the primary event under hypothetical treatment components. One estimator is given by the generalized Nelson--Aalen estimator with inverse probability weighting under covariates isolation, and the other is given based on the efficient influence function. The asymptotic normality of these estimators is established. The first estimator only involves a propensity score model and avoid modeling the cause-specific hazards. The second estimator has robustness against the misspecification of submodels. As an illustration of its potential usefulness, the proposed method is applied to compare effects of different allogeneic stem cell transplantation types on overall survival after transplantation.

stat.ME↗

arXiv2022

A Semi-Synthetic Dataset Generation Framework for Causal Inference in Recommender Systems

Accurate recommendation and reliable explanation are two key issues for modern recommender systems. However, most recommendation benchmarks only concern the prediction of user-item ratings while omitting the underlying causes behind the ratings. For example, the widely-used Yahoo!R3 dataset contains little information on the causes of the user-movie ratings. A solution could be to conduct surveys and require the users to provide such information. In practice, the user surveys can hardly avoid compliance issues and sparse user responses, which greatly hinders the exploration of causality-based recommendation. To better support the studies of causal inference and further explanations in recommender systems, we propose a novel semi-synthetic data generation framework for recommender systems where causal graphical models with missingness are employed to describe the causal mechanism of practical recommendation scenarios. To illustrate the use of our framework, we construct a semi-synthetic dataset with Causal Tags And Ratings (CTAR), based on the movies as well as their descriptive tags and rating information collected from a famous movie rating website. Using the collected data and the causal graph, the user-item-ratings and their corresponding user-item-tags are automatically generated, which provides the reasons (selected tags) why the user rates the items. Descriptive statistics and baseline results regarding the CTAR dataset are also reported. The proposed data generation framework is not limited to recommendation, and the released APIs can be used to generate customized datasets for other research tasks.

cs.IR↗

arXiv2022

On the Opportunity of Causal Learning in Recommendation Systems: Foundation, Estimation, Prediction and Challenges

Recently, recommender system (RS) based on causal inference has gained much attention in the industrial community, as well as the states of the art performance in many prediction and debiasing tasks. Nevertheless, a unified causal analysis framework has not been established yet. Many causal-based prediction and debiasing studies rarely discuss the causal interpretation of various biases and the rationality of the corresponding causal assumptions. In this paper, we first provide a formal causal analysis framework to survey and unify the existing causal-inspired recommendation methods, which can accommodate different scenarios in RS. Then we propose a new taxonomy and give formal causal definitions of various biases in RS from the perspective of violating the assumptions adopted in causal analysis. Finally, we formalize many debiasing and prediction tasks in RS, and summarize the statistical and machine learning-based causal estimation methods, expecting to provide new research opportunities and perspectives to the causal RS community.

cs.IR↗

arXiv2021

Causal Inference with Truncation-by-Death and Unmeasured Confounding

Clinical studies sometimes encounter truncation by death, rendering outcomes undefined. Statistical analysis based solely on observed survivors may give biased results because the characteristics of survivors differ between treatment groups. By principal stratification, the survivor average causal effect was proposed as a causal estimand defined in always-survivors. However, this estimand is not identifiable when there is unmeasured confounding between the treatment assignment and survival or outcome process. In this paper, we consider the comparison between an aggressive treatment and a conservative treatment with monotonicity on survival. First, we show that the survivor average causal effect on the conservative treatment is identifiable based on a substitutional variable under appropriate assumptions, even when the treatment assignment is not ignorable. Next, we propose an augmented inverse probability weighting (AIPW) type estimator for this estimand with double robustness. Finally, large sample properties of this estimator are established. The proposed method is applied to investigate the effect of allogeneic stem cell transplantation types on leukemia relapse.

stat.ME↗