arXiv subjects
Publications and source records attributed to Peng Ding.
This paper introduces a version of decoupling and randomization to establish concentration inequalities for double-indexed permutation statistics. The results yield, among other applications, a new combinatorial Hanson-Wright inequality and a new combinatorial Bennett inequality. Several illustrative examples from rank-based statistics, graph-based statistics, and causal inference are also provided.
The stratified linear permutation statistic arises in various statistics problems, including stratified and post-stratified survey sampling, stratified and post-stratified experiments, conditional permutation tests, etc. Although we can derive the Berry--Esseen bounds for the stratified linear permutation statistic based on existing bounds for the non-stratified statistics, those bounds are not sharp, and moreover, this strategy does not work in general settings with heterogeneous strata with varying sizes. We first use Stein's method to obtain a unified stratified permutational Berry--Esseen bound that can accommodate heterogeneous strata. We then apply the bound to various statistics problems, leading to stronger theoretical quantifications and thereby facilitating statistical inference in those problems.
Treatment effect heterogeneity with respect to covariates is common in instrumental variable (IV) analyses. An intuitive approach, which we call the interacted two-stage least squares (2sls), is to postulate a working linear model of the outcome on the treatment, covariates, and treatment-covariate interactions, and instrument it using the IV, covariates, and IV-covariate interactions. We clarify the causal interpretation of the interacted 2sls under the local average treatment effect (LATE) framework when the IV is valid conditional on the covariates. Our main findings are threefold. First, we show that the coefficients on the treatment-covariate interactions from the interacted 2sls are consistent for estimating treatment effect heterogeneity with respect to covariates among compliers for any outcome-generating process if and only if the product of the IV propensity score and covariates are linear in the covariates, referred to as the linear IV-covariate interactions condition. Second, assuming that the covariate vector has dimension K and includes a constant term, we show that the linear IV-covariate interactions condition holds only if the IV propensity score takes at most K distinct values. As a result, this condition is difficult to satisfy beyond two special cases: (a) the covariates are categorical with K levels, or (b) the IV is randomly assigned. These results underscore the difficulty of interpreting regression coefficients from specifications with treatment-covariate interactions when the covariates are not saturated and the IV is not unconditionally randomized, absent correct specification of the outcome model. Third, as an application of our theory, we show that the interacted 2sls with demeaned covariates is consistent for estimating the LATE under the linear IV-covariate interactions condition.
Generative modeling builds on and substantially advances the classical idea of simulating synthetic data from observed samples. This paper shows that this principle is not only natural but also theoretically well-founded for bootstrap inference: it yields statistically valid confidence intervals that apply simultaneously to both regular and irregular estimators, including settings in which Efron's bootstrap fails. In this sense, the generative modeling-based bootstrap can be viewed as a modern version of the smoothed bootstrap: it could mitigate the curse of dimensionality and remain effective in challenging regimes where estimators may lack root-$n$ consistency or a Gaussian limit.
In empirical research, when we have multiple estimators for the same parameter of interest, a central question arises: how do we combine unbiased but less precise estimators with biased but more precise ones to improve the inference? Under this setting, the point estimation problem has attracted considerable attention. In this paper, we focus on a less studied inference question: how can we conduct valid statistical inference in such settings with unknown bias? We propose a strategy to combine unbiased and biased estimators from a sensitivity analysis perspective. We derive a sequence of confidence intervals indexed by the magnitude of the bias, which enable researchers to assess how conclusions vary with the bias levels. Importantly, we introduce the notion of the b-value, a critical value of the unknown maximum relative bias at which combining estimators does not yield a significant result. We apply this strategy to three canonical combined estimators: the precision-weighted estimator, the pretest estimator, and the soft-thresholding estimator. For each estimator, we characterize the sequence of confidence intervals and determine the bias threshold at which the conclusion changes. Based on the theory, we recommend reporting the b-value based on the soft-thresholding estimator and its associated confidence intervals, which are robust to unknown bias and achieve the lowest worst-case risk among the alternatives.
We formulate factorial difference-in-differences (FDID), a research design that extends canonical difference-in-differences (DID) to settings in which an event affects all units. In many panel data applications, researchers exploit cross-sectional variation in a baseline factor alongside temporal variation in the event, but the corresponding estimand is often implicit and the justification for applying the DID estimator remains unclear. We frame FDID as a factorial design with two factors, the baseline factor $G$ and the exposure level $Z$, and define effect modification and causal moderation as the associative and causal effects of $G$ on the effect of $Z$, respectively. Under standard DID assumptions of no anticipation and parallel trends, the DID estimator identifies effect modification but not causal moderation. Identifying the latter requires an additional \emph{factorial parallel trends} assumption, that is, mean independence between $G$ and potential outcome trends. We extend the framework to conditionally valid assumptions and regression-based implementations, and further to repeated cross-sectional data and continuous $G$. We demonstrate the framework with an empirical application on the role of social capital in famine relief in China.
We study the problem of estimating causal effects under hidden confounding in the following unpaired data setting: we observe some covariates $X$ and an outcome $Y$ under different experimental conditions (environments) but do not observe them jointly; we either observe $X$ or $Y$. Under appropriate regularity conditions, the problem can be cast as an instrumental variable (IV) regression with the environment acting as a (possibly high-dimensional) instrument. When there are many environments but only a few observations per environment, standard two-sample IV estimators fail to be consistent. We propose a GMM-type estimator based on cross-fold sample splitting of the instrument-covariate sample and prove that it is consistent as the number of environments grows but the sample size per environment remains constant. We further extend the method to sparse causal effects via $\ell_1$-regularized estimation and post-selection refitting.
As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 instances and 3422 subtasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on: https://github.com/meituan-longcat/SOP-Maze.
Many scientific questions in biomedical, environmental, and psychological research involve understanding the effects of multiple factors on outcomes. While factorial experiments are ideal for this purpose, randomized controlled treatment assignment is generally infeasible in many empirical studies. Therefore, investigators must rely on observational data, where drawing reliable causal inferences for multiple factors remains challenging. As the number of treatment combinations grows exponentially with the number of factors, some treatment combinations can be rare or missing by chance in observed data, further complicating factorial effects estimation. To address these challenges, we propose a novel weighting method tailored to observational studies with multiple factors. Our approach uses weighted observational data to emulate a randomized factorial experiment, enabling simultaneous estimation of the effects of multiple factors and their interactions. Our investigations reveal a crucial nuance: achieving balance among covariates, as in single-factor scenarios, is necessary but insufficient for unbiasedly estimating factorial effects; balancing the factors is also essential in multi-factor settings. Moreover, we extend our weighting method to handle missing treatment combinations in observed data. Finally, we study the asymptotic behavior of the new weighting estimators and propose a consistent variance estimator, providing reliable inferences on factorial effects in observational studies.
Large language models (LLMs) remain vulnerable to jailbreaking attacks despite their impressive capabilities. Investigating these weaknesses is crucial for robust safety mechanisms. Existing attacks primarily distract LLMs by introducing additional context or adversarial tokens, leaving the core harmful intent unchanged. In this paper, we introduce ISA (Intent Shift Attack), which obfuscates LLMs about the intent of the attacks. More specifically, we establish a taxonomy of intent transformations and leverage them to generate attacks that may be misperceived by LLMs as benign requests for information. Unlike prior methods relying on complex tokens or lengthy context, our approach only needs minimal edits to the original request, and yields natural, human-readable, and seemingly harmless prompts. Extensive experiments on both open-source and commercial LLMs show that ISA achieves over 70% improvement in attack success rate compared to direct harmful prompts. More critically, fine-tuning models on only benign data reformulated with ISA templates elevates success rates to nearly 100%. For defense, we evaluate existing methods and demonstrate their inadequacy against ISA, while exploring both training-free and training-based mitigation strategies. Our findings reveal fundamental challenges in intent inference for LLMs safety and underscore the need for more effective defenses. Our code and datasets are available at https://github.com/NJUNLP/ISA.
Time-series experiments, also called switchback experiments or N-of-1 trials, play increasingly important roles in modern applications in medical and industrial areas. Under the potential outcomes framework, recent research has studied time-series experiments from the design-based perspective, relying solely on the randomness in the design to drive the statistical inference. Focusing on simpler statistical methods, we examine the design-based properties of regression-based methods for estimating treatment effects in time-series experiments. We demonstrate that the treatment effects of interest can be consistently estimated using ordinary least squares with an appropriately specified working model and transformed regressors. Our analysis allows for estimating a diverging number of treatment effects simultaneously, and establishes the consistency and asymptotic normality of the regression-based estimators. Additionally, we show that asymptotically, the heteroskedasticity and autocorrelation consistent variance estimators provide conservative estimates of the true, design-based variances. Importantly, although our approach relies on regression, our design-based framework allows for misspecification of the regression model.
Accurately estimating heterogeneous treatment effects (HTE) in longitudinal settings is essential for personalized decision-making across healthcare, public policy, education, and digital marketing. However, time-varying interventions introduce many unique challenges, such as carryover effects, time-varying heterogeneity, and post-treatment bias, which are not addressed by standard HTE methods. To address these challenges, we introduce TERRA (Transformer-Enabled Recursive R-learner), which facilitates longitudinal HTE estimation with flexible temporal modeling and learning. TERRA has two components. First, we use a Transformer architecture to encode full treatment-feature histories, enabling the representation of long-range temporal dependencies and carryover effects, hence capturing individual- and time-specific treatment effect variation more comprehensively. Second, we develop a recursive residual-learning formulation that generalizes the classical structural nested mean models (SNMMs) beyond parametric specifications, addressing post-treatment bias while reducing reliance on functional assumptions. In simulations and data applications, TERRA consistently outperforms strong baselines in HTE estimation in both accuracy and stability, highlighting the value of combining principled causal structure with high-capacity sequence models for longitudinal HTE.
The capability to precisely adhere to instructions is a cornerstone for Large Language Models (LLMs) to function as dependable agents in real-world scenarios. However, confronted with complex prompts, LLMs frequently encounter difficulties in fulfilling all specified requirements within a single response. Drawing inspiration from recent advancements in Chain-of-Thought (CoT) prompting and self-correction methodologies, we introduce Meeseeks (The name is inspired by Mr. Meeseeks from "Rick and Morty," a character renowned for efficiently accomplishing assigned tasks. See: https://en.wikipedia.org/wiki/Mr._Meeseeks), a fully automated iterative instruction-following benchmark equipped with an integrated feedback mechanism. Meeseeks identifies erroneous components in model responses and provides corresponding feedback accurately, thereby iteratively guiding the model toward self-correction. The dataset contains over 700 curated instances annotated by 32 distinct capability tags in Chinese and English. Extensive experimental results reveal that different state-of-the-art commercial and open-source LLMs exhibit vastly disparate performance, and even after 20 turns of iterative feedback-driven self-correction, nearly all models demonstrate suboptimal performance. We conducted comprehensive analysis from both macro and instance levels, uncovering numerous common issues prevalent in current state-of-the-art models, as well as several counterintuitive phenomena. We've open-sourced our work on https://github.com/ADoublLEN/Meeseeks.
Large Language Models (LLMs) excel at various natural language processing tasks but remain vulnerable to jailbreaking attacks that induce harmful content generation. In this paper, we reveal a critical safety inconsistency: LLMs can more effectively identify harmful requests as discriminators than defend against them as generators. This insight inspires us to explore aligning the model's inherent discrimination and generation capabilities. To this end, we propose SDGO (Self-Discrimination-Guided Optimization), a reinforcement learning framework that leverages the model's own discrimination capabilities as a reward signal to enhance generation safety through iterative self-improvement. Our method does not require any additional annotated data or external models during the training phase. Extensive experiments demonstrate that SDGO significantly improves model safety compared to both prompt-based and training-based baselines while maintaining helpfulness on general benchmarks. By aligning LLMs' discrimination and generation capabilities, SDGO brings robust performance against out-of-distribution (OOD) jailbreaking attacks. This alignment achieves tighter coupling between these two capabilities, enabling the model's generation capability to be further enhanced with only a small amount of discriminative samples. Our code and datasets are available at https://github.com/NJUNLP/SDGO.
This paper re-examines the first normalized incomplete moment, a well-established measure of inequality with wide applications in economic and social sciences. Despite the popularity of the measure itself, existing statistical inference appears to lag behind the needs of modern-age analytics. To fill this gap, we propose an alternative solution that is intuitive, computationally efficient, mathematically equivalent to the existing solutions for "standard" cases, and easily adaptable to "non-standard" ones. The theoretical and practical advantages of the proposed methodology are demonstrated via both simulated and real-life examples. In particular, we discover that a common practice in industry can lead to highly non-trivial challenges for trustworthy statistical inference, or misleading decision making altogether.
Randomized experiments are the gold standard for estimating the average treatment effect (ATE). While covariate adjustment can reduce the asymptotic variances of the unbiased Horvitz-Thompson estimators for the ATE, it suffers from finite-sample biases due to data reuse in both prediction and estimation. Traditional sample-splitting and cross-fitting methods can address the problem of data reuse and obtain unbiased estimators. However, they require that the data are independently and identically distributed, which is usually violated under the design-based inference framework for randomized experiments. To address this challenge, we propose a novel conditional cross-fitting method, under the design-based inference framework, where potential outcomes and covariates are fixed and the randomization is the sole source of randomness. We propose sample-splitting algorithms for various randomized experiments, including Bernoulli randomized experiments, completely randomized experiments, and stratified randomized experiments. Based on the proposed algorithms, we construct unbiased covariate-adjusted ATE estimators and propose valid inference procedures. Our methods can accommodate flexible machine-learning-assisted covariate adjustments and allow for model misspecification.
The American Statistical Association Biopharmaceutical Section (ASA BIOP) scientific working group on real-world evidence (RWE) has been making continuous, extended efforts towards a goal of supporting and advancing regulatory science with respect to clinical studies intended to use real-world data for evidence generation for the purpose of medical product development and evaluation (i.e., RWE studies). In 2023, the working group published a manuscript delineating challenges and opportunities in constructing estimands for RWE studies following the framework in ICH E9(R1) guidance on estimand and sensitivity analysis. As a follow-up task, we describe the other issue, sensitivity analysis. Although the FDA's definition of RWE studies includes randomized trials using RWD as a primary source of evidence generation such as pragmatic trials, here we focus on non-randomized RWE studies and the issue of unmeasured confounding which is a major source of bias for most RWE studies. We review the availability and applicability of sensitivity analysis methods for different types of unmeasured confounding. We also provide some practical and regulatory considerations on using sensitivity analysis for such RWE studies. Then, we use a plasmode case study to demonstrate realistic ways to interpret sensitivity analysis results in supporting regulatory decision-making. We conclude with a brief discussion.
Estimating causal effects in a target population with unmeasured confounders is challenging, especially when instrumental variables (IVs) are unavailable. However, IVs from auxiliary populations with similar problems can help infer causal effects in the target population. While the homogeneous conditional average treatment effect assumption has been widely used for effect transportability, it has not been explored in IV-based data fusion. We include it as a basic approach, though it may be biased when treatment effect heterogeneity exists. As an alternative approach, we introduce the equi-confounding assumption that the unmeasured confounding bias remains the same after adjusting for observed covariates, while allowing conditional average treatment effects to differ across populations. This allows us to identify the confounding bias in the auxiliary population and remove it from the treatment-outcome association in the target population to recover the causal effect. We develop multiply robust estimators under both approaches and demonstrate them through simulation studies and a real data application.