arXiv subjects

Yinchu Zhu

Publications and source records attributed to Yinchu Zhu.

At least 19 recordsLinked to original sources

arXiv2026

Quantile optimization in semidiscrete optimal transport

Optimal transport is the problem of designing a joint distribution for two random variables with fixed marginals. In virtually the entire literature on this topic, the objective is to minimize expected cost. This paper is the first to study a variant in which the goal is to minimize a quantile of the cost, rather than the mean. For the semidiscrete setting, where one distribution is continuous and the other is discrete, we derive a complete characterization of the optimal transport plan and develop simulation-based methods to efficiently compute it. One particularly novel aspect of our approach is the efficient computation of a tie-breaking rule that preserves marginal distributions. In the context of geographical partitioning problems, the optimal plan is shown to produce a novel geometric structure.

econ.EM↗

arXiv2026

Semidiscrete optimal transport with unknown costs

Semidiscrete optimal transport is a challenging generalization of the classical transportation problem in linear programming. The goal is to design a joint distribution for two random variables (one continuous, one discrete) with fixed marginals, in a way that minimizes expected cost. We formulate a novel variant of this problem in which the cost functions are unknown, but can be learned through noisy observations; however, only one function can be sampled at a time. We develop a semi-myopic algorithm that couples online learning with stochastic approximation, and prove that it achieves optimal convergence rates, despite the non-smoothness of the stochastic gradient and the lack of strong concavity in the objective function.

econ.EM↗

arXiv2025

Individual and group fairness in geographical partitioning

Socioeconomic segregation often arises in school districting and other contexts, causing some groups to be over- or under-represented within a particular district. This phenomenon is closely linked with disparities in opportunities and outcomes. We formulate a new class of geographical partitioning problems in which the population is heterogeneous, and it is necessary to ensure fair representation for each group at each facility. We prove that the optimal solution is a novel generalization of the additively weighted Voronoi diagram, and we propose a simple and efficient algorithm to compute it, thus resolving an open question dating back to Dvoretzky et al. (1951). The efficacy and potential for practical insight of the approach are demonstrated in a realistic case study involving seven demographic groups and $78$ district offices.

econ.EM↗

arXiv2025

Minimax Semiparametric Learning With Approximate Sparsity

Estimating linear, mean-square continuous functionals is a pivotal challenge in statistics. In high-dimensional contexts, this estimation is often performed under the assumption of exact model sparsity, meaning that only a small number of parameters are precisely non-zero. This excludes models where linear formulations only approximate the underlying data distribution, such as nonparametric regression methods that use basis expansion such as splines, kernel methods or polynomial regressions. Many recent methods for root-$n$ estimation have been proposed, but the implications of exact model sparsity remain largely unexplored. In particular, minimax optimality for models that are not exactly sparse has not yet been developed. This paper formalizes the concept of approximate sparsity through classical semi-parametric theory. We derive minimax rates under this formulation for a regression slope and an average derivative, finding these bounds to be substantially larger than those in low-dimensional, semi-parametric settings. We identify several new phenomena. We discover new regimes where rate double robustness does not hold, yet root-$n$ estimation is still possible. In these settings, we propose an estimator that achieves minimax optimal rates. Our findings further reveal distinct optimality boundaries for ordered versus unordered nonparametric regression estimation.

math.ST↗

arXiv2025

New possibilities in identification of binary choice models with fixed effects

We study the identification of binary choice models with fixed effects. We propose a condition called sign saturation and show that this condition is sufficient for identifying the model. In particular, this condition can guarantee identification even when all the regressors are bounded, including multiple discrete regressors. We also establish that without this condition, the model is not identified unless the error distribution belongs to a special class. Moreover, we show that sign saturation is also essential for identifying the sign of treatment effects. Finally, we introduce a measure for sign saturation and develop tools for its estimation and inference.

econ.EM↗

arXiv2025

Debiasing and $t$-tests for synthetic control inference on average causal effects

We propose a practical and robust method for making inferences on average treatment effects estimated by synthetic controls. We develop a $K$-fold cross-fitting procedure for bias correction. To avoid the difficult estimation of the long-run variance, inference is based on a self-normalized $t$-statistic, which has an asymptotically pivotal $t$-distribution. Our $t$-test is easy to implement, provably robust against misspecification, and valid with stationary and non-stationary data. It demonstrates an excellent small sample performance in application-based simulations and performs well relative to other methods. We illustrate the usefulness of the $t$-test by revisiting the effect of carbon taxes on emissions.

econ.EM↗

arXiv2024

Statistical Inference For Noisy Matrix Completion Incorporating Auxiliary Information

This paper investigates statistical inference for noisy matrix completion in a semi-supervised model when auxiliary covariates are available. The model consists of two parts. One part is a low-rank matrix induced by unobserved latent factors; the other part models the effects of the observed covariates through a coefficient matrix which is composed of high-dimensional column vectors. We model the observational pattern of the responses through a logistic regression of the covariates, and allow its probability to go to zero as the sample size increases. We apply an iterative least squares (LS) estimation approach in our considered context. The iterative LS methods in general enjoy a low computational cost, but deriving the statistical properties of the resulting estimators is a challenging task. We show that our method only needs a few iterations, and the resulting entry-wise estimators of the low-rank matrix and the coefficient matrix are guaranteed to have asymptotic normal distributions. As a result, individual inference can be conducted for each entry of the unknown matrices. We also propose a simultaneous testing procedure with multiplier bootstrap for the high-dimensional coefficient matrix. This simultaneous inferential tool can help us further investigate the effects of covariates for the prediction of missing entries.

stat.ME↗

arXiv2023

Inference for Low-Rank Models

This paper studies inference in linear models with a high-dimensional parameter matrix that can be well-approximated by a ``spiked low-rank matrix.'' A spiked low-rank matrix has rank that grows slowly compared to its dimensions and nonzero singular values that diverge to infinity. We show that this framework covers a broad class of models of latent-variables which can accommodate matrix completion problems, factor models, varying coefficient models, and heterogeneous treatment effects. For inference, we apply a procedure that relies on an initial nuclear-norm penalized estimation step followed by two ordinary least squares regressions. We consider the framework of estimating incoherent eigenvectors and use a rotation argument to argue that the eigenspace estimation is asymptotically unbiased. Using this framework we show that our procedure provides asymptotically normal inference and achieves the semiparametric efficiency bound. We illustrate our framework by providing low-level conditions for its application in a treatment effects context where treatment assignment might be strongly dependent.

math.ST↗

arXiv2022

Can Two Forecasts Have the Same Conditional Expected Accuracy?

The approach for testing equal predictive accuracy for pairs of forecasting models proposed by Giacomini and White (2006) assumes that the parameters of the underlying forecasting models are estimated using a rolling window of fixed width and incorporates the effect of parameter estimation in the null hypothesis. We show that a necessary and sufficient condition for the conditionally expected loss differential of two forecasting models to be a martingale difference sequence is that the outcome is a simple average of the two forecasts. When the forecasts contain parameter estimation errors, this means that the conditional mean of the outcome has to be a function of past estimation errors--a condition that fails in many situations. We also show that the null can fail even in the absence of parameter estimation for many types of stochastic processes in common use.

stat.ME↗

arXiv2022

Optimal data-driven hiring with equity for underrepresented groups

We present a data-driven prescriptive framework for fair decisions, motivated by hiring. An employer evaluates a set of applicants based on their observable attributes. The goal is to hire the best candidates while avoiding bias with regard to a certain protected attribute. Simply ignoring the protected attribute will not eliminate bias due to correlations in the data. We present a hiring policy that depends on the protected attribute functionally, but not statistically, and we prove that, among all possible fair policies, ours is optimal with respect to the firm's objective. We test our approach on both synthetic and real data, and find that it shows great practical potential to improve equity for underrepresented and historically marginalized groups.

econ.EM↗

arXiv2021

Distributional conformal prediction

We propose a robust method for constructing conditionally valid prediction intervals based on models for conditional distributions such as quantile and distribution regression. Our approach can be applied to important prediction problems including cross-sectional prediction, k-step-ahead forecasts, synthetic controls and counterfactual prediction, and individual treatment effects prediction. Our method exploits the probability integral transform and relies on permuting estimated ranks. Unlike regression residuals, ranks are independent of the predictors, allowing us to construct conditionally valid prediction intervals under heteroskedasticity. We establish approximate conditional validity under consistent estimation and provide approximate unconditional validity under model misspecification, overfitting, and with time series data. We also propose a simple "shape" adjustment of our baseline method that yields optimal prediction intervals.

econ.EM↗

arXiv2021

An Exact and Robust Conformal Inference Method for Counterfactual and Synthetic Controls

We introduce new inference procedures for counterfactual and synthetic control methods for policy evaluation. We recast the causal inference problem as a counterfactual prediction and a structural breaks testing problem. This allows us to exploit insights from conformal prediction and structural breaks testing to develop permutation inference procedures that accommodate modern high-dimensional estimators, are valid under weak and easy-to-verify conditions, and are provably robust against misspecification. Our methods work in conjunction with many different approaches for predicting counterfactual mean outcomes in the absence of the policy intervention. Examples include synthetic controls, difference-in-differences, factor and matrix completion models, and (fused) time series panel data models. Our approach demonstrates an excellent small-sample performance in simulations and is taken to a data application where we re-evaluate the consequences of decriminalizing indoor prostitution. Open-source software for implementing our conformal inference methods is available.

econ.EM↗

arXiv2021

Phase transition of the monotonicity assumption in learning local average treatment effects

We consider the setting in which a strong binary instrument is available for a binary treatment. The traditional LATE approach assumes the monotonicity condition stating that there are no defiers (or compliers). Since this condition is not always obvious, we investigate the sensitivity and testability of this condition. In particular, we focus on the question: does a slight violation of monotonicity lead to a small problem or a big problem? We find a phase transition for the monotonicity condition. On one of the boundary of the phase transition, it is easy to learn the sign of LATE and on the other side of the boundary, it is impossible to learn the sign of LATE. Unfortunately, the impossible side of the phase transition includes data-generating processes under which the proportion of defiers tends to zero. This boundary of phase transition is explicitly characterized in the case of binary outcomes. Outside a special case, it is impossible to test whether the data-generating process is on the nice side of the boundary. However, in the special case that the non-compliance is almost one-sided, such a test is possible. We also provide simple alternatives to monotonicity.

econ.EM↗

arXiv2021

Comments on Leo Breiman's paper 'Statistical Modeling: The Two Cultures' (Statistical Science, 2001, 16(3), 199-231)

Breiman challenged statisticians to think more broadly, to step into the unknown, model-free learning world, with him paving the way forward. Statistics community responded with slight optimism, some skepticism, and plenty of disbelief. Today, we are at the same crossroad anew. Faced with the enormous practical success of model-free, deep, and machine learning, we are naturally inclined to think that everything is resolved. A new frontier has emerged; the one where the role, impact, or stability of the {\it learning} algorithms is no longer measured by prediction quality, but an inferential one -- asking the questions of {\it why} and {\it if} can no longer be safely ignored.

stat.ML↗

arXiv2019

Testability of high-dimensional linear models with non-sparse structures

Understanding statistical inference under possibly non-sparse high-dimensional models has gained much interest recently. For a given component of the regression coefficient, we show that the difficulty of the problem depends on the sparsity of the corresponding row of the precision matrix of the covariates, not the sparsity of the regression coefficients. We develop new concepts of uniform and essentially uniform non-testability that allow the study of limitations of tests across a broad set of alternatives. Uniform non-testability identifies a collection of alternatives such that the power of any test, against any alternative in the group, is asymptotically at most equal to the nominal size. Implications of the new constructions include new minimax testability results that, in sharp contrast to the current results, do not depend on the sparsity of the regression parameters. We identify new tradeoffs between testability and feature correlation. In particular, we show that, in models with weak feature correlations, minimax lower bound can be attained by a test whose power has the $\sqrt{n}$ rate, regardless of the size of the model sparsity.

math.ST↗

arXiv2019

How well can we learn large factor models without assuming strong factors?

In this paper, we consider the problem of learning models with a latent factor structure. The focus is to find what is possible and what is impossible if the usual strong factor condition is not imposed. We study the minimax rate and adaptivity issues in two problems: pure factor models and panel regression with interactive fixed effects. For pure factor models, if the number of factors is known, we develop adaptive estimation and inference procedures that attain the minimax rate. However, when the number of factors is not specified a priori, we show that there is a tradeoff between validity and efficiency: any confidence interval that has uniform validity for arbitrary factor strength has to be conservative; in particular its width is bounded away from zero even when the factors are strong. Conversely, any data-driven confidence interval that does not require as an input the exact number of factors (including weak ones) and has shrinking width under strong factors does not have uniform coverage and the worst-case coverage probability is at most 1/2. For panel regressions with interactive fixed effects, the tradeoff is much better. We find that the minimax rate for learning the regression coefficient does not depend on the factor strength and propose a simple estimator that achieves this rate. However, when weak factors are allowed, uncertainty in the number of factors can cause a great loss of efficiency although the rate is not affected. In most cases, we find that the strong factor condition (and/or exact knowledge of number of factors) improves efficiency, but this condition needs to be imposed by faith and cannot be verified in data for inference purposes.

math.ST↗

arXiv2019

Learning non-smooth models: instrumental variable quantile regressions and related problems

This paper proposes computationally efficient methods that can be used for instrumental variable quantile regressions (IVQR) and related methods with statistical guarantees. This is much needed when we investigate heterogenous treatment effects since interactions between the endogenous treatment and control variables lead to an increased number of endogenous covariates. We prove that the GMM formulation of IVQR is NP-hard and finding an approximate solution is also NP-hard. Hence, solving the problem from a purely computational perspective seems unlikely. Instead, we aim to obtain an estimate that has good statistical properties and is not necessarily the global solution of any optimization problem. The proposal consists of employing $k$-step correction on an initial estimate. The initial estimate exploits the latest advances in mixed integer linear programming and can be computed within seconds. One theoretical contribution is that such initial estimators and Jacobian of the moment condition used in the k-step correction need not be even consistent and merely $k=4\log n$ fast iterations are needed to obtain an efficient estimator. The overall proposal scales well to handle extremely large sample sizes because lack of consistency requirement allows one to use a very small subsample to obtain the initial estimate and the k-step iterations on the full sample can be implemented efficiently. Another contribution that is of independent interest is to propose a tuning-free estimation for the Jacobian matrix, whose definition nvolves conditional densities. This Jacobian estimator generalizes bootstrap quantile standard errors and can be efficiently computed via closed-end solutions. We evaluate the performance of the proposal in simulations and an empirical example on the heterogeneous treatment effect of Job Training Partnership Act.

econ.EM↗

arXiv2019

Inference for Heterogeneous Effects using Low-Rank Estimation of Factor Slopes

We study a panel data model with general heterogeneous effects where slopes are allowed to vary across both individuals and over time. The key dimension reduction assumption we employ is that the heterogeneous slopes can be expressed as having a factor structure so that the high-dimensional slope matrix is low-rank and can thus be estimated using low-rank regularized regression. We provide a simple multi-step estimation procedure for the heterogeneous effects. The procedure makes use of sample-splitting and orthogonalization to accommodate inference following the use of penalized low-rank estimation. We formally verify that the resulting estimator is asymptotically normal allowing simple construction of inferential statements for {the individual-time-specific effects and for cross-sectional averages of these effects}. We illustrate the proposed method in simulation experiments and by estimating the effect of the minimum wage on employment.

math.ST↗