Searcharxiv⌕ Search

arXiv subjects

Fang Han

Publications and source records attributed to Fang Han.

At least 55 records · Page 3Linked to original sources
arXiv2020

High dimensional consistent independence testing with maxima of rank correlations

Testing mutual independence for high-dimensional observations is a fundamental statistical challenge. Popular tests based on linear and simple rank correlations are known to be incapable of detecting non-linear, non-monotone relationships, calling for methods that can account for such dependences. To address this challenge, we propose a family of tests that are constructed using maxima of pairwise rank correlations that permit consistent assessment of pairwise independence. Built upon a newly developed Cramér-type moderate deviation theorem for degenerate U-statistics, our results cover a variety of rank correlations including Hoeffding's $D$, Blum-Kiefer-Rosenblatt's $R$, and Bergsma-Dassios-Yanagimoto's $τ^*$. The proposed tests are distribution-free in the class of multivariate distributions with continuous margins, implementable without the need for permutation, and are shown to be rate-optimal against sparse alternatives under the Gaussian copula model. As a by-product of the study, we reveal an identity between the aforementioned three rank correlation statistics, and hence make a step towards proving a conjecture of Bergsma and Dassios.

math.ST↗
arXiv2020

Optimal estimation of variance in nonparametric regression with random design

Consider the heteroscedastic nonparametric regression model with random design \begin{align*} Y_i = f(X_i) + V^{1/2}(X_i)\varepsilon_i, \quad i=1,2,\ldots,n, \end{align*} with $f(\cdot)$ and $V(\cdot)$ $α$- and $β$-Hölder smooth, respectively. We show that the minimax rate of estimating $V(\cdot)$ under both local and global squared risks is of the order \begin{align*} n^{-\frac{8αβ}{4αβ+ 2α+ β}} \vee n^{-\frac{2β}{2β+1}}, \end{align*} where $a\vee b := \max\{a,b\}$ for any two real numbers $a,b$. This result extends the fixed design rate $n^{-4α} \vee n^{-2β/(2β+1)}$ derived in Wang et al. [2008] in a non-trivial manner, as indicated by the appearances of both $α$ and $β$ in the first term. In the special case of constant variance, we show that the minimax rate is $n^{-8α/(4α+1)}\vee n^{-1}$ for variance estimation, which further implies the same rate for quadratic functional estimation and thus unifies the minimax rate under the nonparametric regression model with those under the density model and the white noise model. To achieve the minimax rate, we develop a U-statistic-based local polynomial estimator and a lower bound that is constructed over a specified distribution family of randomness designed for both $\varepsilon_i$ and $X_i$.

math.ST↗
arXiv2019

On rank estimators in increasing dimensions

The family of rank estimators, including Han's maximum rank correlation (Han, 1987) as a notable example, has been widely exploited in studying regression problems. For these estimators, although the linear index is introduced for alleviating the impact of dimensionality, the effect of large dimension on inference is rarely studied. This paper fills this gap via studying the statistical properties of a larger family of M-estimators, whose objective functions are formulated as U-processes and may be discontinuous in increasing dimension set-up where the number of parameters, $p_{n}$, in the model is allowed to increase with the sample size, $n$. First, we find that often in estimation, as $p_{n}/n\rightarrow 0$, $(p_{n}/n)^{1/2}$ rate of convergence is obtainable. Second, we establish Bahadur-type bounds and study the validity of normal approximation, which we find often requires a much stronger scaling requirement than $p_{n}^{2}/n\rightarrow 0.$ Third, we state conditions under which the numerical derivative estimator of asymptotic covariance matrix is consistent, and show that the step size in implementing the covariance estimator has to be adjusted with respect to $p_{n}$. All theoretical results are further backed up by simulation studies.

math.ST↗
arXiv2019

On Estimation of Isotonic Piecewise Constant Signals

Consider a sequence of real data points $X_1,\ldots, X_n$ with underlying means $θ^*_1,\dots,θ^*_n$. This paper starts from studying the setting that $θ^*_i$ is both piecewise constant and monotone as a function of the index $i$. For this, we establish the exact minimax rate of estimating such monotone functions, and thus give a non-trivial answer to an open problem in the shape-constrained analysis literature. The minimax rate involves an interesting iterated logarithmic dependence on the dimension, a phenomenon that is revealed through characterizing the interplay between the isotonic shape constraint and model selection complexity. We then develop a penalized least-squares procedure for estimating the vector $θ^*=(θ^*_1,\dots,θ^*_n)^T$. This estimator is shown to achieve the derived minimax rate adaptively. For the proposed estimator, we further allow the model to be misspecified and derive oracle inequalities with the optimal rates, and show there exists a computationally efficient algorithm to compute the exact solution.

math.ST↗
arXiv2019

Probability inequalities for high dimensional time series under a triangular array framework

Study of time series data often involves measuring the strength of temporal dependence, on which statistical properties like consistency and central limit theorem are built. Historically, various dependence measures have been proposed. In this note, we first survey some of the most well-used dependence measures as well as various probability and moment inequalities built upon them under a high-dimensional triangular array time series setting. We then argue that this triangular array setting will pose substantially new challenges to the verification of some dependence conditions. In particular, ``textbook results" could now be misleading, and hence are recommended to be used with caution.

math.ST↗
arXiv2019

Asymptotic joint distribution of extreme eigenvalues and trace of large sample covariance matrix in a generalized spiked population model

This paper studies the joint limiting behavior of extreme eigenvalues and trace of large sample covariance matrix in a generalized spiked population model, where the asymptotic regime is such that the dimension and sample size grow proportionally. The form of the joint limiting distribution is applied to conduct Johnson-Graybill-type tests, a family of approaches testing for signals in a statistical model. For this, higher order correction is further made, helping alleviate the impact of finite-sample bias. The proof rests on determining the joint asymptotic behavior of two classes of spectral processes, corresponding to the extreme and linear spectral statistics respectively.

math.ST↗
arXiv2019

Moment bounds for large autocovariance matrices under dependence

The goal of this paper is to obtain expectation bounds for the deviation of large sample autocovariance matrices from their means under weak data dependence. While the accuracy of covariance matrix estimation corresponding to independent data has been well understood, much less is known in the case of dependent data. We make a step towards filling this gap, and establish deviation bounds that depend only on the parameters controlling the "intrinsic dimension" of the data up to some logarithmic terms. Our results have immediate impacts on high dimensional time series analysis, and we apply them to high dimensional linear VAR($d$) model, vector-valued ARCH model, and a model used in Banna et al. (2016).

math.ST↗
arXiv2019

Neural network an1alysis of sleep stages enables efficient diagnosis of narcolepsy

Analysis of sleep for the diagnosis of sleep disorders such as Type-1 Narcolepsy (T1N) currently requires visual inspection of polysomnography records by trained scoring technicians. Here, we used neural networks in approximately 3,000 normal and abnormal sleep recordings to automate sleep stage scoring, producing a hypnodensity graph - a probability distribution conveying more information than classical hypnograms. Accuracy of sleep stage scoring was validated in 70 subjects assessed by six scorers. The best model performed better than any individual scorer (87% versus consensus). It also reliably scores sleep down to 5 instead of 30 second scoring epochs. A T1N marker based on unusual sleep-stage overlaps achieved a specificity of 96% and a sensitivity of 91%, validated in independent datasets. Addition of HLA-DQB1*06:02 typing increased specificity to 99%. Our method can reduce time spent in sleep clinics and automates T1N diagnosis. It also opens the possibility of diagnosing T1N using home sleep studies.

cs.NE↗
arXiv2019

Tail behavior of dependent V-statistics and its applications

We establish exponential inequalities and Cramer-type moderate deviation theorems for a class of V-statistics under strong mixing conditions. Our theory is developed via kernel expansion based on random Fourier features. This type of expansion is new and useful for handling many notorious classes of kernels. While the developed theory has a number of applications, we apply it to lasso-type semiparametric regression estimation and high-dimensional multiple hypothesis testing.

math.ST↗
arXiv2018

An Extreme-Value Approach for Testing the Equality of Large U-Statistic Based Correlation Matrices

There has been an increasing interest in testing the equality of large Pearson's correlation matrices. However, in many applications it is more important to test the equality of large rank-based correlation matrices since they are more robust to outliers and nonlinearity. Unlike the Pearson's case, testing the equality of large rank-based statistics has not been well explored and requires us to develop new methods and theory. In this paper, we provide a framework for testing the equality of two large U-statistic based correlation matrices, which include the rank-based correlation matrices as special cases. Our approach exploits extreme value statistics and the Jackknife estimator for uncertainty assessment and is valid under a fully nonparametric model. Theoretically, we develop a theory for testing the equality of U-statistic based correlation matrices. We then apply this theory to study the problem of testing large Kendall's tau correlation matrices and demonstrate its optimality. For proving this optimality, a novel construction of least favourable distributions is developed for the correlation matrix comparison.

math.ST↗
arXiv2018

On inference validity of weighted U-statistics under data heterogeneity

Motivated by challenges on studying a new correlation measurement being popularized in evaluating online ranking algorithms' performance, this manuscript explores the validity of uncertainty assessment for weighted U-statistics. Without any commonly adopted assumption, we verify Efron's bootstrap and a new resampling procedure's inference validity. Specifically, in its full generality, our theory allows both kernels and weights asymmetric and data points not identically distributed, which are all new issues that historically have not been addressed. For achieving strict generalization, for example, we have to carefully control the order of the "degenerate" term in U-statistics which are no longer degenerate under the empirical measure for non-i.i.d. data. Our result applies to the motivating task, giving the region at which solid statistical inference can be made.

math.ST↗
arXiv2018

Pairwise Difference Estimation of High Dimensional Partially Linear Model

This paper proposes a regularized pairwise difference approach for estimating the linear component coefficient in a partially linear model, with consistency and exact rates of convergence obtained in high dimensions under mild scaling requirements. Our analysis reveals interesting features such as (i) the bandwidth parameter automatically adapts to the model and is actually tuning-insensitive; and (ii) the procedure could even maintain fast rate of convergence for $α$-Hölder class of $α\leq1/2$. Simulation studies show the advantage of the proposed method, and application of our approach to a brain imaging data reveals some biological patterns which fail to be recovered using competing methods.

math.ST↗
arXiv2017

Distribution-Free Tests of Independence in High Dimensions

We consider the testing of mutual independence among all entries in a $d$-dimensional random vector based on $n$ independent observations. We study two families of distribution-free test statistics, which include Kendall's tau and Spearman's rho as important examples. We show that under the null hypothesis the test statistics of these two families converge weakly to Gumbel distributions, and propose tests that control the type I error in the high-dimensional setting where $d>n$. We further show that the two tests are rate-optimal in terms of power against sparse alternatives, and outperform competitors in simulations, especially when $d$ is large.

math.ST↗
arXiv2017

A Provable Smoothing Approach for High Dimensional Generalized Regression with Applications in Genomics

In many applications, linear models fit the data poorly. This article studies an appealing alternative, the generalized regression model. This model only assumes that there exists an unknown monotonically increasing link function connecting the response $Y$ to a single index $X^Tβ^*$ of explanatory variables $X\in\mathbb{R}^d$. The generalized regression model is flexible and covers many widely used statistical models. It fits the data generating mechanisms well in many real problems, which makes it useful in a variety of applications where regression models are regularly employed. In low dimensions, rank-based M-estimators are recommended to deal with the generalized regression model, giving root-$n$ consistent estimators of $β^*$. Applications of these estimators to high dimensional data, however, are questionable. This article studies, both theoretically and practically, a simple yet powerful smoothing approach to handle the high dimensional generalized regression model. Theoretically, a family of smoothing functions is provided, and the amount of smoothing necessary for efficient inference is carefully calculated. Practically, our study is motivated by an important and challenging scientific problem: decoding gene regulation by predicting transcription factors that bind to cis-regulatory elements. Applying our proposed method to this problem shows substantial improvement over the state-of-the-art alternative in real data.

stat.ME↗
arXiv2017

On Gaussian Comparison Inequality and Its Application to Spectral Analysis of Large Random Matrices

Recently, Chernozhukov, Chetverikov, and Kato [Ann. Statist. 42 (2014) 1564--1597] developed a new Gaussian comparison inequality for approximating the suprema of empirical processes. This paper exploits this technique to devise sharp inference on spectra of large random matrices. In particular, we show that two long-standing problems in random matrix theory can be solved: (i) simple bootstrap inference on sample eigenvalues when true eigenvalues are tied; (ii) conducting two-sample Roy's covariance test in high dimensions. To establish the asymptotic results, a generalized $ε$-net argument regarding the matrix rescaled spectral norm and several new empirical process bounds are developed and of independent interest.

math.ST↗
arXiv2017

Testing the science/technology relationship by analysis of patent citations of scientific papers after decomposition of both science and technology

The relationship of scientific knowledge development to technological development is widely recognized as one of the most important and complex aspects of technological evolution. This paper adds to our understanding of the relationship through use of a more rigorous structure for differentiating among technologies based upon technological domains (defined as consisting of the artifacts over time that fulfill a specific generic function using a specific body of technical knowledge).

cs.DL↗
arXiv2016

An Exponential Inequality for U-Statistics under Mixing Conditions

The family of U-statistics plays a fundamental role in statistics. This paper proves a novel exponential inequality for U-statistics under the time series setting. Explicit mixing conditions are given for guaranteeing fast convergence, the bound proves to be analogous to the one under independence, and extension to non-stationary time series is straightforward. The proof relies on a novel decomposition of U-statistics via exploiting the temporal correlatedness structure. Such results are of interest in many fields where high dimensional time series data are present. In particular, applications to high dimensional time series inference are discussed.

math.ST↗