arXiv subjects

Guang Cheng

Publications and source records attributed to Guang Cheng.

At least 163 records · Page 9Linked to original sources

arXiv2019

Distributed Generalized Cross-Validation for Divide-and-Conquer Kernel Ridge Regression and its Asymptotic Optimality

Tuning parameter selection is of critical importance for kernel ridge regression. To this date, data driven tuning method for divide-and-conquer kernel ridge regression (d-KRR) has been lacking in the literature, which limits the applicability of d-KRR for large data sets. In this paper, by modifying the Generalized Cross-validation (GCV, Wahba, 1990) score, we propose a distributed Generalized Cross-Validation (dGCV) as a data-driven tool for selecting the tuning parameters in d-KRR. Not only the proposed dGCV is computationally scalable for massive data sets, it is also shown, under mild conditions, to be asymptotically optimal in the sense that minimizing the dGCV score is equivalent to minimizing the true global conditional empirical loss of the averaged function estimator, extending the existing optimality results of GCV to the divide-and-conquer framework.

stat.ML↗

arXiv2018

Bayesian Fusion Estimation via t-Shrinkage

Shrinkage prior has gained great successes in many data analysis, however, its applications mostly focus on the Bayesian modeling of sparse parameters. In this work, we will apply Bayesian shrinkage to model high dimensional parameter that possesses an unknown blocking structure. We propose to impose heavy-tail shrinkage prior, e.g., $t$ prior, on the differences of successive parameter entries, and such a fusion prior will shrink successive differences towards zero and hence induce posterior blocking. Comparing to conventional Bayesian fused lasso which implements Laplace fusion prior, $t$ fusion prior induces stronger shrinkage effect and enjoys a nice posterior consistency property. Simulation studies and real data analyses show that $t$ fusion has superior performance to the frequentist fusion estimator and Bayesian Laplace-fusion prior. This $t$-fusion strategy is further developed to conduct a Bayesian clustering analysis, and simulation shows that the proposed algorithm obtains better posterior distributional convergence than the classical Dirichlet process modeling.

stat.ME↗

arXiv2018

Distributed Nearest Neighbor Classification

Nearest neighbor is a popular nonparametric method for classification and regression with many appealing properties. In the big data era, the sheer volume and spatial/temporal disparity of big data may prohibit centrally processing and storing the data. This has imposed considerable hurdle for nearest neighbor predictions since the entire training data must be memorized. One effective way to overcome this issue is the distributed learning framework. Through majority voting, the distributed nearest neighbor classifier achieves the same rate of convergence as its oracle version in terms of both the regret and instability, up to a multiplicative constant that depends solely on the data dimension. The multiplicative difference can be eliminated by replacing majority voting with the weighted voting scheme. In addition, we provide sharp theoretical upper bounds of the number of subsamples in order for the distributed nearest neighbor classifier to reach the optimal convergence rate. It is interesting to note that the weighted voting scheme allows a larger number of subsamples than the majority voting one. Our findings are supported by numerical studies using both simulated and real data sets.

math.ST↗

arXiv2018

Statistical Optimality of Interpolated Nearest Neighbor Algorithms

In the era of deep learning, understanding over-fitting phenomenon becomes increasingly important. It is observed that carefully designed deep neural networks achieve small testing error even when the training error is close to zero. One possible explanation is that for many modern machine learning algorithms, over-fitting can greatly reduce the estimation bias, while not increasing the estimation variance too much. To illustrate the above idea, we prove that the proposed interpolated nearest neighbor algorithm achieves the minimax optimal rate in both regression and classification regimes, and observe that they are empirically better than the traditional $k$ nearest neighbor method in some cases.

stat.ML↗

arXiv2018

High Dimensional Robust Inference for Cox Regression Models

We consider high-dimensional inference for potentially misspecified Cox proportional hazard models based on low dimensional results by Lin and Wei [1989]. A de-sparsified Lasso estimator is proposed based on the log partial likelihood function and shown to converge to a pseudo-true parameter vector. Interestingly, the sparsity of the true parameter can be inferred from that of the above limiting parameter. Moreover, each component of the above (non-sparse) estimator is shown to be asymptotically normal with a variance that can be consistently estimated even under model misspecifications. In some cases, this asymptotic distribution leads to valid statistical inference procedures, whose empirical performances are illustrated through numerical examples.

math.ST↗

arXiv2018

Early Stopping for Nonparametric Testing

Early stopping of iterative algorithms is an algorithmic regularization method to avoid over-fitting in estimation and classification. In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup. Specifically, a Wald-type test statistic is obtained based on an iterated estimate produced by functional gradient descent algorithms in a reproducing kernel Hilbert space. A notable contribution is to establish a "sharp" stopping rule: when the number of iterations achieves an optimal order, testing optimality is achievable; otherwise, testing optimality becomes impossible. As a by-product, a similar sharpness result is also derived for minimax optimal estimation under early stopping studied in [11] and [19]. All obtained results hold for various kernel classes, including Sobolev smoothness classes and Gaussian kernel classes.

math.ST↗

arXiv2018

Statistically and Computationally Efficient Variance Estimator for Kernel Ridge Regression

In this paper, we propose a random projection approach to estimate variance in kernel ridge regression. Our approach leads to a consistent estimator of the true variance, while being computationally more efficient. Our variance estimator is optimal for a large family of kernels, including cubic splines and Gaussian kernels. Simulation analysis is conducted to support our theory.

math.ST↗

arXiv2018

Quadratic Discriminant Analysis under Moderate Dimension

Quadratic discriminant analysis (QDA) is a simple method to classify a subject into two populations, and was proven to perform as well as the Bayes rule when the data dimension p is fixed. The main purpose of this paper is to examine the empirical and theoretical behaviors of QDA where p grows proportionally to the sample sizes without imposing any structural assumption on the parameters. The first finding in this moderate dimension regime is that QDA can perform as poorly as random guessing even when the two populations deviate significantly. This motivates a generalized version of QDA that automatically adapts to dimensionality. Under a finite fourth moment condition, we derive misclassification rates for both the generalized QDA and the optimal one. A direct comparison reveals one "easy" case where the difference between two rates converges to zero and one "hard" case where that converges to some strictly positive constant. For the latter, a divide-and-conquer approach over dimension (rather than sample) followed by a screening procedure is proposed to narrow the gap. Various numerical studies are conducted to back up the proposed methodology.

math.ST↗

arXiv2018

Distributed inference for quantile regression processes

The increased availability of massive data sets provides a unique opportunity to discover subtle patterns in their distributions, but also imposes overwhelming computational challenges. To fully utilize the information contained in big data, we propose a two-step procedure: (i) estimate conditional quantile functions at different levels in a parallel computing environment; (ii) construct a conditional quantile regression process through projection based on these estimated quantile curves. Our general quantile regression framework covers both linear models with fixed or growing dimension and series approximation models. We prove that the proposed procedure does not sacrifice any statistical inferential accuracy provided that the number of distributed computing units and quantile levels are chosen properly. In particular, a sharp upper bound for the former and a sharp lower bound for the latter are derived to capture the minimal computational cost from a statistical perspective. As an important application, the statistical inference on conditional distribution functions is considered. Moreover, we propose computationally efficient approaches to conducting inference in the distributed estimation setting described above. Those approaches directly utilize the availability of estimators from sub-samples and can be carried out at almost no additional computational cost. Simulations confirm our statistical inferential theory.

math.ST↗

arXiv2018

Nonparametric Testing under Random Projection

A common challenge in nonparametric inference is its high computational complexity when data volume is large. In this paper, we develop computationally efficient nonparametric testing by employing a random projection strategy. In the specific kernel ridge regression setup, a simple distance-based test statistic is proposed. Notably, we derive the minimum number of random projections that is sufficient for achieving testing optimality in terms of the minimax rate. An adaptive testing procedure is further established without prior knowledge of regularity. One technical contribution is to establish upper bounds for a range of tail sums of empirical kernel eigenvalues. Simulations and real data analysis are conducted to support our theory.

math.ST↗

arXiv2018

Minimax Optimal Estimation in Partially Linear Additive Models under High Dimension

In this paper, we derive minimax rates for estimating both parametric and nonparametric components in partially linear additive models with high dimensional sparse vectors and smooth functional components. The minimax lower bound for Euclidean components is the typical sparse estimation rate that is independent of nonparametric smoothness indices. However, the minimax lower bound for each component function exhibits an interplay between the dimensionality and sparsity of the parametric component and the smoothness of the relevant nonparametric component. Indeed, the minimax risk for smooth nonparametric estimation can be slowed down to the sparse estimation rate whenever the smoothness of the nonparametric component or dimensionality of the parametric component is suffciently large. In the above setting, we demonstrate that penalized least square estimators can nearly achieve minimax lower bounds.

math.ST↗

arXiv2018

Simultaneous Clustering and Estimation of Heterogeneous Graphical Models

We consider joint estimation of multiple graphical models arising from heterogeneous and high-dimensional observations. Unlike most previous approaches which assume that the cluster structure is given in advance, an appealing feature of our method is to learn cluster structure while estimating heterogeneous graphical models. This is achieved via a high dimensional version of Expectation Conditional Maximization (ECM) algorithm (Meng and Rubin, 1993). A joint graphical lasso penalty is imposed on the conditional maximization step to extract both homogeneity and heterogeneity components across all clusters. Our algorithm is computationally efficient due to fast sparse learning routines and can be implemented without unsupervised learning knowledge. The superior performance of our method is demonstrated by extensive experiments and its application to a Glioblastoma cancer dataset reveals some new insights in understanding the Glioblastoma cancer. In theory, a non-asymptotic error bound is established for the output directly from our high dimensional ECM algorithm, and it consists of two quantities: statistical error (statistical accuracy) and optimization error (computational complexity). Such a result gives a theoretical guideline in terminating our ECM iterations.

stat.ML↗

arXiv2017

Influence of Bi addition on the property of Ag-Bi nano-composite coatings

Silver (Ag) coatings have been widely used in many industry areas due to their excellent conductivity. However, wider applications of Ag coatings have been hindered by their poor mechanical properties. In this research, to improve the mechanical performance, Ag-Bi nano-composite coatings were prepared by a novel ionic co-discharge method. A systematic study of the microstructure, mechanical properties, electrical conductivity and antibacterial behavior of the resulting coating was performed. The results indicated that after adding an appropriate amount of Bi containing solution into the Ag plating solution, Ag-Bi nano-particles were in-situ formed and distributed uniformly throughout the coating matrix, resulting in a significant improvement in the mechanical properties. The hardness of Ag-Bi coating was increased by 60% compared to that of the pure Ag coating. The corrosion resistance of Ag-Bi coatings was also enhanced. The outcome of this research may find a broader application in electronics, jewelry, aerospace and other industries.

cond-mat.mtrl-sci↗

arXiv2017

Microstructure and properties of Cu-Sn-Zn-TiO2 Nano-composite coatings on mild steel

Cu-Sn-Zn coatings have been widely used in industry for their unique properties, such as good conductivity, high corrosion resistance and excellent solderability. To further improve the mechanical performance of Cu-Sn-Zn coatings, powder-enhanced method was applied in the current study and Cu-Sn-Zn-TiO2 nano-composite coatings with different TiO2 concentration were fabricated. The microstructure of Cu-Sn-Zn-TiO2 nano-composite coatings were investigated by X-ray diffraction (XRD) and Scanning Electron Microscopy (SEM). The mechanical properties of coatings including microhardness and wear resistance were studied. The results indicate that the incorporation of TiO2 nanoparticle can significantly influence the properties of Cu-Sn-Zn coatings. The microhardness of Cu-Sn-Zn coating was increased to 383 HV from 330 HV with 1g/L TiO2 addition. Also, the corrosion resistance of coating was enhanced. The effects of TiO2 nanoparticle concentration on the microstructure, mechanical properties and corrosion resistance of Cu-Sn-Zn-TiO2 nano-composite coatings were discussed.

physics.app-ph↗

arXiv2017

Gaussian Approximation of General Nonparametric Posterior Distributions

In a general class of Bayesian nonparametric models, we prove that the posterior distribution can be asymptotically approximated by a Gaussian process. Our results apply to nonparametric exponential family that contains both Gaussian and non-Gaussian regression, and also hold for both efficient (root-n) and inefficient (non root-n) estimation. Our general approximation theorem does not rely on posterior conjugacy, and can be verified in a class of Gaussian process priors that has a smoothing spline interpretation [59, 44]. In particular, the limiting posterior measure becomes prior-free under a Bayesian version of "under-smoothing" condition. Finally, we apply our approximation theorem to examine the asymptotic frequentist properties of Bayesian procedures such as credible regions and credible intervals.

math.ST↗

arXiv2017

High Dimensional Inference in Partially Linear Models

We propose two semiparametric versions of the debiased Lasso procedure for the model $Y_i = X_iβ_0 + g_0(Z_i) + ε_i$, where $β_0$ is high dimensional but sparse (exactly or approximately). Both versions are shown to have the same asymptotic normal distribution and do not require the minimal signal condition for statistical inference of any component in $β_0$. Our method also works when $Z_i$ is high dimensional provided that the function classes $E(X_{ij} |Z_i)$s and $E(Y_i|Z_i)$ belong to exhibit certain sparsity features, e.g., a sparse additive decomposition structure. We further develop a simultaneous hypothesis testing procedure based on multiplier bootstrap. Our testing method automatically takes into account of the dependence structure within the debiased estimates, and allows the number of tested components to be exponentially high.

math.ST↗

arXiv2017

Computational Limits of A Distributed Algorithm For Smoothing Spline

In this paper, we explore statistical versus computational trade-off to address a basic question in the application of a distributed algorithm: what is the minimal computational cost in obtaining statistical optimality? In smoothing spline setup, we observe a phase transition phenomenon for the number of deployed machines that ends up being a simple proxy for computing cost. Specifically, a sharp upper bound for the number of machines is established: when the number is below this bound, statistical optimality (in terms of nonparametric estimation or testing) is achievable; otherwise, statistical optimality becomes impossible. These sharp bounds partly capture intrinsic computational limits of the distributed algorithm considered in this paper, and turn out to be fully determined by the smoothness of the regression function. As a side remark, we argue that sample splitting may be viewed as an alternative form of regularization, playing a similar role as smoothing parameter.

math.ST↗

arXiv2017

Quantile Processes for Semi and Nonparametric Regression

A collection of quantile curves provides a complete picture of conditional distributions. Properly centered and scaled versions of estimated curves at various quantile levels give rise to the so-called quantile regression process (QRP). In this paper, we establish weak convergence of QRP in a general series approximation framework, which includes linear models with increasing dimension, nonparametric models and partial linear models. An interesting consequence is obtained in the last class of models, where parametric and non-parametric estimators are shown to be asymptotically independent. Applications of our general process convergence results include the construction of non-crossing quantile curves and the estimation of conditional distribution functions. As a result of independent interest, we obtain a series of Bahadur representations with exponential bounds for tail probabilities of all remainder terms. Bounds of this kind are potentially useful in analyzing statistical inference procedures under divide-and-conquer setup.

math.ST↗