arXiv subjects

Igor Silin

Publications and source records attributed to Igor Silin.

4 recordsLinked to original sources

arXiv2020

Canonical thresholding for non-sparse high-dimensional linear regression

We consider a high-dimensional linear regression problem. Unlike many papers on the topic, we do not require sparsity of the regression coefficients; instead, our main structural assumption is a decay of eigenvalues of the covariance matrix of the data. We propose a new family of estimators, called the canonical thresholding estimators, which pick largest regression coefficients in the canonical form. The estimators admit an explicit form and can be linked to LASSO and Principal Component Regression (PCR). A theoretical analysis for both fixed design and random design settings is provided. Obtained bounds on the mean squared error and the prediction error of a specific estimator from the family allow to clearly state sufficient conditions on the decay of eigenvalues to ensure convergence. In addition, we promote the use of the relative errors, strongly linked with the out-of-sample $R^2$. The study of these relative errors leads to a new concept of joint effective dimension, which incorporates the covariance of the data and the regression coefficients simultaneously, and describes the complexity of a linear regression problem. Some minimax lower bounds are established to showcase the optimality of our procedure. Numerical simulations confirm good performance of the proposed estimators compared to the previously developed methods.

math.ST↗

arXiv2020

Hypothesis testing for eigenspaces of covariance matrix

Eigenspaces of covariance matrices play an important role in statistical machine learning, arising in variety of modern algorithms. Quantitatively, it is convenient to describe the eigenspaces in terms of spectral projectors. This work focuses on hypothesis testing for the spectral projectors, both in one- and two-sample scenario. We present new tests, based on a specific matrix norm developed in order to utilize the structure of the spectral projectors. A new resampling technique of independent interest is introduced and analyzed: it serves as an alternative to the well-known multiplier bootstrap, significantly reducing computational complexity of bootstrap-based methods. We provide theoretical guarantees for the type-I error of our procedures, which remarkably improve the previously obtained results in the field. Moreover, we analyze power of our tests. Numerical experiments illustrate good performance of the proposed methods compared to previously developed ones.

math.ST↗

arXiv2019

Finite sample Bernstein-von Mises theorems for functionals and spectral projectors of the covariance matrix

We demonstrate that a prior influence on the posterior distribution of covariance matrix vanishes as sample size grows. The assumptions on a prior are explicit and mild. The results are valid for a finite sample and admit the dimension $p$ growing with the sample size $n$. We exploit the described fact to derive the finite sample Bernstein-von Mises theorem for functionals of covariance matrix (e.g. eigenvalues) and to find the posterior distribution of the Frobenius distance between spectral projector and empirical spectral projector. This can be useful for constructing sharp confidence sets for the true value of the functional or for the true spectral projector.

math.ST↗

arXiv2019

Bayesian inference for spectral projectors of the covariance matrix

Let $X_1, \ldots, X_n$ be i.i.d. sample in $\mathbb{R}^p$ with zero mean and the covariance matrix $\mathbf{Σ^*}$. The classical PCA approach recovers the projector $\mathbf{P^*_{\mathcal{J}}}$ onto the principal eigenspace of $\mathbf{Σ^*}$ by its empirical counterpart $\mathbf{\widehat{P}_{\mathcal{J}}}$. Recent paper [Koltchinskii, Lounici (2017)] investigated the asymptotic distribution of the Frobenius distance between the projectors $\| \mathbf{\widehat{P}_{\mathcal{J}}} - \mathbf{P^*_{\mathcal{J}}} \|_2$, while [Naumov et al. (2017)] offered a bootstrap procedure to measure uncertainty in recovering this subspace $\mathbf{P^*_{\mathcal{J}}}$ even in a finite sample setup. The present paper considers this problem from a Bayesian perspective and suggests to use the credible sets of the pseudo-posterior distribution on the space of covariance matrices induced by the conjugated Inverse Wishart prior as sharp confidence sets. This yields a numerically efficient procedure. Moreover, we theoretically justify this method and derive finite sample bounds on the corresponding coverage probability. Contrary to [Koltchinskii, Lounici (2017), Naumov et al. (2017)], the obtained results are valid for non-Gaussian data: the main assumption that we impose is the concentration of the sample covariance $\mathbf{\widehatΣ}$ in a vicinity of $\mathbf{Σ^*}$. Numerical simulations illustrate good performance of the proposed procedure even on non-Gaussian data in a rather challenging regime.

math.ST↗