arXiv subjects

Ilya Soloveychik

Publications and source records attributed to Ilya Soloveychik.

29 records · Page 2Linked to original sources

arXiv2017

A Note on the Concentration of Spectral Measure of Wigner's Matrices

In this short note we derive concentration inequalities for the empirical absolute moments of square symmetric matrices with independent symmetrically distributed +/-1 entries. Most of the previous results of this type are limited to functions with bounded Lipschitz constant, and therefore exclude the moments from consideration. Based on the fine analysis of the distribution of the largest eigenvalues developed by Soshnikov and Sinai, we extend the measure concentration tools of Talagrand to encompass power functions.

math.PR↗

arXiv2016

Simultaneous penalized M-estimation of covariance matrices using geodesically convex optimization

A common assumption when sampling $p$-dimensional observations from $K$ distinct group is the equality of the covariance matrices. In this paper, we propose two penalized $M$-estimation approaches for the estimation of the covariance or scatter matrices under the broader assumption that they may simply be close to each other, and hence roughly deviate from some positive definite "center". The first approach begins by generating a pooled $M$-estimator of scatter based on all the data, followed by a penalised $M$-estimator of scatter for each group, with the penalty term chosen so that the individual scatter matrices are shrunk towards the pooled scatter matrix. In the second approach, we minimize the sum of the individual group $M$-estimation cost functions together with an additive joint penalty term which enforces some similarity between the individual scatter estimators, i.e. shrinkage towards a mutual center. In both approaches, we utilize the concept of geodesic convexity to prove the existence and uniqueness of the penalized solution under general conditions. We consider three specific penalty functions based on the Euclidean, the Riemannian, and the Kullback-Leibler distances. In the second approach, the distance based penalties are shown to lead to estimators of the mutual center that are related to the arithmetic, the Riemannian and the harmonic means of positive definite matrices, respectively. A penalty based on an ellipticity measure is also considered which is particularly useful for shape matrix estimators. Fixed point equations are derived for each penalty function and the benefits of the estimators are illustrated in regularized discriminant analysis problem.

stat.ME↗

arXiv2016

Gaussian and Robust Kronecker Product Covariance Estimation: Existence and Uniqueness

We study the Gaussian and robust covariance estimation, assuming the true covariance matrix to be a Kronecker product of two lower dimensional square matrices. In both settings we define the estimators as solutions to the constrained maximum likelihood programs. In the robust case, we consider Tyler's estimator defined as the maximum likelihood estimator of a certain distribution on a sphere. We develop tight sufficient conditions for the existence and uniqueness of the estimates and show that in the Gaussian scenario with the unknown mean, $p/q+q/p + 2$ samples are almost surely enough to guarantee the existence and uniqueness, where $p$ and $q$ are the dimensions of the Kronecker product factors. In the robust case with the known mean, the corresponding sufficient number of samples is $\max[p/q, q/p] + 1$.

stat.AP↗

arXiv2015

Joint Inverse Covariances Estimation with Mutual Linear Structure

We consider the problem of joint estimation of structured inverse covariance matrices. We perform the estimation using groups of measurements with different covariances of the same unknown structure. Assuming the inverse covariances to span a low dimensional linear subspace in the space of symmetric matrices, our aim is to determine this structure. It is then utilized to improve the estimation of the inverse covariances. We propose a novel optimization algorithm discovering and exploiting the underlying structure and provide its efficient implementation. Numerical simulations are presented to illustrate the performance benefits of the proposed algorithm.

stat.ML↗

arXiv2015

Group Symmetric Robust Covariance Estimation

In this paper we consider Tyler's robust covariance M-estimator under group symmetry constraints. We assume that the covariance matrix is invariant to the conjugation action of a unitary matrix group, referred to as group symmetry. Examples of group symmetric structures include circulant, perHermitian and proper quaternion matrices. We introduce a group symmetric version of Tyler's estimator (STyler) and provide an iterative fixed point algorithm to compute it. The classical results claim that at least n=p+1 sample points in general position are necessary to ensure the existence and uniqueness of Tyler's estimator, where p is the ambient dimension. We show that the STyler requires significantly less samples. In some groups even two samples are enough to guarantee its existence and uniqueness. In addition, in the case of elliptical populations, we provide high probability bounds on the error of the STyler. These too, quantify the advantage of exploiting the symmetry structure. Finally, these theoretical results are supported by numerical simulations.ted by numerical simulations.

stat.AP↗

arXiv2015

Joint Covariance Estimation with Mutual Linear Structure

We consider the problem of joint estimation of structured covariance matrices. Assuming the structure is unknown, estimation is achieved using heterogeneous training sets. Namely, given groups of measurements coming from centered populations with different covariances, our aim is to determine the mutual structure of these covariance matrices and estimate them. Supposing that the covariances span a low dimensional affine subspace in the space of symmetric matrices, we develop a new efficient algorithm discovering the structure and using it to improve the estimation. Our technique is based on the application of principal component analysis in the matrix space. We also derive an upper performance bound of the proposed algorithm in the Gaussian scenario and compare it with the Cramer-Rao lower bound. Numerical simulations are presented to illustrate the performance benefits of the proposed method.

math.ST↗

arXiv2014

Tyler's Covariance Matrix Estimator in Elliptical Models with Convex Structure

We address structured covariance estimation in elliptical distributions by assuming that the covariance is a priori known to belong to a given convex set, e.g., the set of Toeplitz or banded matrices. We consider the General Method of Moments (GMM) optimization applied to robust Tyler's scatter M-estimator subject to these convex constraints. Unfortunately, GMM turns out to be non-convex due to the objective. Instead, we propose a new COCA estimator - a convex relaxation which can be efficiently solved. We prove that the relaxation is tight in the unconstrained case for a finite number of samples, and in the constrained case asymptotically. We then illustrate the advantages of COCA in synthetic simulations with structured compound Gaussian distributions. In these examples, COCA outperforms competing methods such as Tyler's estimator and its projection onto the structure set.

stat.ML↗

arXiv2014

Performance Analysis of Tyler's Covariance Estimator

This paper analyzes the performance of Tyler's M-estimator of the scatter matrix in elliptical populations. We focus on the non-asymptotic setting and derive the estimation error bounds depending on the number of samples n and the dimension p. We show that under quite mild conditions the squared Frobenius norm of the error of the inverse estimator decays like p^2/n with high probability.

math.ST↗

arXiv2014

Error Bound for Compound Wishart Matrices

In this paper we consider non-asymptotic behavior of the real compound Wishart matrices that generalize the classical real Wishart distribution. In particular, we consider matrices of the form 1/nXBX', where X consists of real centered Gaussian elements and B is an arbitrary real matrix and sequences of such matrices for varying n. We show how the expectation of deviations from the mean can be bounded for compound Wishart matrices.

stat.AP↗

arXiv2013

Covariance Estimation in Elliptical Models with Convex Structure

We address structured covariance estimation in Elliptical distribution. We assume it is a priori known that the covariance belongs to a given convex set, e.g., the set of Toeplitz or banded matrices. We consider the General Method of Moments (GMM) optimization subject to these convex constraints. Unfortunately, GMM is still non-convex due to objective. Instead, we propose COCA - a convex relaxation which can be efficiently solved. We prove that the relaxation is tight in the unconstrained case for a finite number of samples, and in the constrained case asymptotically. We then illustrate the advantages of COCA in synthetic simulations with structured Compound Gaussian distributions. In these examples, COCA outperforms competing methods as Tyler's estimate and its projection onto a convex set.

math.ST↗

arXiv2013

Group Symmetry and non-Gaussian Covariance Estimation

We consider robust covariance estimation with group symmetry constraints. Non-Gaussian covariance estimation, e.g., Tyler scatter estimator and Multivariate Generalized Gaussian distribution methods, usually involve non-convex minimization problems. Recently, it was shown that the underlying principle behind their success is an extended form of convexity over the geodesics in the manifold of positive definite matrices. A modern approach to improve estimation accuracy is to exploit prior knowledge via additional constraints, e.g., restricting the attention to specific classes of covariances which adhere to prior symmetry structures. In this paper, we prove that such group symmetry constraints are also geodesically convex and can therefore be incorporated into various non-Gaussian covariance estimators. Practical examples of such sets include: circulant, persymmetric and complex/quaternion proper structures. We provide a simple numerical technique for finding maximum likelihood estimates under such constraints, and demonstrate their performance advantage using synthetic experiments.

stat.ML↗