arXiv subjects

Noureddine El Karoui

Publications and source records attributed to Noureddine El Karoui.

32 records · Page 2Linked to original sources

arXiv2012

High-dimensionality effects in the Markowitz problem and other quadratic programs with linear constraints: Risk underestimation

We first study the properties of solutions of quadratic programs with linear equality constraints whose parameters are estimated from data in the high-dimensional setting where p, the number of variables in the problem, is of the same order of magnitude as n, the number of observations used to estimate the parameters. The Markowitz problem in Finance is a subcase of our study. Assuming normality and independence of the observations we relate the efficient frontier computed empirically to the "true" efficient frontier. Our computations show that there is a separation of the errors induced by estimating the mean of the observations and estimating the covariance matrix. In particular, the price paid for estimating the covariance matrix is an underestimation of the variance by a factor roughly equal to 1-p/n. Therefore the risk of the optimal population solution is underestimated when we estimate it by solving a similar quadratic program with estimated parameters. We also characterize the statistical behavior of linear functionals of the empirical optimal vector and show that they are biased estimators of the corresponding population quantities.

math.ST↗

arXiv2012

Weak Recovery Conditions from Graph Partitioning Bounds and Order Statistics

We study a weaker formulation of the nullspace property which guarantees recovery of sparse signals from linear measurements by l_1 minimization. We require this condition to hold only with high probability, given a distribution on the nullspace of the coding matrix A. Under some assumptions on the distribution of the reconstruction error, we show that testing these weak conditions means bounding the optimal value of two classical graph partitioning problems: the k-Dense-Subgraph and MaxCut problems. Both problems admit efficient, relatively tight relaxations and we use a randomization argument to produce new approximation bounds for k-Dense-Subgraph. We test the performance of our results on several families of coding matrices.

math.OC↗

arXiv2012

Performance-based regularization in mean-CVaR portfolio optimization

We introduce performance-based regularization (PBR), a new approach to addressing estimation risk in data-driven optimization, to mean-CVaR portfolio optimization. We assume the available log-return data is iid, and detail the approach for two cases: nonparametric and parametric (the log-return distribution belongs in the elliptical family). The nonparametric PBR method penalizes portfolios with large variability in mean and CVaR estimations. The parametric PBR method solves the empirical Markowitz problem instead of the empirical mean-CVaR problem, as the solutions of the Markowitz and mean-CVaR problems are equivalent when the log-return distribution is elliptical. We derive the asymptotic behavior of the nonparametric PBR solution, which leads to insight into the effect of penalization, and justification of the parametric PBR method. We also show via simulations that the PBR methods produce efficient frontiers that are, on average, closer to the population efficient frontier than the empirical approach to the mean-CVaR problem, with less variability.

q-fin.PM↗

arXiv2012

Optimizing Automated Classification of Periodic Variable Stars in New Synoptic Surveys

Efficient and automated classification of periodic variable stars is becoming increasingly important as the scale of astronomical surveys grows. Several recent papers have used methods from machine learning and statistics to construct classifiers on databases of labeled, multi--epoch sources with the intention of using these classifiers to automatically infer the classes of unlabeled sources from new surveys. However, the same source observed with two different synoptic surveys will generally yield different derived metrics (features) from the light curve. Since such features are used in classifiers, this survey-dependent mismatch in feature space will typically lead to degraded classifier performance. In this paper we show how and why feature distributions change using OGLE and \textit{Hipparcos} light curves. To overcome survey systematics, we apply a method, \textit{noisification}, which attempts to empirically match distributions of features between the labeled sources used to construct the classifier and the unlabeled sources we wish to classify. Results from simulated and real--world light curves show that noisification can significantly improve classifier performance. In a three--class problem using light curves from \textit{Hipparcos} and OGLE, noisification reduces the classifier error rate from 27.0% to 7.0%. We recommend that noisification be used for upcoming surveys such as Gaia and LSST and describe some of the promises and challenges of applying noisification to these surveys.

astro-ph.IM↗

arXiv2011

Geometric sensitivity of random matrix results: consequences for shrinkage estimators of covariance and related statistical methods

Shrinkage estimators of covariance are an important tool in modern applied and theoretical statistics. They play a key role in regularized estimation problems, such as ridge regression (aka Tykhonov regularization), regularized discriminant analysis and a variety of optimization problems. In this paper, we bring to bear the tools of random matrix theory to understand their behavior, and in particular, that of quadratic forms involving inverses of those estimators, which are important in practice. We use very mild assumptions compared to the usual assumptions made in random matrix theory, requiring only mild conditions on the moments of linear and quadratic forms in our random vectors. In particular, we show that our results apply for instance to log-normal data, which are of interest in financial applications. Our study highlights the relative sensitivity of random matrix results (and their practical consequences) to geometric assumptions which are often implicitly made by random matrix theorists and may not be relevant in data analytic practice.

math.ST↗

arXiv2010

On information plus noise kernel random matrices

Kernel random matrices have attracted a lot of interest in recent years, from both practical and theoretical standpoints. Most of the theoretical work so far has focused on the case were the data is sampled from a low-dimensional structure. Very recently, the first results concerning kernel random matrices with high-dimensional input data were obtained, in a setting where the data was sampled from a genuinely high-dimensional structure---similar to standard assumptions in random matrix theory. In this paper, we consider the case where the data is of the type "information${}+{}$noise." In other words, each observation is the sum of two independent elements: one sampled from a "low-dimensional" structure, the signal part of the data, the other being high-dimensional noise, normalized to not overwhelm but still affect the signal. We consider two types of noise, spherical and elliptical. In the spherical setting, we show that the spectral properties of kernel random matrices can be understood from a new kernel matrix, computed only from the signal part of the data, but using (in general) a slightly different kernel. The Gaussian kernel has some special properties in this setting. The elliptical setting, which is important from a robustness standpoint, is less prone to easy interpretation.

math.ST↗

arXiv2010

Second order accurate distributed eigenvector computation for extremely large matrices

We propose a second-order accurate method to estimate the eigenvectors of extremely large matrices thereby addressing a problem of relevance to statisticians working in the analysis of very large datasets. More specifically, we show that averaging eigenvectors of randomly subsampled matrices efficiently approximates the true eigenvectors of the original matrix under certain conditions on the incoherence of the spectral decomposition. This incoherence assumption is typically milder than those made in matrix completion and allows eigenvectors to be sparse. We discuss applications to spectral methods in dimensionality reduction and information retrieval.

math.NA↗

arXiv2010

The spectrum of kernel random matrices

We place ourselves in the setting of high-dimensional statistical inference where the number of variables $p$ in a dataset of interest is of the same order of magnitude as the number of observations $n$. We consider the spectrum of certain kernel random matrices, in particular $n\times n$ matrices whose $(i,j)$th entry is $f(X_i'X_j/p)$ or $f(\Vert X_i-X_j\Vert^2/p)$ where $p$ is the dimension of the data, and $X_i$ are independent data vectors. Here $f$ is assumed to be a locally smooth function. The study is motivated by questions arising in statistics and computer science where these matrices are used to perform, among other things, nonlinear versions of principal component analysis. Surprisingly, we show that in high-dimensions, and for the models we analyze, the problem becomes essentially linear--which is at odds with heuristics sometimes used to justify the usage of these methods. The analysis also highlights certain peculiarities of models widely studied in random matrix theory and raises some questions about their relevance as tools to model high-dimensional data encountered in practice.

math.ST↗

arXiv2009

Concentration of measure and spectra of random matrices: Applications to correlation matrices, elliptical distributions and beyond

We place ourselves in the setting of high-dimensional statistical inference, where the number of variables $p$ in a data set of interest is of the same order of magnitude as the number of observations $n$. More formally, we study the asymptotic properties of correlation and covariance matrices, in the setting where $p/n\toρ\in(0,\infty),$ for general population covariance. We show that, for a large class of models studied in random matrix theory, spectral properties of large-dimensional correlation matrices are similar to those of large-dimensional covarance matrices. We also derive a Marucenko--Pastur-type system of equations for the limiting spectral distribution of covariance matrices computed from data with elliptical distributions and generalizations of this family. The motivation for this study comes partly from the possible relevance of such distributional assumptions to problems in econometrics and portfolio optimization, as well as robustness questions for certain classical random matrix results. A mathematical theme of the paper is the important use we make of concentration inequalities.

math.PR↗

arXiv2009

Operator norm consistent estimation of large-dimensional sparse covariance matrices

Estimating covariance matrices is a problem of fundamental importance in multivariate statistics. In practice it is increasingly frequent to work with data matrices $X$ of dimension $n\times p$, where $p$ and $n$ are both large. Results from random matrix theory show very clearly that in this setting, standard estimators like the sample covariance matrix perform in general very poorly. In this "large $n$, large $p$" setting, it is sometimes the case that practitioners are willing to assume that many elements of the population covariance matrix are equal to 0, and hence this matrix is sparse. We develop an estimator to handle this situation. The estimator is shown to be consistent in operator norm, when, for instance, we have $p\asymp n$ as $n\to\infty$. In other words the largest singular value of the difference between the estimator and the population covariance matrix goes to zero. This implies consistency of all the eigenvalues and consistency of eigenspaces associated to isolated eigenvalues. We also propose a notion of sparsity for matrices, that is, "compatible" with spectral analysis and is independent of the ordering of the variables.

math.ST↗

arXiv2007

Tracy--Widom limit for the largest eigenvalue of a large class of complex sample covariance matrices

We consider the asymptotic fluctuation behavior of the largest eigenvalue of certain sample covariance matrices in the asymptotic regime where both dimensions of the corresponding data matrix go to infinity. More precisely, let $X$ be an $n\times p$ matrix, and let its rows be i.i.d. complex normal vectors with mean 0 and covariance $Σ_p$. We show that for a large class of covariance matrices $Σ_p$, the largest eigenvalue of $X^*X$ is asymptotically distributed (after recentering and rescaling) as the Tracy--Widom distribution that appears in the study of the Gaussian unitary ensemble. We give explicit formulas for the centering and scaling sequences that are easy to implement and involve only the spectral distribution of the population covariance, $n$ and $p$. The main theorem applies to a number of covariance models found in applications. For example, well-behaved Toeplitz matrices as well as covariance matrices whose spectral distribution is a sum of atoms (under some conditions on the mass of the atoms) are among the models the theorem can handle. Generalizations of the theorem to certain spiked versions of our models and a.s. results about the largest eigenvalue are given. We also discuss a simple corollary that does not require normality of the entries of the data matrix and some consequences for applications in multivariate statistics.

math.PR↗

arXiv2007

A rate of convergence result for the largest eigenvalue of complex white Wishart matrices

It has been recently shown that if $X$ is an $n\times N$ matrix whose entries are i.i.d. standard complex Gaussian and $l_1$ is the largest eigenvalue of $X^*X$, there exist sequences $m_{n,N}$ and $s_{n,N}$ such that $(l_1-m_{n,N})/s_{n,N}$ converges in distribution to $W_2$, the Tracy--Widom law appearing in the study of the Gaussian unitary ensemble. This probability law has a density which is known and computable. The cumulative distribution function of $W_2$ is denoted $F_2$. In this paper we show that, under the assumption that $n/N\to γ\in(0,\infty)$, we can find a function $M$, continuous and nonincreasing, and sequences $\tildeμ_{n,N}$ and $\tildeσ_{n,N}$ such that, for all real $s_0$, there exists an integer $N(s_0,γ)$ for which, if $(n\wedge N)\geq N(s_0,γ)$, we have, with $l_{n,N}=(l_1-\tildeμ_{n,N})/\tildeσ_{n,N}$, \[\forall s\geq s_0\qquad (n\wedge N)^{2/3}|P(l_{n,N}\leq s)-F_2(s)|\leq M(s_0)\exp(-s).\] The surprisingly good 2/3 rate and qualitative properties of the bounding function help explain the fact that the limiting distribution $W_2$ is a good approximation to the empirical distribution of $l_{n,N}$ in simulations, an important fact from the point of view of (e.g., statistical) applications.

math.PR↗

arXiv2006

Spectrum estimation for large dimensional covariance matrices using random matrix theory

Estimating the eigenvalues of a population covariance matrix from a sample covariance matrix is a problem of fundamental importance in multivariate statistics; the eigenvalues of covariance matrices play a key role in many widely techniques, in particular in Principal Component Analysis (PCA). In many modern data analysis problems, statisticians are faced with large datasets where the sample size, n, is of the same order of magnitude as the number of variables p. Random matrix theory predicts that in this context, the eigenvalues of the sample covariance matrix are not good estimators of the eigenvalues of the population covariance. We propose to use a fundamental result in random matrix theory, the Marcenko-Pastur equation, to better estimate the eigenvalues of large dimensional covariance matrices. The Marcenko-Pastur equation holds in very wide generality and under weak assumptions. The estimator we obtain can be thought of as "shrinking" in a non linear fashion the eigenvalues of the sample covariance matrix to estimate the population eigenvalue. Inspired by ideas of random matrix theory, we also suggest a change of point of view when thinking about estimation of high-dimensional vectors: we do not try to estimate directly the vectors but rather a probability measure that describes them. We think this is a theoretically more fruitful way to think statistically about these problems. Our estimator gives fast and good or very good results in extended simulations. Our algorithmic approach is based on convex optimization. We also show that the proposed estimator is consistent.

math.ST↗

arXiv2003

On the largest eigenvalue of Wishart matrices with identity covariance when n, p and p/n tend to infinity

Let X be a n*p matrix and l_1 the largest eigenvalue of the covariance matrix X^{*}*X. The "null case" where X_{i,j} are independent Normal(0,1) is of particular interest for principal component analysis. For this model, when n, p tend to infinity and n/p tends to gamma in (0,\infty), it was shown in Johnstone (2001) that l_1, properly centered and scaled, converges to the Tracy-Widom law. We show that with the same centering and scaling, the result is true even when p/n or n/p tends to infinity. The derivation uses ideas and techniques quite similar to the ones presented in Johnstone (2001). Following Soshnikov (2002), we also show that the same is true for the joint distribution of the k largest eigenvalues, where k is a fixed integer. Numerical experiments illustrate the fact that the Tracy-Widom approximation is reasonable even when one of the dimension is "small".

math.ST↗