Phase transition for conditional covariance matrices estimated by importance sampling, and implications for cross-entropy schemes in high dimension
Motivated by the estimation of covariance matrices by importance sampling arising in the cross-entropy (CE) algorithm, we study a random matrix model $\hat \Sigma = {\bf X} L {\bf X}^\top$ with two distinct features: $\bf X$ and $L$ are dependent, and $L$ is heavy-tailed. In the high-dimensional regime $d \to \infty$, we prove under suitable assumptions that a phase transition occurs in the polynomial regime $n = d^\kappa$, with $n$ the sample size. Namely, we prove that $\lVert \hat \Sigma - E \hat \Sigma \rVert \Rightarrow 0$ if and only if $\kappa > \kappa_*$ for some threshold $\kappa_*$ determined by the behavior of the maximum likelihood ratios. Moreover, we identify general situations where $\kappa_* = 1/\lambda_1$, with $\lambda_1$ the smallest eigenvalue of the covariance matrix of the auxiliary distribution used to estimate $\hat \Sigma$ by importance sampling. This suggests that importance sampling will work better with covariance matrices having a large smallest eigenvalue. We carry this insight into recent CE schemes proposed to estimate the probability of high-dimensional rare events. Through numerical simulations, we demonstrate that better CE schemes are also the ones with larger smallest eigenvalue, even though these algorithms were not designed to smooth the spectrum. This new spectral interpretation raises stimulating questions and opens research directions for the design of efficient high-dimensional algorithms.