Shrinking the Sample Covariance Matrix using Convex Penalties on the Matrix-Log Transformation

Abstract

For $q$-dimensional data, penalized versions of the sample covariance matrix are important when the sample size is small or modest relative to $q$. Since the negative log-likelihood under multivariate normal sampling is convex in $\Sigma^{-1}$, the inverse of its covariance matrix, it is common to add to it a penalty which is also convex in $\Sigma^{-1}$. More recently, Deng-Tsui (2013) and Yu et al.(2017) have proposed penalties which are functions of the eigenvalues of $\Sigma$, and are convex in $\log \Sigma$, but not in $\Sigma^{-1}$. The resulting penalized optimization problem is not convex in either $\log \Sigma$ or $\Sigma^{-1}$. In this paper, we note that this optimization problem is geodesically convex in $\Sigma$, which allows us to establish the existence and uniqueness of the corresponding penalized covariance matrices. More generally, we show the equivalence of convexity in $\log \Sigma$ and geodesic convexity for penalties on $\Sigma$ which are strictly functions of their eigenvalues. In addition, when using such penalties, we show that the resulting optimization problem reduces to to a $q$-dimensional convex optimization problem on the eigenvalues of $\Sigma$, which can then be readily solved via Newton-Raphson. Finally, we argue that it is better to apply these penalties to the shape matrix $\Sigma/(\det \Sigma)^{1/q}$ rather than to $\Sigma$ itself. A simulation study and an example illustrate the advantages of applying the penalty to the shape matrix.