Phase transition and higher order analysis of $L_q$ regularization under dependence

Abstract

We study the problem of estimating a $k$-sparse signal ${\mbox{$\beta$}}_0\in{\bf R}^p$ from a set of noisy observations ${\bf y}\in{\bf R}^n$ under the model ${\bf y}={\bf X}{\mbox{$\beta$}}+{\bf w}$, where ${\bf X}\in{\bf R}^{n\times p}$ is the measurement matrix the row of which is drawn from distribution $N(0,{\mbox{$\Sigma$}})$. We consider the class of $L_q$-regularized least squares (LQLS) given by the formulation $\hat{\mbox{$\beta$}}(\lambda,q)=\text{argmin}_{{\mbox{$\beta$}}\in{\bf R}^p}\frac{1}{2}\|{\bf y}-{\bf X}{\mbox{$\beta$}}\|^2_2+\lambda\|{\mbox{$\beta$}}\|_q^q$, where $\|\cdot\|_q$ $(0\le q\le 2)$ denotes the $L_q$-norm. In the setting $p,n,k\rightarrow\infty$ with fixed $k/p=\epsilon$ and $n/p=\delta$, we derive the asymptotic risk of $\hat{\mbox{$\beta$}}(\lambda,q)$ for arbitrary covariance matrix ${\mbox{$\Sigma$}}$ which generalizes the existing results for standard Gaussian design, i.e. $X_{ij}\overset{i.i.d}{\sim}N(0,1)$. We perform a higher-order analysis for LQLS in the small-error regime in which the first dominant term can be used to determine the phase transition behavior of LQLS. Our results show that the first dominant term does not depend on the covariance structure of ${\mbox{$\Sigma$}}$ in the cases $0\le q< 1$ and $1< q\le 2$ which indicates that the correlations among predictors only affect the phase transition curve in the case $q=1$ a.k.a. LASSO. To study the influence of the covariance structure of ${\mbox{$\Sigma$}}$ on the performance of LQLS in the cases $0\le q< 1$ and $1<q\le 2$, we derive the explicit formulas for the second dominant term in the expansion of the asymptotic risk in terms of small error. Extensive computational experiments confirm that our analytical predictions are consistent with numerical results.