The Dantzig selector and sparsity oracle inequalities

Abstract

Let \[Y_j=f_*(X_j)+ξ_j,\qquad j=1,...,n,\] where $X,X_1,...,X_n$ are i.i.d. random variables in a measurable space $(S,\mathcal{A})$ with distribution $Π$ and $ξ,ξ_1,... ,ξ_n$ are i.i.d. random variables with ${\mathbb{E}}ξ=0$ independent of $(X_1,...,X_n).$ Given a dictionary $h_1,...,h_N:S\mapsto{\mathbb{R}},$ let $f_λ:=\sum_{j=1}^Nλ_jh_j$, $λ=(λ_1,...,λ_N)\in{\mathbb{R}}^N.$ Given $\varepsilon>0,$ define \[\hatΛ_{\varepsilon}:=\Biggl\{\lam bda\in{\mathbb{R}}^N:\max_{1\leq k\leq N}\Biggl|n^{-1}\sum_{j=1}^n\big l(f_λ(X_j)-Y_j\bigr)h_k(X_j)\Biggr|\leq\varepsilon \Biggr\}\] and \[\hatλ:=\hatλ^{\varepsilon}\in \operatorname {Arg min}\limits_{λ\in\hatΛ_{\varepsilon}}\|λ\|_{\ell_1}.\] In the case where $f_*:=f_{λ^*},λ^*\in {\mathbb{R}}^N,$ Candes and Tao [Ann. Statist. 35 (2007) 2313--2351] suggested using $\hatλ$ as an estimator of $λ^*.$ They called this estimator ``the Dantzig selector''. We study the properties of $f_{\hatλ}$ as an estimator of $f_*$ for regression models with random design, extending some of the results of Candes and Tao (and providing alternative proofs of these results).