arXiv subjects
Publications and source records attributed to Shahar Mendelson.
We obtain a new upper estimate on the Euclidean diameter of the intersection of the kernel of a random matrix with iid rows with a given convex body. The proof is based on a small-ball argument rather than on concentration and thus the estimate holds for relatively general matrix ensembles.
We study the performance of Empirical Risk Minimization in noisy phase retrieval problems, indexed by subsets of $\R^n$ and relative to subgaussian sampling; that is, when the given data is $y_i=\inr{a_i,x_0}^2+w_i$ for a subgaussian random vector $a$, independent noise $w$ and a fixed but unknown $x_0$ that belongs to a given subset of $\R^n$. We show that ERM produces $\hat{x}$ whose Euclidean distance to either $x_0$ or $-x_0$ depends on the gaussian mean-width of the indexing set and on the signal-to-noise ratio of the problem. The bound coincides with the one for linear regression when $\|x_0\|_2$ is of the order of a constant. In addition, we obtain a minimax lower bound for the problem and identify sets for which ERM is a minimax procedure. As examples, we study the class of $d$-sparse vectors in $\R^n$ and the unit ball in $\ell_1^n$.
We present a new bound for suprema of a special type of chaos processes indexed by a set of matrices, which is based on a chaining method. As applications we show significantly improved estimates for the restricted isometry constants of partial random circulant matrices and time-frequency structured random matrices. In both cases the required condition on the number $m$ of rows in terms of the sparsity $s$ and the vector length $n$ is $m \gtrsim s \log^2 s \log^2 n$.
Given a finite class of functions F, the problem of aggregation is to construct a procedure with a risk as close as possible to the risk of the best element in the class. A classical procedure (PAC-Bayesian statistical learning theory (2004) Paris 6, Statistical Learning Theory and Stochastic Optimization (2001) Springer, Ann. Statist. 28 (2000) 75-87) is the aggregate with exponential weights (AEW), defined by \[\tilde{f}^{\mathrm{AEW}}=\sum_{f\in F}\hatθ(f)f,\qquad where \hatθ(f)=\frac{\exp(-({n}/{T})R_n(f))}{\sum_{g\in F}\exp(-({n}/{T})R_n(g))},\] where $T>0$ is called the temperature parameter and $R_n(\cdot)$ is an empirical risk. In this article, we study the optimality of the AEW in the regression model with random design and in the low-temperature regime. We prove three properties of AEW. First, we show that AEW is a suboptimal aggregation procedure in expectation with respect to the quadratic risk when $T\leq c_1$, where $c_1$ is an absolute positive constant (the low-temperature regime), and that it is suboptimal in probability even for high temperatures. Second, we show that as the cardinality of the dictionary grows, the behavior of AEW might deteriorate, namely, that in the low-temperature regime it might concentrate with high probability around elements in the dictionary with risk greater than the risk of the best function in the dictionary by at least an order of $1/\sqrt{n}$. Third, we prove that if a geometric condition on the dictionary (the so-called "Bernstein condition) is assumed, then AEW is indeed optimal both in high probability and in expectation in the low-temperature regime. Moreover, under that assumption, the complexity term is essentially the logarithm of the cardinality of the set of "almost minimizers" rather than the logarithm of the cardinality of the entire dictionary. This result holds for small values of the temperature parameter, thus complementing an analogous result for high temperatures.
We consider stability and uniqueness in real phase retrieval problems over general input sets. Specifically, we assume the data consists of noisy quadratic measurements of an unknown input x in R^n that lies in a general set T and study conditions under which x can be stably recovered from the measurements. In the noise-free setting we derive a general expression on the number of measurements needed to ensure that a unique solution can be found in a stable way, that depends on the set T through a natural complexity parameter. This parameter can be computed explicitly for many sets T of interest. For example, for k-sparse inputs we show that O(k\log(n/k)) measurements are needed, and when x can be any vector in R^n, O(n) measurements suffice. In the noisy case, we show that if one can find a value for which the empirical risk is bounded by a given, computable constant (that depends on the set T), then the error with respect to the true input is bounded above by an another, closely related complexity parameter of the set. By choosing an appropriate number N of measurements, this bound can be made arbitrarily small, and it decays at a rate faster than N^{-1/2+δ} for any δ>0. In particular, for k-sparse vectors stable recovery is possible from O(k\log(n/k)\log k) noisy measurements, and when x can be any vector in R^n, O(n \log n) noisy measurements suffice. We also show that the complexity parameter for the quadratic problem is the same as the one used for analyzing stability in linear measurements under very general conditions. Thus, no substantial price has to be paid in terms of stability if there is no knowledge of the phase.
We show that empirical risk minimization procedures and regularized empirical risk minimization procedures satisfy nonexact oracle inequalities in an unbounded framework, under the assumption that the class has a subexponential envelope function. The main novelty, in addition to the boundedness assumption free setup, is that those inequalities can yield fast rates even in situations in which exact oracle inequalities only hold with slower rates. We apply these results to show that procedures based on $\ell_1$ and nuclear norms regularization functions satisfy oracle inequalities with a residual term that decreases like $1/n$ for every $L_q$-loss functions ($q\geq2$), while only assuming that the tail behavior of the input and output variables are well behaved. In particular, no RIP type of assumption or "incoherence condition" are needed to obtain fast residual terms in those setups. We also apply these results to the problems of convex aggregation and model selection.
We present a very general chaining method which allows one to control the supremum of the empirical process $\sup_{h \in H} |N^{-1}\sum_{i=1}^N h^2(X_i)-\E h^2|$ in rather general situations. We use this method to establish two main results. First, a quantitative (non asymptotic) version of the classical Bai-Yin Theorem on the singular values of a random matrix with i.i.d entries that have heavy tails, and second, a sharp estimate on the quadratic empirical process when $H=\{\inr{t,\cdot} : t \in T\}$, $T \subset \R^n$ and $μ$ is an isotropic, unconditional, log-concave measure.
We show that for a typical coordinate projection of a subgaussian class of functions, the infimum over signs $\inf_{(ε_i)}{\sup_{f\in F}}|{\sum_{i=1}^kε_i}f(X_i)|$ is asymptotically smaller than the expectation over signs as a function of the dimension $k$, if the canonical Gaussian process indexed by $F$ is continuous. To that end, we establish a bound on the discrepancy of an arbitrary subset of $\mathbb {R}^k$ using properties of the canonical Gaussian process the set indexes, and then obtain quantitative structural information on a typical coordinate projection of a subgaussian class.
We present an argument based on the multidimensional and the uniform central limit theorems, proving that, under some geometrical assumptions between the target function $T$ and the learning class $F$, the excess risk of the empirical risk minimization algorithm is lower bounded by \[\frac{\mathbb{E}\sup_{q\in Q}G_q}{\sqrt{n}}δ,\] where $(G_q)_{q\in Q}$ is a canonical Gaussian process associated with $Q$ (a well chosen subset of $F$) and $δ$ is a parameter governing the oscillations of the empirical excess risk function over a small ball in $F$.
We study the empirical process indexed by F^2=\{f^2 : f \in F\}, where F is a class of mean-zero functions on a probability space. We present a sharp bound on the supremum of that process which depends on the ψ_1 diameter of the class F (rather than on the ψ_2 one) and on the complexity parameter γ_2(F,ψ_2). In addition, we present optimal bounds on the random diameters \sup_{f \in F} \max_{|I|=m} (\sum_{i \in I} f^2(X_i))^{1/2} using the same parameters. As applications, we extend several well known results in Asymptotic Geometric Analysis to any isotropic, log-concave ensemble on R^n.
Under mild assumptions on the kernel, we obtain the best known error rates in a regularized learning scenario taking place in the corresponding reproducing kernel Hilbert space (RKHS). The main novelty in the analysis is a proof that one can use a regularization term that grows significantly slower than the standard quadratic growth in the RKHS norm.
In this article we prove that for any orthonormal system $(\vphi_j)_{j=1}^n \subset L_2$ that is bounded in $L_{\infty}$, and any $1 < k <n$, there exists a subset $I$ of cardinality greater than $n-k$ such that on $\spa\{\vphi_i\}_{i \in I}$, the $L_1$ norm and the $L_2$ norm are equivalent up to a factor $μ(\log μ)^{5/2}$, where $μ= \sqrt{n/k} \sqrt{\log k}$. The proof is based on a new estimate of the supremum of an empirical process on the unit ball of a Banach space with a good modulus of convexity, via the use of majorizing measures.
Discussion of "2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization" by V. Koltchinskii [arXiv:0708.0083]
We show that a modified Empirical process converges to the limiting Gaussian process whenever the limit is continuous. The modification depends on the properties of the limit via Talagrand's characterization of the continuity of Gaussian processes.
We present a simple solution to a question posed by Candes, Romberg and Tao on the uniform uncertainty principle for Bernoulli random matrices. More precisely, we show that a rectangular k*n random subgaussian matrix (with k < n) has the property that by arbitrarily extracting any m (with m < k) columns, the resulting submatrices are arbitrarily close to (multiples of) isometries of a Euclidean space. We obtain the optimal estimate for m as a function of k,n and the degree of "closeness" to an isometry. We also give a short and self-contained solution of the reconstruction problem for sparse vectors.
Let $F$ be a class of functions on a probability space $(Ω,μ)$ and let $X_1,...,X_k$ be independent random variables distributed according to $μ$. We establish high probability tail estimates of the form $\sup_{f \in F} |\{i : |f(X_i)| \geq t \}$ using a natural parameter associated with $F$. We use this result to analyze weakly bounded empirical processes indexed by $F$ and processes of the form $Z_f=|k^{-1}\sum_{i=1}^k |f|^p(X_i)-\E|f|^p|$ for $p>1$. We also present some geometric applications of this approach, based on properties of the random operator $Γ=k^{-1/2}\sum_{i=1}^k \inr{X_i,\cdot}e_i$, where the $(X_i)_{i=1}^k$ are sampled according to an isotropic, log-concave measure on $\R^n$.
We propose new bounds on the error of learning algorithms in terms of a data-dependent notion of complexity. The estimates we establish give optimal rates and are based on a local and empirical version of Rademacher averages, in the sense that the Rademacher averages are computed from the data, on a subset of functions with small empirical error. We present some applications to classification and prediction with convex function classes, and with kernel classes in particular.
We present a randomized method to approximate any vector $v$ from some set $T \subset \R^n$. The data one is given is the set $T$, and $k$ scalar products $(\inr{X_i,v})_{i=1}^k$, where $(X_i)_{i=1}^k$ are i.i.d. isotropic subgaussian random vectors in $\R^n$, and $k \ll n$. We show that with high probability, any $y \in T$ for which $(\inr{X_i,y})_{i=1}^k$ is close to the data vector $(\inr{X_i,v})_{i=1}^k$ will be a good approximation of $v$, and that the degree of approximation is determined by a natural geometric parameter associated with the set $T$. We also investigate a random method to identify exactly any vector which has a relatively short support using linear subgaussian measurements as above. It turns out that our analysis, when applied to $\{-1,1\}$-valued vectors with i.i.d, symmetric entries, yields new information on the geometry of faces of random $\{-1,1\}$-polytope; we show that a $k$-dimensional random $\{-1,1\}$-polytope with $n$ vertices is $m$-neighborly for very large $m\le {ck/\log (c' n/k)}$. The proofs are based on new estimates on the behavior of the empirical process $\sup_{f \in F} |k^{-1}\sum_{i=1}^k f^2(X_i) -\E f^2 |$ when $F$ is a subset of the $L_2$ sphere. The estimates are given in terms of the $γ_2$ functional with respect to the $ψ_2$ metric on $F$, and hold both in exponential probability and in expectation.