Searcharxiv⌕ Search

arXiv subjects

Vladimir Koltchinskii

Publications and source records attributed to Vladimir Koltchinskii.

At least 37 records · Page 2Linked to original sources
arXiv2014

Concentration Inequalities and Moment Bounds for Sample Covariance Operators

Let $X,X_1,\dots, X_n,\dots$ be i.i.d. centered Gaussian random variables in a separable Banach space $E$ with covariance operator $Σ:$ $$ Σ:E^{\ast}\mapsto E,\ \ Σu = {\mathbb E}\langle X,u\rangle, u\in E^{\ast}. $$ The sample covariance operator $\hat Σ:E^{\ast}\mapsto E$ is defined as $$ \hat Σu := n^{-1}\sum_{j=1}^n \langle X_j,u\rangle X_j, u\in E^{\ast}. $$ The goal of the paper is to obtain concentration inequalities and expectation bounds for the operator norm $\|\hat Σ-Σ\|$ of the deviation of the sample covariance operator from the true covariance operator. In particular, it is shown that $$ {\mathbb E}\|\hat Σ-Σ\|\asymp \|Σ\|\biggl(\sqrt{\frac{{\bf r}(Σ)}{n}}\bigvee \frac{{\bf r}(Σ)}{n}\biggr), $$ where $$ {\bf r}(Σ):=\frac{\Bigl({\mathbb E}\|X\|\Bigr)^2}{\|Σ\|}. $$ Moreover, under the assumption that ${\bf r}(Σ)\lesssim n,$ it is proved that, for all $t\geq 1,$ with probability at least $1-e^{-t}$ \begin{align*} \Bigl|\|\hatΣ- Σ\|-{\mathbb E}\|\hatΣ- Σ\|\Bigr| \lesssim \|Σ\|\biggl(\sqrt{\frac{t}{n}}\bigvee \frac{t}{n}\biggr). \end{align*}

math.PR↗
arXiv2013

Bounding the smallest singular value of a random matrix without concentration

Given $X$ a random vector in ${\mathbb{R}}^n$, set $X_1,...,X_N$ to be independent copies of $X$ and let $Γ=\frac{1}{\sqrt{N}}\sum_{i=1}^N e_i$ be the matrix whose rows are $\frac{X_1}{\sqrt{N}},\dots, \frac{X_N}{\sqrt{N}}$. We obtain sharp probabilistic lower bounds on the smallest singular value $λ_{\min}(Γ)$ in a rather general situation, and in particular, under the assumption that $X$ is an isotropic random vector for which $\sup_{t\in S^{n-1}}{\mathbb{E}}| |^{2+η} \leq L$ for some $L,η>0$. Our results imply that a Bai-Yin type lower bound holds for $η>2$, and, up to a log-factor, for $η=2$ as well. The bounds hold without any additional assumptions on the Euclidean norm $\|X\|_{\ell_2^n}$. Moreover, we establish a nontrivial lower bound even without any higher moment assumptions (corresponding to the case $η=0$), if the linear forms satisfy a weak `small ball' property.

math.PR↗
arXiv2013

Low rank estimation of smooth kernels on graphs

Let (V,A) be a weighted graph with a finite vertex set V, with a symmetric matrix of nonnegative weights A and with Laplacian $Δ$. Let $S_*:V\times V\mapsto{\mathbb{R}}$ be a symmetric kernel defined on the vertex set V. Consider n i.i.d. observations $(X_j,X_j',Y_j),j=1,\ldots,n$, where $X_j,X_j'$ are independent random vertices sampled from the uniform distribution in V and $Y_j\in{\mathbb{R}}$ is a real valued response variable such that ${\mathbb{E}}(Y_j|X_j,X_j')=S_*(X_j,X_j'),j=1,\ldots,n$. The goal is to estimate the kernel $S_*$ based on the data $(X_1,X_1',Y_1),\ldots,(X_n,X_n',Y_n)$ and under the assumption that $S_*$ is low rank and, at the same time, smooth on the graph (the smoothness being characterized by discrete Sobolev norms defined in terms of the graph Laplacian). We obtain several results for such problems including minimax lower bounds on the $L_2$-error and upper bounds for penalized least squares estimators both with nonconvex and with convex penalties.

math.ST↗
arXiv2012

Sparsity in multiple kernel learning

The problem of multiple kernel learning based on penalized empirical risk minimization is discussed. The complexity penalty is determined jointly by the empirical $L_2$ norms and the reproducing kernel Hilbert space (RKHS) norms induced by the kernels with a data-driven choice of regularization parameters. The main focus is on the case when the total number of kernels is large, but only a relatively small number of them is needed to represent the target function, so that the problem is sparse. The goal is to establish oracle inequalities for the excess risk of the resulting prediction rule showing that the method is adaptive both to the unknown design distribution and to the sparsity of the problem.

math.ST↗
arXiv2012

Sharp Oracle Inequalities in Low Rank Estimation

The paper deals with the problem of penalized empirical risk minimization over a convex set of linear functionals on the space of Hermitian matrices with convex loss and nuclear norm penalty. Such penalization is often used in low rank matrix recovery in the cases when the target function can be well approximated by a linear functional generated by a Hermitian matrix of relatively small rank (comparing with the size of the matrix). Our goal is to prove sharp low rank oracle inequalities that involve the excess risk (the approximation error) with constant equal to one and the random error term with correct dependence on the rank of the oracle.

math.ST↗
arXiv2012

Low Rank Estimation of Similarities on Graphs

Let (V, E) be a graph with vertex set V and edge set E. Let (X, X', Y) \in V \times V \times {-1, 1} be a random triple, where X, X' are independent uniformly distributed vertices and Y is a label indicating whether X, X' are "similar" (Y = +1), or not (Y = -1). Our goal is to estimate the regression function S\ast (u, v) = E(Y |X = u, X = v), u, v \in V based on training data consisting of n i.i.d. copies of (X, X',Y). We are interested in this problem in the case when S\ast is a symmetric low rank kernel and, in addition to this, it is assumed that S\ast is "smooth" on the graph. We study estimators based on a modified least squares method with complexity penalization involving both the nuclear norm and Sobolev type norms of symmetric kernels on the graph and prove upper bounds on L2 -type errors of such estimators with explicit dependence both on the rank of S\ast and on the degree of its smoothness.

math.ST↗
arXiv2010

Von Neumann Entropy Penalization and Low Rank Matrix Estimation

A problem of statistical estimation of a Hermitian nonnegatively definite matrix of unit trace (for instance, a density matrix in quantum state tomography) is studied. The approach is based on penalized least squares method with a complexity penalty defined in terms of von Neumann entropy. A number of oracle inequalities have been proved showing how the error of the estimator depends on the rank and other characteristics of the oracles. The methods of proofs are based on empirical processes theory and probabilistic inequalities for random matrices, in particular, noncommutative versions of Bernstein inequality.

math.ST↗
arXiv2009

The Dantzig selector and sparsity oracle inequalities

Let \[Y_j=f_*(X_j)+ξ_j,\qquad j=1,...,n,\] where $X,X_1,...,X_n$ are i.i.d. random variables in a measurable space $(S,\mathcal{A})$ with distribution $Π$ and $ξ,ξ_1,... ,ξ_n$ are i.i.d. random variables with ${\mathbb{E}}ξ=0$ independent of $(X_1,...,X_n).$ Given a dictionary $h_1,...,h_N:S\mapsto{\mathbb{R}},$ let $f_λ:=\sum_{j=1}^Nλ_jh_j$, $λ=(λ_1,...,λ_N)\in{\mathbb{R}}^N.$ Given $\varepsilon>0,$ define \[\hatΛ_{\varepsilon}:=\Biggl\{\lam bda\in{\mathbb{R}}^N:\max_{1\leq k\leq N}\Biggl|n^{-1}\sum_{j=1}^n\big l(f_λ(X_j)-Y_j\bigr)h_k(X_j)\Biggr|\leq\varepsilon \Biggr\}\] and \[\hatλ:=\hatλ^{\varepsilon}\in \operatorname {Arg min}\limits_{λ\in\hatΛ_{\varepsilon}}\|λ\|_{\ell_1}.\] In the case where $f_*:=f_{λ^*},λ^*\in {\mathbb{R}}^N,$ Candes and Tao [Ann. Statist. 35 (2007) 2313--2351] suggested using $\hatλ$ as an estimator of $λ^*.$ They called this estimator ``the Dantzig selector''. We study the properties of $f_{\hatλ}$ as an estimator of $f_*$ for regression models with random design, extending some of the results of Candes and Tao (and providing alternative proofs of these results).

math.ST↗
arXiv2009

Sparse recovery in convex hulls via entropy penalization

Let $(X,Y)$ be a random couple in $S\times T$ with unknown distribution $P$ and $(X_1,Y_1),...,(X_n,Y_n)$ be i.i.d. copies of $(X,Y).$ Denote $P_n$ the empirical distribution of $(X_1,Y_1),...,(X_n,Y_n).$ Let $h_1,...,h_N:S\mapsto [-1,1]$ be a dictionary that consists of $N$ functions. For $λ\in {\mathbb{R}}^N,$ denote $f_λ:=\sum_{j=1}^Nλ_jh_j.$ Let $\ell:T\times {\mathbb{R}}\mapsto {\mathbb{R}}$ be a given loss function and suppose it is convex with respect to the second variable. Let $(\ell \bullet f)(x,y):=\ell(y;f(x)).$ Finally, let $Λ\subset {\mathbb{R}}^N$ be the simplex of all probability distributions on $\{1,...,N\}.$ Consider the following penalized empirical risk minimization problem \begin{eqnarray*}\hatλ^{\varepsilon}:={\mathop {argmin}_{λ\in Λ}}\Biggl[P_n(\ell \bullet f_λ)+\varepsilon \sum_{j=1}^Nλ_j\log λ_j\Biggr]\end{eqnarray*} along with its distribution dependent version \begin{eqnarray*}λ^{\varepsilon}:={\mathop {argmin}_{λ\in Λ}}\Biggl[P(\ell \bullet f_λ)+\varepsilon \sum_{j=1}^Nλ_j\log λ_j\Biggr],\end{eqnarray*} where $\varepsilon\geq 0$ is a regularization parameter. It is proved that the ``approximate sparsity'' of $λ^{\varepsilon}$ implies the ``approximate sparsity'' of $\hatλ^{\varepsilon}$ and the impact of ``sparsity'' on bounding the excess risk of the empirical solution is explored. Similar results are also discussed in the case of entropy penalized density estimation.

math.ST↗
arXiv2007

Integral curves of noisy vector fields and statistical problems in diffusion tensor imaging: nonparametric kernel estimation and hypotheses testing

Let $v$ be a vector field in a bounded open set $G\subset {\mathbb {R}}^d$. Suppose that $v$ is observed with a random noise at random points $X_i, i=1,...,n,$ that are independent and uniformly distributed in $G.$ The problem is to estimate the integral curve of the differential equation \[\frac{dx(t)}{dt}=v(x(t)),\qquad t\geq 0,x(0)=x_0\in G,\] starting at a given point $x(0)=x_0\in G$ and to develop statistical tests for the hypothesis that the integral curve reaches a specified set $Γ\subset G.$ We develop an estimation procedure based on a Nadaraya--Watson type kernel regression estimator, show the asymptotic normality of the estimated integral curve and derive differential and integral equations for the mean and covariance function of the limit Gaussian process. This provides a method of tracking not only the integral curve, but also the covariance matrix of its estimate. We also study the asymptotic distribution of the squared minimal distance from the integral curve to a smooth enough surface $Γ\subset G$. Building upon this, we develop testing procedures for the hypothesis that the integral curve reaches $Γ$. The problems of this nature are of interest in diffusion tensor imaging, a brain imaging technique based on measuring the diffusion tensor at discrete locations in the cerebral white matter, where the diffusion of water molecules is typically anisotropic. The diffusion tensor data is used to estimate the dominant orientations of the diffusion and to track white matter fibers from the initial location following these orientations. Our approach brings more rigorous statistical tools to the analysis of this problem providing, in particular, hypothesis testing procedures that might be useful in the study of axonal connectivity of the white matter.

math.ST↗
arXiv2007

2004 IMS Medallion Lecture: Local Rademacher complexities and oracle inequalities in risk minimization

Let $\mathcal{F}$ be a class of measurable functions $f:S\mapsto [0,1]$ defined on a probability space $(S,\mathcal{A},P)$. Given a sample (X_1,...,X_n) of i.i.d. random variables taking values in S with common distribution P, let P_n denote the empirical measure based on (X_1,...,X_n). We study an empirical risk minimization problem $P_nf\to \min$, $f\in \mathcal{F}$. Given a solution $\hat{f}_n$ of this problem, the goal is to obtain very general upper bounds on its excess risk \[\mathcal{E}_P(\hat{f}_n):=P\hat{f}_n-\inf_{f\in \mathcal{F}}Pf,\] expressed in terms of relevant geometric parameters of the class $\mathcal{F}$. Using concentration inequalities and other empirical processes tools, we obtain both distribution-dependent and data-dependent upper bounds on the excess risk that are of asymptotically correct order in many examples. The bounds involve localized sup-norms of empirical and Rademacher processes indexed by functions from the class. We use these bounds to develop model selection techniques in abstract risk minimization problems that can be applied to more specialized frameworks of regression and classification.

math.ST↗
arXiv2006

Empirical graph Laplacian approximation of Laplace--Beltrami operators: Large sample results

Let ${M}$ be a compact Riemannian submanifold of ${{\bf R}^m}$ of dimension $\scriptstyle{d}$ and let ${X_1,...,X_n}$ be a sample of i.i.d. points in ${M}$ with uniform distribution. We study the random operators $$ Δ_{h_n,n}f(p):=\frac{1}{nh_n^{d+2}}\sum_{i=1}^n K(\frac{p-X_i}{h_n})(f(X_i)-f(p)), p\in M $$ where ${K(u):={\frac{1}{(4π)^{d/2}}}e^{-\|u\|^2/4}}$ is the Gaussian kernel and ${h_n\to 0}$ as ${n\to\infty.}$ Such operators can be viewed as graph laplacians (for a weighted graph with vertices at data points) and they have been used in the machine learning literature to approximate the Laplace-Beltrami operator of ${M,}$ ${Δ_Mf}$ (divided by the Riemannian volume of the manifold). We prove several results on a.s. and distributional convergence of the deviations ${Δ_{h_n,n}f(p)-{\frac{1}{|μ|}}Δ_Mf(p)}$ for smooth functions ${f}$ both pointwise and uniformly in ${f}$ and ${p}$ (here ${|μ|=μ(M)}$ and $μ$ is the Riemannian volume measure). In particular, we show that for any class ${\cal F}$ of three times differentiable functions on ${M}$ with uniformly bounded derivatives $$ \sup_{p\in M}\sup_{f\in F}\Big|Δ_{h_n,p}f(p)-\frac{1}{|μ|}Δ_Mf(p)\Big|= O\Big(\sqrt{\frac{\log(1/h_n)}{nh_n^{d+2}}}\Big) a.s. $$ as soon as $$ nh_n^{d+2}/\log h_n^{-1}\to \infty and nh^{d+4}_n/\log h_n^{-1}\to 0, $$ and also prove asymptotic normality of ${Δ_{h_n,p}f(p)-{\frac{1}{|μ|}}Δ_Mf(p)}$ (functional CLT) for a fixed ${p\in M}$ and uniformly in ${f}.$

math.PR↗
arXiv2006

High Dimensional Probability

About forty years ago it was realized by several researchers that the essential features of certain objects of Probability theory, notably Gaussian processes and limit theorems, may be better understood if they are considered in settings that do not impose structures extraneous to the problems at hand. For instance, in the case of sample continuity and boundedness of Gaussian processes, the essential feature is the metric or pseudometric structure induced on the index set by the covariance structure of the process, regardless of what the index set may be. This point of view ultimately led to the Fernique-Talagrand majorizing measure characterization of sample boundedness and continuity of Gaussian processes, thus solving an important problem posed by Kolmogorov. Similarly, separable Banach spaces provided a minimal setting for the law of large numbers, the central limit theorem and the law of the iterated logarithm, and this led to the elucidation of the minimal (necessary and/or sufficient) geometric properties of the space under which different forms of these theorems hold. However, in light of renewed interest in Empirical processes, a subject that has considerably influenced modern Statistics, one had to deal with a non-separable Banach space, namely $\mathcal{L}_{\infty}$. With separability discarded, the techniques developed for Gaussian processes and for limit theorems and inequalities in separable Banach spaces, together with combinatorial techniques, led to powerful inequalities and limit theorems for sums of independent bounded processes over general index sets, or, in other words, for general empirical processes.

math.PR↗
arXiv2006

Concentration inequalities and asymptotic results for ratio type empirical processes

Let $\mathcal{F}$ be a class of measurable functions on a measurable space $(S,\mathcal{S})$ with values in $[0,1]$ and let \[P_n=n^{-1}\sum_{i=1}^nδ_{X_i}\] be the empirical measure based on an i.i.d. sample $(X_1,...,X_n)$ from a probability distribution $P$ on $(S,\mathcal{S})$. We study the behavior of suprema of the following type: \[\sup_{r_n<σ_Pf\leq δ_n}\frac{|P_nf-Pf|}{ϕ(σ_Pf)},\] where $σ_Pf\ge\operatorname {Var}^{1/2}_Pf$ and $ϕ$ is a continuous, strictly increasing function with $ϕ(0)=0$. Using Talagrand's concentration inequality for empirical processes, we establish concentration inequalities for such suprema and use them to derive several results about their asymptotic behavior, expressing the conditions in terms of expectations of localized suprema of empirical processes. We also prove new bounds for expected values of sup-norms of empirical processes in terms of the largest $σ_Pf$ and the $L_2(P)$ norm of the envelope of the function class, which are especially suited for estimating localized suprema. With this technique, we extend to function classes most of the known results on ratio type suprema of empirical processes, including some of Alexander's results for VC classes of sets. We also consider applications of these results to several important problems in nonparametric statistics and in learning theory (including general excess risk bounds in empirical risk minimization and their versions for $L_2$-regression and classification and ratio type bounds for margin distributions in classification).

math.PR↗
arXiv2005

Complexities of convex combinations and bounding the generalization error in classification

We introduce and study several measures of complexity of functions from the convex hull of a given base class. These complexity measures take into account the sparsity of the weights of a convex combination as well as certain clustering properties of the base functions involved in it. We prove new upper confidence bounds on the generalization error of ensemble (voting) classification algorithms that utilize the new complexity measures along with the empirical distributions of classification margins, providing a better explanation of generalization performance of large margin classification methods.

math.PR↗
arXiv2004

Weighted uniform consistency of kernel density estimators

Let f_n denote a kernel density estimator of a continuous density f in d dimensions, bounded and positive. Let Ψ(t) be a positive continuous function such that \|Ψf^β\|_{\infty}<\infty for some 0<β<1/2. Under natural smoothness conditions, necessary and sufficient conditions for the sequence \sqrt\frac{nh_n^d}{2|\log h_n^d|}\|Ψ(t)(f_n(t)-Ef_n(t))\|_{\infty} to be stochastically bounded and to converge a.s. to a constant are obtained. Also, the case of larger values of βis studied where a similar sequence with a different norming converges a.s. either to 0 or to +\infty, depending on convergence or divergence of a certain integral involving the tail probabilities of Ψ(X). The results apply as well to some discontinuous not strictly positive densities.

math.PR↗
arXiv2004

Some Local Measures of Complexity of Convex Hulls and Generalization Bounds

We investigate measures of complexity of function classes based on continuity moduli of Gaussian and Rademacher processes. For Gaussian processes, we obtain bounds on the continuity modulus on the convex hull of a function class in terms of the same quantity for the class itself. We also obtain new bounds on generalization error in terms of localized Rademacher complexities. This allows us to prove new results about generalization performance for convex hulls in terms of characteristics of the base class. As a byproduct, we obtain a simple proof of some of the known bounds on the entropy of convex hulls.

math.PR↗