arXiv subjects

Nikita Zhivotovskiy

Publications and source records attributed to Nikita Zhivotovskiy.

46 records · Page 3Linked to original sources

arXiv2020

Sharper bounds for uniformly stable algorithms

Deriving generalization bounds for stable algorithms is a classical question in learning theory taking its roots in the early works by Vapnik and Chervonenkis (1974) and Rogers and Wagner (1978). In a series of recent breakthrough papers by Feldman and Vondrak (2018, 2019), it was shown that the best known high probability upper bounds for uniformly stable learning algorithms due to Bousquet and Elisseef (2002) are sub-optimal in some natural regimes. To do so, they proved two generalization bounds that significantly outperform the simple generalization bound of Bousquet and Elisseef (2002). Feldman and Vondrak also asked if it is possible to provide sharper bounds and prove corresponding high probability lower bounds. This paper is devoted to these questions: firstly, inspired by the original arguments of Feldman and Vondrak (2019), we provide a short proof of the moment bound that implies the generalization bound stronger than both recent results (Feldman and Vondrak, 2018, 2019). Secondly, we prove general lower bounds, showing that our moment bound is sharp (up to a logarithmic factor) unless some additional properties of the corresponding random variables are used. Our main probabilistic result is a general concentration inequality for weakly correlated random variables, which may be of independent interest.

cs.LG↗

arXiv2020

Proper Learning, Helly Number, and an Optimal SVM Bound

The classical PAC sample complexity bounds are stated for any Empirical Risk Minimizer (ERM) and contain an extra logarithmic factor $\log(1/ε)$ which is known to be necessary for ERM in general. It has been recently shown by Hanneke (2016) that the optimal sample complexity of PAC learning for any VC class C is achieved by a particular improper learning algorithm, which outputs a specific majority-vote of hypotheses in C. This leaves the question of when this bound can be achieved by proper learning algorithms, which are restricted to always output a hypothesis from C. In this paper we aim to characterize the classes for which the optimal sample complexity can be achieved by a proper learning algorithm. We identify that these classes can be characterized by the dual Helly number, which is a combinatorial parameter that arises in discrete geometry and abstract convexity. In particular, under general conditions on C, we show that the dual Helly number is bounded if and only if there is a proper learner that obtains the optimal joint dependence on $ε$ and $δ$. As further implications of our techniques we resolve a long-standing open problem posed by Vapnik and Chervonenkis (1974) on the performance of the Support Vector Machine by proving that the sample complexity of SVM in the realizable case is $Θ((n/ε)+(1/ε)\log(1/δ))$, where $n$ is the dimension. This gives the first optimal PAC bound for Halfspaces achieved by a proper learning algorithm, and moreover is computationally efficient.

cs.LG↗

arXiv2020

Noise sensitivity of the top eigenvector of a Wigner matrix

We investigate the noise sensitivity of the top eigenvector of a Wigner matrix in the following sense. Let $v$ be the top eigenvector of an $N\times N$ Wigner matrix. Suppose that $k$ randomly chosen entries of the matrix are resampled, resulting in another realization of the Wigner matrix with top eigenvector $v^{[k]}$. We prove that, with high probability, when $k \ll N^{5/3-o(1)}$, then $v$ and $v^{[k]}$ are almost collinear and when $k\gg N^{5/3}$, then $v^{[k]}$ is almost orthogonal to $v$.

math.PR↗

arXiv2020

When are epsilon-nets small?

In many interesting situations the size of epsilon-nets depends only on $ε$ together with different complexity measures. The aim of this paper is to give a systematic treatment of such complexity measures arising in Discrete and Computational Geometry and Statistical Learning, and to bridge the gap between the results appearing in these two fields. As a byproduct, we obtain several new upper bounds on the sizes of epsilon-nets that generalize/improve the best known general guarantees. In particular, our results work with regimes when small epsilon-nets of size $o(\frac{1}ε)$ exist, which are not usually covered by standard upper bounds. Inspired by results in Statistical Learning we also give a short proof of the Haussler's upper bound on packing numbers.

cs.CG↗

arXiv2019

Uniform Hanson-Wright type concentration inequalities for unbounded entries via the entropy method

This paper is devoted to uniform versions of the Hanson-Wright inequality for a random vector $X \in \mathbb{R}^n$ with independent subgaussian components. The core technique of the paper is based on the entropy method combined with truncations of both gradients of functions of interest and of the coordinates of $X$ itself. Our results recover, in particular, the classic uniform bound of Talagrand (1996) for Rademacher chaoses and the more recent uniform result of Adamczak (2015), which holds under certain rather strong assumptions on the distribution of $X$. We provide several applications of our techniques: we establish a version of the standard Hanson-Wright inequality, which is tighter in some regimes. Extending our techniques we show a version of the dimension-free matrix Bernstein inequality that holds for random matrices with a subexponential spectral norm. We apply the derived inequality to the problem of covariance estimation with missing observations and prove an almost optimal high probability version of the recent result of Lounici (2014). Finally, we show a uniform Hanson-Wright type inequality in the Ising model under Dobrushin's condition. A closely related question was posed by Marton (2003).

math.PR↗

arXiv2019

Robust covariance estimation under $L_4-L_2$ norm equivalence

Let $X$ be a centered random vector taking values in $\mathbb{R}^d$ and let $Σ= \mathbb{E}(X\otimes X)$ be its covariance matrix. We show that if $X$ satisfies an $L_4-L_2$ norm equivalence, there is a covariance estimator $\hatΣ$ that exhibits the optimal performance one would expect had $X$ been a gaussian vector. The procedure also improves the current state-of-the-art regarding high probability bounds in the subgaussian case (sharp results were only known in expectation or with constant probability). In both scenarios the new bound does not depend explicitly on the dimension $d$, but rather on the effective rank of the covariance matrix $Σ$.

math.ST↗

arXiv2018

Concentration of the spectral norm of Erdős-Rényi random graphs

We present results on the concentration properties of the spectral norm $\|A_p\|$ of the adjacency matrix $A_p$ of an Erdős-Rényi random graph $G(n,p)$. First we consider the Erdős-Rényi random graph process and prove that $\|A_p\|$ is uniformly concentrated over the range $p\in [C\log n/n,1]$. The analysis is based on delocalization arguments, uniform laws of large numbers, together with the entropy method to prove concentration inequalities. As an application of our techniques we prove sharp sub-Gaussian moment inequalities for $\|A_p\|$ for all $p\in [c\log^3n/n,1]$ that improve the general bounds of Alon, Krivelevich, and Vu (2001) and some of the more recent results of Erdős et al. (2013). Both results are consistent with the asymptotic result of Füredi and Komlós (1981) that holds for fixed $p$ as $n\to \infty$.

math.PR↗

arXiv2018

Optimal learning via local entropies and sample compression

The aim of this paper is to provide several novel upper bounds on the excess risk with a primal focus on classification problems. We suggest two approaches and the obtained bounds are represented via the distribution dependent local entropies of the classes or the sizes of specific sample com- pression schemes. We show that in some cases, our guarantees are optimal up to constant factors and outperform previously known results. As an application of our results, we provide a new tight PAC bound for the hard-margin SVM, an extended analysis of certain empirical risk minimizers under log-concave distributions, a new variant of an online to batch conversion, and distribution dependent localized bounds in the aggregation framework. We also develop techniques that allow to replace empirical covering number or covering numbers with bracketing by the coverings with respect to the distribution of the data. The proofs for the sample compression schemes are based on the moment method combined with the analysis of voting algorithms.

math.ST↗

arXiv2017

Localization of VC Classes: Beyond Local Rademacher Complexities

In this paper we introduce an alternative localization approach for binary classification that leads to a novel complexity measure: fixed points of the local empirical entropy. We show that this complexity measure gives a tight control over complexity in the upper bounds. Our results are accompanied by a novel minimax lower bound that involves the same quantity. In particular, we practically answer the question of optimality of ERM under bounded noise for general VC classes.

math.ST↗

arXiv2016

Permutational Rademacher Complexity: a New Complexity Measure for Transductive Learning

Transductive learning considers situations when a learner observes $m$ labelled training points and $u$ unlabelled test points with the final goal of giving correct answers for the test points. This paper introduces a new complexity measure for transductive learning called Permutational Rademacher Complexity (PRC) and studies its properties. A novel symmetrization inequality is proved, which shows that PRC provides a tighter control over expected suprema of empirical processes compared to what happens in the standard i.i.d. setting. A number of comparison results are also provided, which show the relation between PRC and other popular complexity measures used in statistical learning theory, including Rademacher complexity and Transductive Rademacher Complexity (TRC). We argue that PRC is a more suitable complexity measure for transductive learning. Finally, these results are combined with a standard concentration argument to provide novel data-dependent risk bounds for transductive learning.

stat.ML↗