arXiv subjects

Victor-Emmanuel Brunel

Publications and source records attributed to Victor-Emmanuel Brunel.

36 records · Page 2Linked to original sources

arXiv2019

Differentially private sub-Gaussian location estimators

We tackle the problem of estimating a location parameter with differential privacy guarantees and sub-Gaussian deviations. Recent work in statistics has focused on the study of estimators that achieve sub-Gaussian type deviations even for heavy tailed data. We revisit some of these estimators through the lens of differential privacy and show that a naive application of the Laplace mechanism can lead to sub-optimal results. We design two private algorithms for estimating the median that lead to estimators with sub-Gaussian type errors. Unlike most existing differentially private median estimators, both algorithms are well defined for unbounded random variables that are not even required to have finite moments. We then turn to the problem of sub-Gaussian mean estimation and show that under heavy tails natural differentially private alternatives lead to strictly worse deviations than their non-private sub-Gaussian counterparts. This is in sharp contrast with recent results that show that from an asymptotic perspective the cost of differential privacy is negligible.

math.ST↗

arXiv2019

A nonasymptotic law of iterated logarithm for general M-estimators

M-estimators are ubiquitous in machine learning and statistical learning theory. They are used both for defining prediction strategies and for evaluating their precision. In this paper, we propose the first non-asymptotic "any-time" deviation bounds for general M-estimators, where "any-time" means that the bound holds with a prescribed probability for every sample size. These bounds are nonasymptotic versions of the law of iterated logarithm. They are established under general assumptions such as Lipschitz continuity of the loss function and (local) curvature of the population risk. These conditions are satisfied for most examples used in machine learning, including those ensuring robustness to outliers and to heavy tailed distributions. As an example of application, we consider the problem of best arm identification in a parametric stochastic multi-arm bandit setting. We show that the established bound can be converted into a new algorithm, with provably optimal theoretical guarantees. Numerical experiments illustrating the validity of the algorithm are reported.

math.ST↗

arXiv2019

Best Arm Identification for Contaminated Bandits

This paper studies active learning in the context of robust statistics. Specifically, we propose a variant of the Best Arm Identification problem for \emph{contaminated bandits}, where each arm pull has probability $\varepsilon$ of generating a sample from an arbitrary contamination distribution instead of the true underlying distribution. The goal is to identify the best (or approximately best) true distribution with high probability, with a secondary goal of providing guarantees on the quality of this distribution. The primary challenge of the contaminated bandit setting is that the true distributions are only partially identifiable, even with infinite samples. To address this, we develop tight, non-asymptotic sample complexity bounds for high-probability estimation of the first two robust moments (median and median absolute deviation) from contaminated samples. These concentration inequalities are the main technical contributions of the paper and may be of independent interest. Using these results, we adapt several classical Best Arm Identification algorithms to the contaminated bandit setting and derive sample complexity upper bounds for our problem. Finally, we provide matching information-theoretic lower bounds on the sample complexity (up to a small logarithmic factor).

math.ST↗

arXiv2019

Learning rates for Gaussian mixtures under group invariance

We study the pointwise maximum likelihood estimation rates for a class of Gaussian mixtures that are invariant under the action of some isometry group. This model is also known as multi-reference alignment, where random isometries of a given vector are observed, up to Gaussian noise. We completely characterize the speed of the maximum likelihood estimator, by giving a comprehensive description of the likelihood geometry of the model. We show that the unknown parameter can always be decomposed into two components, one of which can be estimated at the fast rate $n^{-1/2}$, the other one being estimated at the slower rate $n^{-1/4}$. We provide an algebraic description and a geometric interpretation of these facts.

math.ST↗

arXiv2018

Learning Signed Determinantal Point Processes through the Principal Minor Assignment Problem

Symmetric determinantal point processes (DPP's) are a class of probabilistic models that encode the random selection of items that exhibit a repulsive behavior. They have attracted a lot of attention in machine learning, when returning diverse sets of items is sought for. Sampling and learning these symmetric DPP's is pretty well understood. In this work, we consider a new class of DPP's, which we call signed DPP's, where we break the symmetry and allow attractive behaviors. We set the ground for learning signed DPP's through a method of moments, by solving the so called principal assignment problem for a class of matrices $K$ that satisfy $K_{i,j}=\pm K_{j,i}$, $i\neq j$, in polynomial time.

math.ST↗

arXiv2018

A change-point problem and inference for segment signals

We address the problem of detection and estimation of one or two change-points in the mean of a series of random variables. We use the formalism of set estimation in regression: To each point of a design is attached a binary label that indicates whether that point belongs to an unknown segment and this label is contaminated with noise. The endpoints of the unknown segment are the change-points. We study the minimal size of the segment which allows statistical detection in different scenarios, including when the endpoints are separated from the boundary of the domain of the design, or when they are separated from one another. We compare this minimal size with the minimax rates of convergence for estimation of the segment under the same scenarios. The aim of this extensive study of a simple yet fundamental version of the change-point problem is twofold: Understanding the impact of the location and the separation of the change points on detection and estimation and bringing insights about the estimation and detection of convex bodies in higher dimensions.

math.ST↗

arXiv2018

Methods for Estimation of Convex Sets

In the framework of shape constrained estimation, we review methods and works done in convex set estimation. These methods mostly build on stochastic and convex geometry, empirical process theory, functional analysis, linear programming, extreme value theory, etc. The statistical problems that we review include density support estimation, estimation of the level sets of densities or depth functions, nonparametric regression, etc. We focus on the estimation of convex sets under the Nikodym and Hausdorff metrics, which require different techniques and, quite surprisingly, lead to very different results, in particular in density support estimation. Finally, we discuss computational issues in high dimensions.

math.ST↗

arXiv2018

Uniform Behaviors of Random Polytopes under the Hausdorff Metric

We study the Hausdorff distance between a random polytope, defined as the convex hull of i.i.d. random points, and the convex hull of the support of their distribution. As particular examples, we consider uniform distributions on convex bodies, densities that decay at a certain rate when approaching the boundary of a convex body, projections of uniform distributions on higher dimensional convex bodies and uniform distributions on the boundary of convex bodies. We essentially distinguish two types of convex bodies: those with a smooth boundary and polytopes. In the case of uniform distributions, we prove that, in some sense, the random polytope achieves its best statistical accuracy under the Hausdorff metric when the support has a smooth boundary and its worst statistical accuracy when the support is a polytope. This is somewhat surprising, since the exact opposite is true under the Nikodym metric. We prove rate optimality of most our results in a minimax sense. In the case of uniform distributions, we extend our results to a rescaled version of the Hausdorff metric. We also tackle the estimation of functionals of the support of a distribution such as its mean width and its diameter. Finally, we show that high dimensional random polytopes can be approximated with simple polyhedral representations that significantly decrease their computational complexity without affecting their statistical accuracy.

math.ST↗

arXiv2018

Estimation of convex supports from noisy measurements

A popular class of problem in statistics deals with estimating the support of a density from $n$ observations drawn at random from a $d$-dimensional distribution. The one-dimensional case reduces to estimating the end points of a univariate density. In practice, an experimenter may only have access to a noisy version of the original data. Therefore, a more realistic model allows for the observations to be contaminated with additive noise. In this paper, we consider estimation of convex bodies when the additive noise is distributed according to a multivariate Gaussian distribution, even though our techniques could easily be adapted to other noise distributions. Unlike standard methods in deconvolution that are implemented by thresholding a kernel density estimate, our method avoids tuning parameters and Fourier transforms altogether. We show that our estimator, computable in $(O(\ln n))^{(d-1)/2}$ time, converges at a rate of $ O_d(\log\log n/\sqrt{\log n}) $ in Hausdorff distance, in accordance with the polylogarithmic rates encountered in Gaussian deconvolution problems. Part of our analysis also involves the optimality of the proposed estimator. We provide a lower bound for the minimax rate of estimation in Hausdorff distance that is $Ω_d(1/\log^2 n)$.

math.ST↗

arXiv2017

Maximum likelihood estimation of determinantal point processes

Determinantal point processes (DPPs) have wide-ranging applications in machine learning, where they are used to enforce the notion of diversity in subset selection problems. Many estimators have been proposed, but surprisingly the basic properties of the maximum likelihood estimator (MLE) have received little attention. The difficulty is that it is a non-concave maximization problem, and such functions are notoriously difficult to understand in high dimensions, despite their importance in modern machine learning. Here we study both the local and global geometry of the expected log-likelihood function. We prove several rates of convergence for the MLE and give a complete characterization of the case where these are parametric. We also exhibit a potential curse of dimensionality where the asymptotic variance of the MLE scales exponentially with the dimension of the problem. Moreover, we exhibit an exponential number of saddle points, and give evidence that these may be the only critical points.

math.ST↗

arXiv2017

Rates of estimation for determinantal point processes

Determinantal point processes (DPPs) have wide-ranging applications in machine learning, where they are used to enforce the notion of diversity in subset selection problems. Many estimators have been proposed, but surprisingly the basic properties of the maximum likelihood estimator (MLE) have received little attention. In this paper, we study the local geometry of the expected log-likelihood function to prove several rates of convergence for the MLE. We also give a complete characterization of the case where the MLE converges at a parametric rate. Even in the latter case, we also exhibit a potential curse of dimensionality where the asymptotic variance of the MLE is exponentially large in the dimension of the problem.

math.ST↗

arXiv2017

Uniform deviation and moment inequalities for random polytopes with general densities in arbitrary convex bodies

We prove an exponential deviation inequality for the convex hull of a finite sample of i.i.d. random points with a density supported on an arbitrary convex body in $\R^d$, $d\geq 2$. When the density is uniform, our result yields rate optimal upper bounds for all the moments of the missing volume of the convex hull, uniformly over all convex bodies of $\R^d$: We make no restrictions on their volume, location in the space or smoothness of their boundary. After extending an identity due to Efron, we also prove upper bounds for the moments of the number of vertices of the random polytope. Surprisingly, these bounds do not depend on the underlying density and we prove that the growth rates that we obtain are tight in a certain sense.

math.PR↗

arXiv2017

Learning Determinantal Point Processes with Moments and Cycles

Determinantal Point Processes (DPPs) are a family of probabilistic models that have a repulsive behavior, and lend themselves naturally to many tasks in machine learning where returning a diverse set of objects is important. While there are fast algorithms for sampling, marginalization and conditioning, much less is known about learning the parameters of a DPP. Our contribution is twofold: (i) we establish the optimal sample complexity achievable in this problem and show that it is governed by a natural parameter, which we call the \emph{cycle sparsity}; (ii) we propose a provably fast combinatorial algorithm that implements the method of moments efficiently and achieves optimal sample complexity. Finally, we give experimental results that confirm our theoretical findings.

math.ST↗

arXiv2017

Concentration of the empirical level sets of Tukey's halfspace depth

Tukey depth, aka halfspace depth, has attracted much interest in data analysis, because it is a natural way of measuring the notion of depth relative to a cloud of points or, more generally, to a probability measure. Given an i.i.d. sample, we investigate the concentration of upper level sets of the Tukey depth relative to that sample around their population version. We also study the concentration of the upper level sets of a discretized version of Tukey depth.

math.ST↗

arXiv2014

Convex set detection

We address the problem of one dimensional segment detection and estimation, in a regression setup. At each point of a fixed or random design, one observes whether that point belongs to the unknown segment or not, up to some additional noise. We try to understand what the minimal size of the segment is so it can be accurately seen by some statistical procedure, and how this minimal size depends on some a priori knowledge about the location of the unknown segment.

math.ST↗

arXiv2013

A universal deviation inequality for random polytopes

We consider the convex hull of a finite sample of i.i.d. points uniformly distributed in a convex body in $\R^d$, $d\geq 2$. We prove an exponential deviation inequality, which leads to rate optimal upper bounds on all the moments of the missing volume of the convex hull, uniformly over all convex bodies of $\R^d$, with no restriction on their volume, location in the space and smoothness of the boundary.

math.ST↗

arXiv2013

Adaptive estimation of convex and polytopal support

We estimate the support of a uniform density, when it is assumed to be a convex polytope or, more generally, a convex body in $\R^d$. In the polytopal case, we construct an estimator achieving a rate which does not depend on the dimension $d$, unlike the other estimators that have been proposed so far. For $d\geq 3$, our estimator has a better risk than the previous ones, and it is nearly minimax, up to a logarithmic factor. We also propose an estimator which is adaptive with respect to the structure of the boundary of the unknown support.

math.ST↗

arXiv2012

Adaptive Estimation of Convex Sets and Convex Polytopes from Noisy Data

We estimate convex polytopes and general convex sets in $\mathbb R^d,d\geq 2$ in the regression framework. We measure the risk of our estimators using a $L^1$-type loss function and prove upper bounds on these risks. We show that, in the case of polytopes, these estimators achieve the minimax rate. For polytopes, this minimax rate is $\frac{\ln n}{n}$, which differs from the parametric rate for non-regular families by a logarithmic factor, and we show that this extra factor is essential. Using polytopal approximations we extend our results to general convex sets, and we achieve the minimax rate up to a logarithmic factor. In addition we provide an estimator that is adaptive with respect to the number of vertices of the unknown polytope, and we prove that this estimator is optimal in all classes of polytopes with a given number of vertices.

math.ST↗