arXiv subjects

Matthieu Lerasle

Publications and source records attributed to Matthieu Lerasle.

31 records · Page 2Linked to original sources

arXiv2017

Robust machine learning by median-of-means : theory and practice

We introduce new estimators for robust machine learning based on median-of-means (MOM) estimators of the mean of real valued random variables. These estimators achieve optimal rates of convergence under minimal assumptions on the dataset. The dataset may also have been corrupted by outliers on which no assumption is granted. We also analyze these new estimators with standard tools from robust statistics. In particular, we revisit the concept of breakdown point. We modify the original definition by studying the number of outliers that a dataset can contain without deteriorating the estimation properties of a given estimator. This new notion of breakdown number, that takes into account the statistical performances of the estimators, is non-asymptotic in nature and adapted for machine learning purposes. We proved that the breakdown number of our estimator is of the order of (number of observations)*(rate of convergence). For instance, the breakdown number of our estimators for the problem of estimation of a d-dimensional vector with a noise variance sigma^2 is sigma^2d and it becomes sigma^2 s log(d/s) when this vector has only s non-zero component. Beyond this breakdown point, we proved that the rate of convergence achieved by our estimator is (number of outliers) divided by (number of observation). Besides these theoretical guarantees, the major improvement brought by these new estimators is that they are easily computable in practice. In fact, basically any algorithm used to approximate the standard Empirical Risk Minimizer (or its regularized versions) has a robust version approximating our estimators. As a proof of concept, we study many algorithms for the classical LASSO estimator. A byproduct of the MOM algorithms is a measure of depth of data that can be used to detect outliers.

math.ST↗

arXiv2017

Cross-validation improved by aggregation: Agghoo

Cross-validation is widely used for selecting among a family of learning rules. This paper studies a related method, called aggregated hold-out (Agghoo), which mixes cross-validation with aggregation; Agghoo can also be related to bagging. According to numerical experiments, Agghoo can improve significantly cross-validation's prediction error, at the same computational cost; this makes it very promising as a general-purpose tool for prediction. We provide the first theoretical guarantees on Agghoo, in the supervised classification setting, ensuring that one can use it safely: at worse, Agghoo performs like the hold-out, up to a constant factor. We also prove a non-asymptotic oracle inequality, in binary classification under the margin condition, which is sharp enough to get (fast) minimax rates.

math.ST↗

arXiv2016

Non parametric estimation for random walks in random environment

We consider a random walk in i.i.d. random environment with distribution $ν$ on Z. The problem we are interested in is to provide an estimator of the cumulative distribution function (c.d.f.) F of $ν$ from the observation of one trajectory of the random walk. For that purpose we first estimate the moments of $ν$, then combine these moment estimators to obtain a collection of estimators (F M n) M $\ge$1 of F , our final estimator is chosen among this collection by Lepskii's method. This estimator is therefore easily computable in practice. We derive convergence rates for this estimator depending on the H{ö}lder regularity of F and on the divergence rate of the walk. Our rate is optimal when the chain realizes a trade-off between a fast exploration of the sites, allowing to get more informations and a larger number of visits of each sites, allowing a better recovery of the environment itself.

math.ST↗

arXiv2016

Sharp oracle inequalities and slope heuristic for specification probabilities estimation in discrete random fields

We study the problem of estimating the one-point specification probabilities in non-necessary finite discrete random fields from partially observed independent samples. Our procedures are based on model selection by minimization of a penalized empirical criterion. The selected estimators satisfy sharp oracle inequalities in $L_2$-risk. We also obtain theoretical results on the slope heuristic for this problem, justifying the slope algorithm to calibrate the leading constant in the penalty. The practical performances of our methods are investigated in two simulation studies. We illustrate the usefulness of our approach by applying the methods to a multi-unit neuronal data from a rat hippocampus.

math.ST↗

arXiv2015

Choice of V for V-Fold Cross-Validation in Least-Squares Density Estimation

This paper studies V-fold cross-validation for model selection in least-squares density estimation. The goal is to provide theoretical grounds for choosing V in order to minimize the least-squares loss of the selected estimator. We first prove a non-asymptotic oracle inequality for V-fold cross-validation and its bias-corrected version (V-fold penalization). In particular, this result implies that V-fold penalization is asymptotically optimal in the nonparametric case. Then, we compute the variance of V-fold cross-validation and related criteria, as well as the variance of key quantities for model selection performance. We show that these variances depend on V like 1+4/(V-1), at least in some particular cases, suggesting that the performance increases much from V=2 to V=5 or 10, and then is almost constant. Overall, this can explain the common advice to take V=5---at least in our setting and when the computational power is limited---, as supported by some simulation experiments. An oracle inequality and exact formulas for the variance are also proved for Monte-Carlo cross-validation, also known as repeated cross-validation, where the parameter V is replaced by the number B of random splits of the data.

math.ST↗

arXiv2015

The number of potential winners in Bradley-Terry model in random environment

We consider a Bradley-Terry model in random environment where each player faces each other once. More precisely the strengths of the players are assumed to be random and we study the influence of their distributions on the asymptotic number of potential winners.First we prove that under mild assumptions, mainly on their moments, if the strengths are unbounded, the asymptotic probability that the best player wins is 1. We also exhibit a sufficient convexity condition to obtain the same result when the strengths are bounded. When this last condition fails, the number of potential winners grows at a rate depending on the tail of the distribution of strengths. We also study the minimal strength required for an additional player to win in this last case.

math.PR↗

arXiv2015

Sub-Gaussian mean estimators

We discuss the possibilities and limitations of estimating the mean of a real-valued random variable from independent and identically distributed observations from a non-asymptotic point of view. In particular, we define estimators with a sub-Gaussian behavior even for certain heavy-tailed distributions. We also prove various impossibility results for mean estimators.

math.ST↗

arXiv2012

Markov Approximations of chains of infinite order in the $\bar{d}$-metric

We derive explicit upper bounds for the $\bar{d}$-distance between a chain of infinite order and its canonical $k$-steps Markov approximation. Our proof is entirely constructive and involves a "coupling from the past" argument. The new method covers non necessarily continuous probability kernels, and chains with null transition probabilities. These results imply in particular the Bernoulli property for these processes.

math.PR↗

arXiv2011

Optimal model selection for density estimation of stationary data under various mixing conditions

We propose a block-resampling penalization method for marginal density estimation with nonnecessary independent observations. When the data are $β$ or $τ$-mixing, the selected estimator satisfies oracle inequalities with leading constant asymptotically equal to 1. We also prove in this setting the slope heuristic, which is a data-driven method to optimize the leading constant in the penalty.

math.ST↗

arXiv2010

An Oracle Approach for Interaction Neighborhood Estimation in Random Fields

We consider the problem of interaction neighborhood estimation from the partial observation of a finite number of realizations of a random field. We introduce a model selection rule to choose estimators of conditional probabilities among natural candidates. Our main result is an oracle inequality satisfied by the resulting estimator. We use then this selection rule in a two-step procedure to evaluate the interacting neighborhoods. The selection rule selects a small prior set of possible interacting points and a cutting step remove from this prior set the irrelevant points. We also prove that the Ising models satisfy the assumptions of the main theorems, without restrictions on the temperature, on the structure of the interacting graph or on the range of the interactions. It provides therefore a large class of applications for our results. We give a computationally efficient procedure in these models. We finally show the practical efficiency of our approach in a simulation study.

math.ST↗

arXiv2010

Optimal model selection in density estimation

We build penalized least-squares estimators using the slope heuristic and resampling penalties. We prove oracle inequalities for the selected estimator with leading constant asymptotically equal to 1. We compare the practical performances of these methods in a short simulation study.

math.ST↗

arXiv2010

Adaptive non-asymptotic confidence balls in density estimation

We build confidence balls for the common density $s$ of a real valued sample $X_1,...,X_n$. We use resampling methods to estimate the projection of $s$ onto finite dimensional linear spaces and a model selection procedure to choose an optimal approximation space. The covering property is ensured for all $n\geq2$ and the balls are adaptive over a collection of linear spaces.

math.ST↗

arXiv2009

Adaptive density estimation for stationary processes

We propose an algorithm to estimate the common density $s$ of a stationary process $X_1,...,X_n$. We suppose that the process is either $β$ or $τ$-mixing. We provide a model selection procedure based on a generalization of Mallows' $C_p$ and we prove oracle inequalities for the selected estimator under a few prior assumptions on the collection of models and on the mixing coefficients. We prove that our estimator is adaptive over a class of Besov spaces, namely, we prove that it achieves the same rates of convergence as in the i.i.d framework.

math.ST↗