arXiv subjects

Peter J. Rousseeuw

Publications and source records attributed to Peter J. Rousseeuw.

49 records · Page 3Linked to original sources

arXiv2018

Robust Monitoring of Time Series with Application to Fraud Detection

Time series often contain outliers and level shifts or structural changes. These unexpected events are of the utmost importance in fraud detection, as they may pinpoint suspicious transactions. The presence of such unusual events can easily mislead conventional time series analysis and yield erroneous conclusions. In this paper we provide a unified framework for detecting outliers and level shifts in short time series that may have a seasonal pattern. The approach combines ideas from the FastLTS algorithm for robust regression with alternating least squares. The double wedge plot is proposed, a graphical display which indicates outliers and potential level shifts. The methodology was developed to detect potential fraud cases in time series of imports into the European Union, and is illustrated on two such series.

stat.CO↗

arXiv2018

Discussion of "The power of monitoring"

This is an invited comment on the discussion paper "The power of monitoring: how to make the most of a contaminated multivariate sample" by A. Cerioli, M. Riani, A. Atkinson and A. Corbellini that will appear in the journal Statistical Methods & Applications.

stat.ME↗

arXiv2017

Anomaly Detection by Robust Statistics

Real data often contain anomalous cases, also known as outliers. These may spoil the resulting analysis but they may also contain valuable information. In either case, the ability to detect such anomalies is essential. A useful tool for this purpose is robust statistics, which aims to detect the outliers by first fitting the majority of the data and then flagging data points that deviate from it. We present an overview of several robust methods and the resulting graphical outlier detection tools. We discuss robust procedures for univariate, low-dimensional, and high-dimensional data, such as estimating location and scatter, linear regression, principal component analysis, classification, clustering, and functional data analysis. Also the challenging new topic of cellwise outliers is introduced.

stat.ML↗

arXiv2017

Detecting deviating data cells

A multivariate dataset consists of $n$ cases in $d$ dimensions, and is often stored in an $n$ by $d$ data matrix. It is well-known that real data may contain outliers. Depending on the situation, outliers may be (a) undesirable errors which can adversely affect the data analysis, or (b) valuable nuggets of unexpected information. In statistics and data analysis the word outlier usually refers to a row of the data matrix, and the methods to detect such outliers only work when at least half the rows are clean. But often many rows have a few contaminated cell values, which may not be visible by looking at each variable (column) separately. We propose the first method to detect deviating data cells in a multivariate sample which takes the correlations between the variables into account. It has no restriction on the number of clean rows, and can deal with high dimensions. Other advantages are that it provides estimates of the `expected' values of the outlying cells, while imputing missing values at the same time. We illustrate the method on several real data sets, where it uncovers more structure than found by purely columnwise methods or purely rowwise methods. The proposed method can help to diagnose why a certain row is outlying, e.g. in process control. It may also serve as an initial step for estimating multivariate location and scatter matrices.

stat.ME↗

arXiv2017

Minimum Covariance Determinant and Extensions

The Minimum Covariance Determinant (MCD) method is a highly robust estimator of multivariate location and scatter, for which a fast algorithm is available. Since estimating the covariance matrix is the cornerstone of many multivariate statistical methods, the MCD is an important building block when developing robust multivariate techniques. It also serves as a convenient and efficient tool for outlier detection. The MCD estimator is reviewed, along with its main properties such as affine equivariance, breakdown value, and influence function. We discuss its computation, and list applications and extensions of the MCD in applied and methodological multivariate statistics. Two recent extensions of the MCD are described. The first one is a fast deterministic algorithm which inherits the robustness of the MCD while being almost affine equivariant. The second is tailored to high-dimensional data, possibly with more dimensions than cases, and incorporates regularization to prevent singular matrices.

stat.ME↗

arXiv2017

Econometric applications of high-breakdown robust regression techniques

A literature search shows that robust regression techniques are rarely used in applied econometrics. We list several misconceptions about robustness which lead to this situation. We show that most data sets are not normal, least squares performs very poorly even in large data sets with small numbers of outliers, and that commonly used techniques for achieving robustness fail to do so. We then provide newly developed techniques from the statistics literature which are easy to understand, and achieve robustness. We show the practical use of these techniques by re-analyzing three regression models from recent literature, and arriving at different conclusions from those reached by the authors.

stat.AP↗

arXiv2017

Dissimilar Symmetric Word Pairs in the Human Genome

In this work we explore the dissimilarity between symmetric word pairs, by comparing the inter-word distance distribution of a word to that of its reversed complement. We propose a new measure of dissimilarity between such distributions. Since symmetric pairs with different patterns could point to evolutionary features, we search for the pairs with the most dissimilar behaviour. We focus our study on the complete human genome and its repeat-masked version.

stat.AP↗

arXiv2017

A Measure of Directional Outlyingness with Applications to Image Data and Video

Functional data covers a wide range of data types. They all have in common that the observed objects are functions of of a univariate argument (e.g. time or wavelength) or a multivariate argument (say, a spatial position). These functions take on values which can in turn be univariate (such as the absorbance level) or multivariate (such as the red/green/blue color levels of an image). In practice it is important to be able to detect outliers in such data. For this purpose we introduce a new measure of outlyingness that we compute at each gridpoint of the functions' domain. The proposed Directional Outlyingness} (DO) measure accounts for skewness in the data and only requires O(n) computation time per direction. We derive the influence function of the DO and compute a cutoff for outlier detection. The resulting heatmap and functional outlier map reflect local and global outlyingness of a function. To illustrate the performance of the method on real data it is applied to spectra, MRI images, and video surveillance data.

stat.ME↗

arXiv2016

Multivariate and functional classification using depth and distance

We construct classifiers for multivariate and functional data. Our approach is based on a kind of distance between data points and classes. The distance measure needs to be robust to outliers and invariant to linear transformations of the data. For this purpose we can use the bagdistance which is based on halfspace depth. It satisfies most of the properties of a norm but is able to reflect asymmetry when the class is skewed. Alternatively we can compute a measure of outlyingness based on the skew-adjusted projection depth. In either case we propose the DistSpace transform which maps each data point to the vector of its distances to all classes, followed by k-nearest neighbor (kNN) classification of the transformed data points. This combines invariance and robustness with the simplicity and wide applicability of kNN. The proposal is compared with other methods in experiments with real and simulated data.

stat.ME↗

arXiv2016

Finding Outliers in Surface Data and Video

Surface, image and video data can be considered as functional data with a bivariate domain. To detect outlying surfaces or images, a new method is proposed based on the mean and the variability of the degree of outlyingness at each grid point. A rule is constructed to flag the outliers in the resulting functional outlier map. Heatmaps of their outlyingness indicate the regions which are most deviating from the regular surfaces. The method is applied to fluorescence excitation-emission spectra after fitting a PARAFAC model, to MRI image data which are augmented with their gradients, and to video surveillance data.

stat.ME↗

arXiv2015

Statistical depth meets computational geometry: a short survey

During the past two decades there has been a lot of interest in developing statistical depth notions that generalize the univariate concept of ranking to multivariate data. The notion of depth has also been extended to regression models and functional data. However, computing such depth functions as well as their contours and deepest points is not trivial. Techniques of computational geometry appear to be well-suited for the development of such algorithms. Both the statistical and the computational geometry communities have done much work in this direction, often in close collaboration. We give a short survey of this work, focusing mainly on depth and multivariate medians, and end by listing some other areas of statistics where computational geometry has been of great help in constructing efficient algorithms.

stat.ME↗

arXiv2008

High-Breakdown Robust Multivariate Methods

When applying a statistical method in practice it often occurs that some observations deviate from the usual assumptions. However, many classical methods are sensitive to outliers. The goal of robust statistics is to develop methods that are robust against the possibility that one or several unannounced outliers may occur anywhere in the data. These methods then allow to detect outlying observations by their residuals from a robust fit. We focus on high-breakdown methods, which can deal with a substantial fraction of outliers in the data. We give an overview of recent high-breakdown robust methods for multivariate settings such as covariance estimation, multiple and multivariate regression, discriminant analysis, principal components and multivariate calibration.

stat.ME↗

arXiv2005

Breakdown and groups

The concept of breakdown point was introduced by Hampel [Ph.D. dissertation (1968), Univ. California, Berkeley; Ann. Math. Statist. 42 (1971) 1887-1896] and developed further by, among others, Huber [Robust Statistics (1981). Wiley, New York] and Donoho and Huber [In A Festschrift for Erich L. Lehmann (1983) 157-184. Wadsworth, Belmont, CA]. It has proved most successful in the context of location, scale and regression problems. Attempts to extend the concept to other situations have not met with general acceptance. In this paper we argue that this is connected to the fact that in the location, scale and regression problems the translation and affine groups give rise to a definition of equivariance for statistical functionals. Comparisons in terms of breakdown points seem only useful when restricted to equivariant functionals and even here the connection between breakdown and equivariance is a tenuous one.

math.ST↗