arXiv subjects

Rosa E. Lillo

Publications and source records attributed to Rosa E. Lillo.

29 records · Page 2Linked to original sources

arXiv2021

Detecting and Classifying Outliers in Big Functional Data

We propose two new outlier detection methods, for identifying and classifying different types of outliers in (big) functional data sets. The proposed methods are based on an existing method called Massive Unsupervised Outlier Detection (MUOD). MUOD detects and classifies outliers by computing for each curve, three indices, all based on the concept of linear regression and correlation, which measure outlyingness in terms of shape, magnitude and amplitude, relative to the other curves in the data. 'Semifast-MUOD', the first method, uses a sample of the observations in computing the indices, while 'Fast-MUOD', the second method, uses the point-wise or $L_1$ median in computing the indices. The classical boxplot is used to separate the indices of the outliers from those of the typical observations. Performance evaluation of the proposed methods using simulated data show significant improvements compared to MUOD, both in outlier detection and computational time. We show that Fast-MUOD is especially well suited to handling big and dense functional datasets with very small computational time compared to other methods. Further comparisons with some recent outlier detection methods for functional data also show superior or comparable outlier detection accuracy of the proposed methods. We apply the proposed methods on weather, population growth, and video data.

stat.ME↗

arXiv2021

Towards a mathematical framework to inform Neural Network modelling via Polynomial Regression

Even when neural networks are widely used in a large number of applications, they are still considered as black boxes and present some difficulties for dimensioning or evaluating their prediction error. This has led to an increasing interest in the overlapping area between neural networks and more traditional statistical methods, which can help overcome those problems. In this article, a mathematical framework relating neural networks and polynomial regression is explored by building an explicit expression for the coefficients of a polynomial regression from the weights of a given neural network, using a Taylor expansion approach. This is achieved for single hidden layer neural networks in regression problems. The validity of the proposed method depends on different factors like the distribution of the synaptic potentials or the chosen activation function. The performance of this method is empirically tested via simulation of synthetic data generated from polynomials to train neural networks with different structures and hyperparameters, showing that almost identical predictions can be obtained when certain conditions are met. Lastly, when learning from polynomial generated data, the proposed method produces polynomials that approximate correctly the data locally.

stat.ML↗

arXiv2020

Automatic elimination of the pectoral muscle in mammograms based on anatomical features

Digital mammogram inspection is the most popular technique for early detection of abnormalities in human breast tissue. When mammograms are analyzed through a computational method, the presence of the pectoral muscle might affect the results of breast lesions detection. This problem is particularly evident in the mediolateral oblique view (MLO), where pectoral muscle occupies a large part of the mammography. Therefore, identifying and eliminating the pectoral muscle are essential steps for improving the automatic discrimination of breast tissue. In this paper, we propose an approach based on anatomical features to tackle this problem. Our method consists of two steps: (1) a process to remove the noisy elements such as labels, markers, scratches and wedges, and (2) application of an intensity transformation based on the Beta distribution. The novel methodology is tested with 322 digital mammograms from the Mammographic Image Analysis Society (mini-MIAS) database and with a set of 84 mammograms for which the area normalized error was previously calculated. The results show a very good performance of the method.

eess.IV↗

arXiv2019

Robust regression based on shrinkage estimators

A robust estimator is proposed for the parameters that characterize the linear regression problem. It is based on the notion of shrinkages, often used in Finance and previously studied for outlier detection in multivariate data. A thorough simulation study is conducted to investigate: the efficiency with normal and heavy-tailed errors, the robustness under contamination, the computational times, the affine equivariance and breakdown value of the regression estimator. Two classical data-sets often used in the literature and a real socio-economic data-set about the Living Environment Deprivation of areas in Liverpool (UK), are studied. The results from the simulations and the real data examples show the advantages of the proposed robust estimator in regression.

stat.ME↗

arXiv2019

Multivariate outlier detection based on a robust Mahalanobis distance with shrinkage estimators

A collection of robust Mahalanobis distances for multivariate outlier detection is proposed, based on the notion of shrinkage. Robust intensity and scaling factors are optimally estimated to define the shrinkage. Some properties are investigated, such as affine equivariance and breakdown value. The performance of the proposal is illustrated through the comparison to other techniques from the literature, in a simulation study and with a real dataset. The behavior when the underlying distribution is heavy-tailed or skewed, shows the appropriateness of the method when we deviate from the common assumption of normality. The resulting high correct detection rates and low false detection rates in the vast majority of cases, as well as the significantly smaller computation time shows the advantages of our proposal.

stat.ME↗

arXiv2018

On the estimation of extreme directional multivariate quantiles

In multivariate extreme value theory (MEVT), the focus is on analysis outside of the observable sampling zone, which implies that the region of interest is associated to high risk levels. This work provides tools to include directional notions into the MEVT, giving the opportunity to characterize the recently introduced directional multivariate quantiles (DMQ) at high levels. Then, an out-sample estimation method for these quantiles is given. A bootstrap procedure carries out the estimation of the tuning parameter in this multivariate framework and helps with the estimation of the DMQ. Asymptotic normality for the proposed estimator is provided and the methodology is illustrated with simulated data-sets. Finally, a real-life application to a financial case is also performed.

stat.AP↗

arXiv2018

An empirical comparison of global and local functional depths

A functional data depth provides a center-outward ordering criterion which allows the definition of measures such as median, trimmed means, central regions or ranks in a functional framework. A functional data depth can be global or local. With global depths, the degree of centrality of a curve $x$ depends equally on the rest of the sample observations, while with local depths, the contribution of each observation in defining the degree of centrality of $x$ decreases as the distance from $x$ increases. We empirically compare the global and the local approaches to the functional depth problem focusing on three global and two local functional depths. First, we consider two real data sets and show that global and local depths may provide different insights. Second, we use simulated data to show when we should expect differences between a global and a local approach to the functional depth problem.

stat.ME↗

arXiv2016

Directional Multivariate Extremes in Environmental Phenomena

Several environmental phenomena can be described by different correlated variables that must be considered jointly in order to be more representative of the nature of these phenomena. For such events, identification of extremes is inappropriate if it is based on marginal analysis. Extremes have usually been linked to the notion of quantile, which is an important tool to analyze risk in the univariate setting. We propose to identify multivariate extremes and analyze environmental phenomena in terms of the directional multivariate quantile, which allows us to analyze the data considering all the variables implied in the phenomena, as well as look at the data in interesting directions that can better describe an environmental catastrophe. Since there are many references in the literature that propose extremes detection based on copula models, we also generalize the copula method by introducing the directional approach. Advantages and disadvantages of the non-parametric proposal that we introduce and the copula methods are provided in the paper. We show with simulated and real data sets how by considering the first principal component direction we can improve the visualization of extremes. Finally, two cases of study are analyzed: a synthetic case of flood risk at a dam (a 3-variable case), and a real case study of sea storms (a 5-variable case).

stat.AP↗

arXiv2015

A Directional Multivariate Value at Risk

In economics, insurance and finance, value at risk (VaR) is a widely used measure of the risk of loss on a specific portfolio of financial assets. For a given portfolio, time horizon, and probability $α$, the $100α\%$ VaR is defined as a threshold loss value, such that the probability that the loss on the portfolio over the given time horizon exceeds this value is $α$. That is to say, it is a quantile of the distribution of the losses, which has both good analytic properties and easy interpretation as a risk measure. However, its extension to the multivariate framework is not unique because a unique definition of multivariate quantile does not exist. In the current literature, the multivariate quantiles are related to a specific partial order considered in $\mathbb{R}^{n}$, or to a property of the univariate quantile that is desirable to be extended to $\mathbb{R}^{n}$. In this work, we introduce a multivariate value at risk as a vector-valued directional risk measure, based on a directional multivariate quantile, which has recently been introduced in the literature. The directional approach allows the manager to consider external information or risk preferences in her/his analysis. We have derived some properties of the risk measure and we have compared the univariate \textit{VaR} over the marginals with the components of the directional multivariate VaR. We have also analyzed the relationship between some families of copulas, for which it is possible to obtain closed forms of the multivariate VaR that we propose. Finally, comparisons with other alternative multivariate VaR given in the literature, are provided in terms of robustness.

q-fin.RM↗

arXiv2013

The Mahalanobis distance for functional data with applications to classification

This paper presents a general notion of Mahalanobis distance for functional data that extends the classical multivariate concept to situations where the observed data are points belonging to curves generated by a stochastic process. More precisely, a new semi-distance for functional observations that generalize the usual Mahalanobis distance for multivariate datasets is introduced. For that, the development uses a regularized square root inverse operator in Hilbert spaces. Some of the main characteristics of the functional Mahalanobis semi-distance are shown. Afterwards, new versions of several well known functional classification procedures are developed using the Mahalanobis distance for functional data as a measure of proximity between functional observations. The performance of several well known functional classification procedures are compared with those methods used in conjunction with the Mahalanobis distance for functional data, with positive results, through a Monte Carlo study and the analysis of two real data examples.

math.ST↗

arXiv2010

Bayesian inference for double Pareto lognormal queues

In this article we describe a method for carrying out Bayesian estimation for the double Pareto lognormal (dPlN) distribution which has been proposed as a model for heavy-tailed phenomena. We apply our approach to estimate the $\mathit{dPlN}/M/1$ and $M/\mathit{dPlN}/1$ queueing systems. These systems cannot be analyzed using standard techniques due to the fact that the dPlN distribution does not possess a Laplace transform in closed form. This difficulty is overcome using some recent approximations for the Laplace transform of the interarrival distribution for the $\mathit{Pareto}/M/1$ system. Our procedure is illustrated with applications in internet traffic analysis and risk theory.

stat.AP↗