arXiv subjects

Hervé Cardot

Publications and source records attributed to Hervé Cardot.

At least 19 recordsLinked to original sources

arXiv2026

Statistical description and dimension reduction of continuous time categorical trajectories with multivariate functional principal components

Getting tools that allow simple representations and comparisons of a set of categorical trajectories is of major interest for statisticians. Without loosing any information, we associate to each state a binary random indicator function, taking values in $\{0,1\}$, and turn the problem of statistical description of the categorical trajectories into a multivariate functional principal components analysis. This viewpoint encompasses experimental frameworks where two or more states can be observed simultaneously. The sample paths being piecewise constant, with a finite number of jumps, this a rare case in functional data analysis in which the trajectories are not supposed to be continuous and can be observed exhaustively. Under the weak hypothesis assuming only continuity in probability of the $0-1$ trajectories, the means and the (multivariate) covariance function are continuous and have interpretations in terms of departure from independence of the joint probabilities. Considering a functional data point of view, we show that the binary trajectories, which are right-continuous functions with left-hand limits, can be seen as random elements in the Hilbert space of square integrable functions. The multivariate functional principal components are simple to interpret and we show that we can get consistent estimators of the mean trajectories and the covariance functions under weak regularity assumptions. The ability of the approach to represent categorical trajectories in a small dimension space is illustrated on a data set of sensory perceptions, considering different gustometer-controlled stimuli experiments.

stat.ME↗

arXiv2025

Bootstrap inference for linear regression between variables that are never jointly observed: application in in vivo experiments

In modern experimental science, there is a common problem of estimating the coefficients of a linear regression in a context where the variables of interest cannot be observed simultaneously. When there is a categorical variable that is observed on all statistical units, we consider two estimators of linear regression that take this additional information into account: an estimator based on moments and an estimator based on optimal transport theory. These estimators are shown to be consistent and asymptotically Gaussian under weak hypotheses. The asymptotic variance has no explicit expression, except in some special cases, for which reason a stratified bootstrap approach is developed to construct confidence intervals for the estimated parameters, whose consistency is also shown. A simulation study evaluating and comparing the finite sample performance of these estimators demonstrates the advantages of the bootstrap approach in several realistic scenarios. An application to in vivo experiments, conducted in the context of studying radio-induced adverse effects in mice, revealed important relationships between the biomarkers of interest that could not be identified with the considered naive approach.

stat.ME↗

arXiv2025

Joint clustering with alignment for temporal data in a one-point-per-experiment setting

Temporal data, obtained in the setting where it is only possible to observe one time point per experiment, is widely used in different research fields, yet remains insufficiently addressed from the statistical point of view. Such data often contain observations of a large number of entities, in which case it is of interest to identify a small number of representative behavior types. In this paper, we propose a new method that simultaneously performs clustering and alignment of temporal objects inferred from these data, providing insight into the relationships between entities. Simulations confirm the ability of the proposed approach to leverage multiple properties of the complex data we target such as accessible uncertainties, correlations and a small number of time points. We illustrate it on real data encoding cellular response to a radiation treatment with high energy, supported with the results of an enrichment analysis.

stat.ME↗

arXiv2019

Estimating finite mixtures of semi-Markov chains: an application to the segmentation of temporal sensory data

In food science, it is of great interest to get information about the temporal perception of aliments to create new products, to modify existing ones or more generally to understand the perception mechanisms. Temporal Dominance of Sensations (TDS) is a technique to measure temporal perception which consists in choosing sequentially attributes describing a food product over tasting. This work introduces new statistical models based on finite mixtures of semi-Markov chains in order to describe data collected with the TDS protocol, allowing different temporal perceptions for a same product within a population. The identifiability of the parameters of such mixture models is discussed. Sojourn time distributions are fitted with gamma probability distribution and a penalty is added to the log likelihood to ensure convergence of the EM algorithm to a non degenerate solution. Information criterions are employed for determining the number of mixture components. Then, the individual qualitative trajectories are clustered with the help of the maximum a posteriori probability (MAP) approach. A simulation study confirms the good behavior of the proposed estimation procedure. The methodology is illustrated on an example of consumers perception of a Gouda cheese and assesses the existence of several behaviors in terms of perception of this product.

stat.ME↗

arXiv2018

Conditional bias robust estimation of the total of curve data by sampling in a finite population: an illustration on electricity load curves

For marketing or power grid management purposes, many studies based on the analysis of the total electricity consumption curves of groups of customers are now carried out by electricity companies. Aggregated total or mean load curves are estimated using individual curves measured at fine time grid and collected according to some sampling design. Due to the skewness of the distribution of electricity consumptions, these samples often contain outlying curves which may have an important impact on the usual estimation procedures. We introduce several robust estimators of the total consumption curve which are not sensitive to such outlying curves. These estimators are based on the conditional bias approach and robust functional methods. We also derive mean square error estimators of these robust estimators and finally, we evaluate and compare the performance of the suggested estimators on Irish electricity data.

stat.ME↗

arXiv2016

Fast Estimation of the Median Covariation Matrix with Application to Online Robust Principal Components Analysis

The geometric median covariation matrix is a robust multivariate indicator of dispersion which can be extended without any difficulty to functional data. We define estimators, based on recursive algorithms, that can be simply updated at each new observation and are able to deal rapidly with large samples of high dimensional data without being obliged to store all the data in memory. Asymptotic convergence properties of the recursive algorithms are studied under weak conditions. The computation of the principal components can also be performed online and this approach can be useful for online outlier detection. A simulation study clearly shows that this robust indicator is a competitive alternative to minimum covariance determinant when the dimension of the data is small and robust principal components analysis based on projection pursuit and spherical projections for high dimension data. An illustration on a large sample and high dimensional dataset consisting of individual TV audiences measured at a minute scale over a period of 24 hours confirms the interest of considering the robust principal components analysis based on the median covariation matrix. All studied algorithms are available in the R package Gmedian on CRAN.

math.ST↗

arXiv2015

Online Principal Component Analysis in High Dimension: Which Algorithm to Choose?

In the current context of data explosion, online techniques that do not require storing all data in memory are indispensable to routinely perform tasks like principal component analysis (PCA). Recursive algorithms that update the PCA with each new observation have been studied in various fields of research and found wide applications in industrial monitoring, computer vision, astronomy, and latent semantic indexing, among others. This work provides guidance for selecting an online PCA algorithm in practice. We present the main approaches to online PCA, namely, perturbation techniques, incremental methods, and stochastic optimization, and compare their statistical accuracy, computation time, and memory requirements using artificial and real data. Extensions to missing data and to functional data are discussed. All studied algorithms are available in the R package onlinePCA on CRAN.

stat.ML↗

arXiv2015

Online estimation of the geometric median in Hilbert spaces : non asymptotic confidence balls

Estimation procedures based on recursive algorithms are interesting and powerful techniques that are able to deal rapidly with (very) large samples of high dimensional data. The collected data may be contaminated by noise so that robust location indicators, such as the geometric median, may be preferred to the mean. In this context, an estimator of the geometric median based on a fast and efficient averaged non linear stochastic gradient algorithm has been developed by Cardot, Cénac and Zitt (2013). This work aims at studying more precisely the non asymptotic behavior of this algorithm by giving non asymptotic confidence balls. This new result is based on the derivation of improved $L^2$ rates of convergence as well as an exponential inequality for the martingale terms of the recursive non linear Robbins-Monro algorithm.

math.ST↗

arXiv2015

Estimating with kernel smoothers the mean of functional data in a finite population setting. A note on variance estimation in presence of partially observed trajectories

In the near future, millions of load curves measuring the electricity consumption of French households in small time grids (probably half hours) will be available. All these collected load curves represent a huge amount of information which could be exploited using survey sampling techniques. In particular, the total consumption of a specific cus- tomer group (for example all the customers of an electricity supplier) could be estimated using unequal probability random sampling methods. Unfortunately, data collection may undergo technical problems resulting in missing values. In this paper we study a new estimation method for the mean curve in the presence of missing values which consists in extending kernel estimation techniques developed for longitudinal data analysis to sampled curves. Three nonparametric estimators that take account of the missing pieces of trajectories are suggested. We also study pointwise variance estimators which are based on linearization techniques. The particular but very important case of stratified sampling is then specifically studied. Finally, we discuss some more practical aspects such as choosing the bandwidth values for the kernel and estimating the probabilities of observation of the trajectories.

stat.ME↗

arXiv2013

Confidence bands for Horvitz-Thompson estimators using sampled noisy functional data

When collections of functional data are too large to be exhaustively observed, survey sampling techniques provide an effective way to estimate global quantities such as the population mean function. Assuming functional data are collected from a finite population according to a probabilistic sampling scheme, with the measurements being discrete in time and noisy, we propose to first smooth the sampled trajectories with local polynomials and then estimate the mean function with a Horvitz-Thompson estimator. Under mild conditions on the population size, observation times, regularity of the trajectories, sampling scheme, and smoothing bandwidth, we prove a Central Limit theorem in the space of continuous functions. We also establish the uniform consistency of a covariance function estimator and apply the former results to build confidence bands for the mean function. The bands attain nominal coverage and are obtained through Gaussian process simulations conditional on the estimated covariance function. To select the bandwidth, we propose a cross-validation method that accounts for the sampling weights. A simulation study assesses the performance of our approach and highlights the influence of the sampling scheme and bandwidth choice.

math.ST↗

arXiv2013

Variance estimation and asymptotic confidence bands for the mean estimator of sampled functional data with high entropy unequal probability sampling designs

For fixed size sampling designs with high entropy it is well known that the variance of the Horvitz-Thompson estimator can be approximated by the Hájek formula. The interest of this asymptotic variance approximation is that it only involves the first order inclusion probabilities of the statistical units. We extend this variance formula when the variable under study is functional and we prove, under general conditions on the regularity of the individual trajectories and the sampling design, that we can get a uniformly convergent estimator of the variance function of the Horvitz-Thompson estimator of the mean function. Rates of convergence to the true variance function are given for the rejective sampling. We deduce, under conditions on the entropy of the sampling design, that it is possible to build confidence bands whose coverage is asymptotically the desired one via simulation of Gaussian processes with variance function given by the Hájek formula. Finally, the accuracy of the proposed variance estimator is evaluated on samples of electricity consumption data measured every half an hour over a period of one week.

math.ST↗

arXiv2013

Uniform convergence and asymptotic confidence bands for model-assisted estimators of the mean of sampled functional data

When the study variable is functional and storage capacities are limited or transmission costs are high, selecting with survey sampling techniques a small fraction of the observations is an interesting alternative to signal compression techniques, particularly when the goal is the estimation of simple quantities such as means or totals. We extend, in this functional framework, model-assisted estimators with linear regression models that can take account of auxiliary variables whose totals over the population are known. We first show, under weak hypotheses on the sampling design and the regularity of the trajectories, that the estimator of the mean function as well as its variance estimator are uniformly consistent. Then, under additional assumptions, we prove a functional central limit theorem and we assess rigorously a fast technique based on simulations of Gaussian processes which is employed to build asymptotic confidence bands. The accuracy of the variance function estimator is evaluated on a real dataset of sampled electricity consumption curves measured every half an hour over a period of one week.

math.ST↗

arXiv2012

Recursive estimation of the conditional geometric median in Hilbert spaces

A recursive estimator of the conditional geometric median in Hilbert spaces is studied. It is based on a stochastic gradient algorithm whose aim is to minimize a weighted L1 criterion and is consequently well adapted for robust online estimation. The weights are controlled by a kernel function and an associated bandwidth. Almost sure convergence and L2 rates of convergence are proved under general conditions on the conditional distribution as well as the sequence of descent steps of the algorithm and the sequence of bandwidths. Asymptotic normality is also proved for the averaged version of the algorithm with an optimal rate of convergence. A simulation study confirms the interest of this new and fast algorithm when the sample sizes are large. Finally, the ability of these recursive algorithms to deal with very high-dimensional data is illustrated on the robust estimation of television audience profiles conditional on the total time spent watching television over a period of 24 hours.

math.ST↗

arXiv2011

A fast and recursive algorithm for clustering large datasets with $k$-medians

Clustering with fast algorithms large samples of high dimensional data is an important challenge in computational statistics. Borrowing ideas from MacQueen (1967) who introduced a sequential version of the $k$-means algorithm, a new class of recursive stochastic gradient algorithms designed for the $k$-medians loss criterion is proposed. By their recursive nature, these algorithms are very fast and are well adapted to deal with large samples of data that are allowed to arrive sequentially. It is proved that the stochastic gradient algorithm converges almost surely to the set of stationary points of the underlying loss criterion. A particular attention is paid to the averaged versions, which are known to have better performances, and a data-driven procedure that allows automatic selection of the value of the descent step is proposed. The performance of the averaged sequential estimator is compared on a simulation study, both in terms of computation speed and accuracy of the estimations, with more classical partitioning techniques such as $k$-means, trimmed $k$-means and PAM (partitioning around medoids). Finally, this new online clustering technique is illustrated on determining television audience profiles with a sample of more than 5000 individual television audiences measured every minute over a period of 24 hours.

stat.CO↗

arXiv2011

Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm

With the progress of measurement apparatus and the development of automatic sensors it is not unusual anymore to get thousands of samples of observations taking values in high dimension spaces such as functional spaces. In such large samples of high dimensional data, outlying curves may not be uncommon and even a few individuals may corrupt simple statistical indicators such as the mean trajectory. We focus here on the estimation of the geometric median which is a direct generalization of the real median and has nice robustness properties. The geometric median being defined as the minimizer of a simple convex functional that is differentiable everywhere when the distribution has no atoms, it is possible to estimate it with online gradient algorithms. Such algorithms are very fast and can deal with large samples. Furthermore they also can be simply updated when the data arrive sequentially. We state the almost sure consistency and the L2 rates of convergence of the stochastic gradient estimator as well as the asymptotic normality of its averaged version. We get that the asymptotic distribution of the averaged version of the algorithm is the same as the classic estimators which are based on the minimization of the empirical loss function. The performances of our averaged sequential estimator, both in terms of computation speed and accuracy of the estimations, are evaluated with a small simulation study. Our approach is also illustrated on a sample of more 5000 individual television audiences measured every second over a period of 24 hours.

math.ST↗

arXiv2010

Horvitz-Thompson estimators for functional data: asymptotic confidence bands and optimal allocation for stratified sampling

When dealing with very large datasets of functional data, survey sampling approaches are useful in order to obtain estimators of simple functional quantities, without being obliged to store all the data. We propose here a Horvitz--Thompson estimator of the mean trajectory. In the context of a superpopulation framework, we prove under mild regularity conditions that we obtain uniformly consistent estimators of the mean function and of its variance function. With additional assumptions on the sampling design we state a functional Central Limit Theorem and deduce asymptotic confidence bands. Stratified sampling is studied in detail, and we also obtain a functional version of the usual optimal allocation rule considering a mean variance criterion. These techniques are illustrated by means of a test population of N=18902 electricity meters for which we have individual electricity consumption measures every 30 minutes over one week. We show that stratification can substantially improve both the accuracy of the estimators and reduce the width of the global confidence bands compared to simple random sampling without replacement.

stat.ME↗

arXiv2009

Properties of Design-Based Functional Principal Components Analysis

This work aims at performing Functional Principal Components Analysis (FPCA) with Horvitz-Thompson estimators when the observations are curves collected with survey sampling techniques. One important motivation for this study is that FPCA is a dimension reduction tool which is the first step to develop model assisted approaches that can take auxiliary information into account. FPCA relies on the estimation of the eigenelements of the covariance operator which can be seen as nonlinear functionals. Adapting to our functional context the linearization technique based on the influence function developed by Deville (1999), we prove that these estimators are asymptotically design unbiased and consistent. Under mild assumptions, asymptotic variances are derived for the FPCA' estimators and consistent estimators of them are proposed. Our approach is illustrated with a simulation study and we check the good properties of the proposed estimators of the eigenelements as well as their variance estimators obtained with the linearization approach.

math.ST↗

arXiv2007

Quantile regression when the covariates are functions

This paper deals with a linear model of regression on quantiles when the explanatory variable takes values in some functional space and the response is scalar. We propose a spline estimator of the functional coefficient that minimizes a penalized L1 type criterion. Then, we study the asymptotic behavior of this estimator. The penalization is of primary importance to get existence and convergence.

math.ST↗