arXiv subjects

Olivier Wintenberger

Publications and source records attributed to Olivier Wintenberger.

At least 37 records · Page 2Linked to original sources

arXiv2025

Minimax-optimal and Locally-adaptive Online Nonparametric Regression

We study adversarial online nonparametric regression with general convex losses and propose a parameter-free learning algorithm that achieves minimax optimal rates. Our approach leverages chaining trees to compete against H{ö}lder functions and establishes optimal regret bounds. While competing with nonparametric function classes can be challenging, they often exhibit local patterns - such as local H{ö}lder continuity - that online algorithms can exploit. Without prior knowledge, our method dynamically tracks and adapts to different H{ö}lder profiles by pruning a core chaining tree structure, aligning itself with local smoothness variations. This leads to the first computationally efficient algorithm with locally adaptive optimal rates for online regression in an adversarial setting. Finally, we discuss how these notions could be extended to a boosting framework, offering promising directions for future research.

math.ST↗

arXiv2024

Semi-Discrete Optimal Transport: Nearly Minimax Estimation With Stochastic Gradient Descent and Adaptive Entropic Regularization

Optimal Transport (OT) based distances are powerful tools for machine learning to compare probability measures and manipulate them using OT maps. In this field, a setting of interest is semi-discrete OT, where the source measure $μ$ is continuous, while the target $ν$ is discrete. Recent works have shown that the minimax rate for the OT map is $\mathcal{O}(t^{-1/2})$ when using $t$ i.i.d. subsamples from each measure (two-sample setting). An open question is whether a better convergence rate can be achieved when the full information of the discrete measure $ν$ is known (one-sample setting). In this work, we answer positively to this question by (i) proving an $\mathcal{O}(t^{-1})$ lower bound rate for the OT map, using the similarity between Laguerre cells estimation and density support estimation, and (ii) proposing a Stochastic Gradient Descent (SGD) algorithm with adaptive entropic regularization and averaging acceleration. To nearly achieve the desired fast rate, characteristic of non-regular parametric problems, we design an entropic regularization scheme decreasing with the number of samples. Another key step in our algorithm consists of using a projection step that permits to leverage the local strong convexity of the regularized OT problem. Our convergence analysis integrates online convex optimization and stochastic gradient techniques, complemented by the specificities of the OT semi-dual. Moreover, while being as computationally and memory efficient as vanilla SGD, our algorithm achieves the unusual fast rates of our theory in numerical experiments.

stat.ML↗

arXiv2024

Estimation of the Spectral Measure from ConvexCombinations of Regularly Varying RandomVectors

The extremal dependence structure of a regularly varying random vector Xis fully described by its limiting spectral measure. In this paper, we investigate how torecover characteristics of the measure, such as extremal coefficients, from the extremalbehaviour of convex combinations of components of X. Our considerations result in aclass of new estimators of moments of the corresponding combinations for the spectralvector. We show asymptotic normality by means of a functional limit theorem and, focusingon the estimation of extremal coefficients, we verify that the minimal asymptoticvariance can be achieved by a plug-in estimator using subsampling bootstrap. We illustratethe benefits of our approach on simulated and real data.

math.ST↗

arXiv2024

Online Learning Approach for Survival Analysis

We introduce an online mathematical framework for survival analysis, allowing real time adaptation to dynamic environments and censored data. This framework enables the estimation of event time distributions through an optimal second order online convex optimization algorithm-Online Newton Step (ONS). This approach, previously unexplored, presents substantial advantages, including explicit algorithms with non-asymptotic convergence guarantees. Moreover, we analyze the selection of ONS hyperparameters, which depends on the exp-concavity property and has a significant influence on the regret bound. We propose a stochastic approach that guarantees logarithmic stochastic regret for ONS. Additionally, we introduce an adaptive aggregation method that ensures robustness in hyperparameter selection while maintaining fast regret bounds. The findings of this paper can extend beyond the survival analysis field, and are relevant for any case characterized by poor exp-concavity and unstable ONS. Finally, these assertions are illustrated by simulation experiments.

cs.LG↗

arXiv2023

Moment conditions for random coefficient AR($\infty$) under non-negativity assumptions

We consider random coefficient autoregressive models of infinite order (AR($\infty$)) under the assumption of non-negativity of the coefficients. We develop novel methods yielding sufficient or necessary conditions for finiteness of moments, based on combinatorial expressions of first and second moments. The methods based on first moments recover previous sufficient conditions by Doukhan and Wintenberger in our setting. The second moment method provides in particular a necessary and sufficient condition for finiteness of second moments which is different, but shown to be equivalent to the classical criterion of Nicholls and Quinn in the case of finite order. We further illustrate our results through two examples.

math.PR↗

arXiv2023

Learning from time-dependent streaming data with online stochastic algorithms

This paper addresses stochastic optimization in a streaming setting with time-dependent and biased gradient estimates. We analyze several first-order methods, including Stochastic Gradient Descent (SGD), mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages. Our non-asymptotic analysis establishes novel heuristics that link dependence, biases, and convexity levels, enabling accelerated convergence. Specifically, our findings demonstrate that (i) time-varying mini-batch SGD methods have the capability to break long- and short-range dependence structures, (ii) biased SGD methods can achieve comparable performance to their unbiased counterparts, and (iii) incorporating Polyak-Ruppert averaging can accelerate the convergence of the stochastic optimization algorithms. To validate our theoretical findings, we conduct a series of experiments using both simulated and real-life time-dependent data.

cs.LG↗

arXiv2023

Adaptive Probabilistic Forecasting of Electricity (Net-)Load

Electricity load forecasting is a necessary capability for power system operators and electricity market participants. The proliferation of local generation, demand response, and electrification of heat and transport are changing the fundamental drivers of electricity load and increasing the complexity of load modelling and forecasting. We address this challenge in two ways. First, our setting is adaptive; our models take into account the most recent observations available, yielding a forecasting strategy able to automatically respond to changes in the underlying process. Second, we consider probabilistic rather than point forecasting; indeed, uncertainty quantification is required to operate electricity systems efficiently and reliably. Our methodology relies on the Kalman filter, previously used successfully for adaptive point load forecasting. The probabilistic forecasts are obtained by quantile regressions on the residuals of the point forecasting model. We achieve adaptive quantile regressions using the online gradient descent; we avoid the choice of the gradient step size considering multiple learning rates and aggregation of experts. We apply the method to two data sets: the regional net-load in Great Britain and the demand of seven large cities in the United States. Adaptive procedures improve forecast performance substantially in both use cases for both point and probabilistic forecasting.

stat.AP↗

arXiv2023

Non-Asymptotic Analysis of Stochastic Approximation Algorithms for Streaming Data

We introduce a streaming framework for analyzing stochastic approximation/optimization problems. This streaming framework is analogous to solving optimization problems using time-varying mini-batches that arrive sequentially. We provide non-asymptotic convergence rates of various gradient-based algorithms; this includes the famous Stochastic Gradient (SG) descent (a.k.a. Robbins-Monro algorithm), mini-batch SG and time-varying mini-batch SG algorithms, as well as their iterated averages (a.k.a. Polyak-Ruppert averaging). We show i) how to accelerate convergence by choosing the learning rate according to the time-varying mini-batches, ii) that Polyak-Ruppert averaging achieves optimal convergence in terms of attaining the Cramer-Rao lower bound, and iii) how time-varying mini-batches together with Polyak-Ruppert averaging can provide variance reduction and accelerate convergence simultaneously, which is advantageous for many learning problems, such as online, sequential, and large-scale learning. We further demonstrate these favorable effects for various time-varying mini-batches.

cs.LG↗

arXiv2023

Stochastic Online Convex Optimization. Application to probabilistic time series forecasting

We introduce a general framework of stochastic online convex optimization to obtain fast-rate stochastic regret bounds. We prove that algorithms such as online newton steps and a scale-free 10 version of Bernstein online aggregation achieve best-known rates in unbounded stochastic settings. We apply our approach to calibrate parametric probabilistic forecasters of non-stationary sub-gaussian time series. Our fast-rate stochastic regret bounds are any-time valid. Our proofs combine self-bounded and Poissonnian inequalities for martingales and sub-gaussian random variables, respectively, under a stochastic exp-concavity assumption.

cs.LG↗

arXiv2023

Self-normalized partial sums of heavy-tailed time series

We study the joint limit behavior of sums, maxima and $\ell^p$-type moduli for samples taken from an $\mathbb{R}^d$-valued regularly varying stationary sequence with infinite variance. As a consequence, we can determine the distributional limits for ratios of sums and maxima, studentized sums, and other self-normalized quantities in terms of hybrid characteristic functions and Laplace transforms. These transforms enable one to calculate moments of the limits and to characterize the differences between the iid and stationary cases in terms of indices which describe effects of extremal clustering on functionals acting on the dependent sequence.

math.PR↗

arXiv2023

Multivariate sparse clustering for extremes

Identifying directions where extreme events occur is a major challenge in multivariate extreme value analysis. In this paper, we use the concept of sparse regular variation introduced by Meyer and Wintenberger (2021)} to infer the tail dependence of a random vector X. This approach relies on the Euclidean projection onto the simplex which better exhibits the sparsity structure of the tail of X than the standard methods. Our procedure based on a rigorous methodology aims at capturing clusters of extremal coordinates of X. It also includes the identification of the threshold above which the values taken by X are considered as extreme. We provide an efficient and scalable algorithm called MUSCLE and apply it on numerical examples to highlight the relevance of our findings. Finally we illustrate our approach with financial return data.

math.ST↗

arXiv2022

Large deviations of lp-blocks of regularly varying time series and applications to cluster inference

In the regularly varying time series setting, a cluster of exceedances is a short period for which the supremum norm exceeds a high threshold. We propose to study a generalization of this notion considering short periods, or blocks, with lp-norm above a high threshold. Our main result derives new large deviation principles of extremal lp-blocks, which guide us to define and characterize spectral cluster processes in lp. We then study cluster inference in lp to motivate our results. We design consistent disjoint block estimators to infer features of cluster processes. Our estimators promote the use of large empirical quantiles from the lp-norm of blocks as threshold levels which eases implementation and also facilitates comparison for different p>0. Our approach highlights the advantages of cluster inference based on extremal l$α$--blocks, where $α$>0 is the index of regular variation of the series. We focus on inferring important indices in extreme value theory, e.g., the extremal index.

math.PR↗

arXiv2022

Extremes for stationary regularly varying random fields over arbitrary index sets

We consider the clustering of extremes for stationary regularly varying random fields over arbitrary growing index sets. We study sufficient assumptions on the index set such that the limit of the point random fields of the exceedances above a high threshold exists. Under the so-called anti-clustering condition, the extremal dependence is only local. Thus the index set can have a general form compared to previous literature [3, 21]. However, we cannot describe the clustering of extreme values in terms of the usual spectral tail measure [23] except for hyperrectangles or index sets in the lattice case. Using the recent extension of the spectral measure for star-shaped equipped space [18], the $\upsilon$-spectral tail measure provides a natural extension that describes the clustering effect in full generality.

math.PR↗

arXiv2021

Viking: Variational Bayesian Variance Tracking

We consider the problem of time series forecasting in an adaptive setting. We focus on the inference of state-space models under unknown and potentially time-varying noise variances. We introduce an augmented model in which the variances are represented as auxiliary gaussian latent variables in a tracking mode. As variances are nonnegative, a transformation is chosen and applied to these latent variables. The inference relies on the online variational Bayesian methodology, which consists in minimizing a Kullback-Leibler divergence at each time step. We observe that the minimum of the Kullback-Leibler divergence is an extension of the Kalman filter taking into account the variance uncertainty. We design a novel algorithm, named Viking, using these optimal recursive updates. For auxiliary latent variables, we use second-order bounds whose optimum admit closed-form solutions. Experiments on synthetic data show that Viking behaves well and is robust to misspecification.

cs.LG↗

arXiv2021

Some variations on the extremal index

We re-consider Leadbetter's extremal index for stationary sequences. It has interpretation as reciprocal of the expected size of an extremal cluster above high thresholds. We focus on heavy-tailed time series, in particular on regularly varying stationary sequences, and discuss recent research in extreme value theory for these models. A regularly varying time series has multivariate regularly varying finite-dimensional distributions. Thanks to results by Basrak and Segers we have explicit representations of the limiting cluster structure of extremes, leading to explicit expressions of the limiting point process of exceedances and the extremal index as a summary measure of extremal clustering. The extremal index appears in various situations which do not seem to be directly related, like the convergence of maxima and point processes. We consider different representations of the extremal index which arise from the considered context. We discuss the theory and apply it to a regularly varying AR(1) process and the solution to an affine stochastic recurrence equation

math.PR↗

arXiv2021

Asymptotic Independence ex machina -- Extreme Value Theory for the Diagonal BEKK-ARCH(1) Model

We consider multivariate stationary processes $(\boldsymbol{X}_t)$ satisfying a stochastic recurrence equation of the form $$ \boldsymbol{X}_t= \mathbb{ M}_t \boldsymbol{X}_{t-1} + \boldsymbol{Q}_t,$$ where $(\boldsymbol{Q}_t)$ are iid random vectors and $$ \mathbb{M}_t=\mathrm{Diag}(b_1+c_1 M_t, \dots, b_d+c_d M_t) $$ are iid diagonal matrices and $(M_t)$ are iid random variables. We obtain a full characterization of the multivariate regular variation properties of $(\boldsymbol{X}_t)$, proving that coordinates $X_{t,i}$ and $X_{t,j}$ are asymptotically independent even though all coordinates rely on the same random input $(M_t)$. We describe extremal properties of $(\boldsymbol{X}_t)$ in the framework of vector scaling regular variation. Our results are applied to some multivariate autoregressive conditional heteroskedasticity (BEKK-ARCH and CCC-GARCH) processes.

math.PR↗

arXiv2021

Sparse regular variation

Regular variation provides a convenient theoretical framework to study large events. In the multivariate setting, the dependence structure of the positive extremes is characterized by a measure - the spectral measure - defined on the positive orthant of the unit sphere. This measure gathers information on the localization of extreme events and has often a sparse support since severe events do not simultaneously occur in all directions. However, it is defined through weak convergence which does not provide a natural way to capture this sparsity structure.In this paper, we introduce the notion of sparse regular variation which allows to better learn the dependence structure of extreme events. This concept is based on the Euclidean projection onto the simplex for which efficient algorithms are known. We prove that under mild assumptions sparse regular variation and regular variation are two equivalent notions and we establish several results for sparsely regularly varying random vectors. Finally, we illustrate on numerical examples how this new concept allows one to detect extremal directions.

stat.ML↗

arXiv2021

Hidden regular variation for point processes and the single/multiple large point heuristic

We consider regular variation for marked point processes with independent heavy-tailed marks and prove a single large point heuristic: the limit measure is concentrated on the cone of point measures with one single point. We then investigate successive hidden regular variation removing the cone of point measures with at most $k$ points, $k\geq 1$, and prove a multiple large point phenomenon: the limit measure is concentrated on the cone of point measures with $k+1$ points. We show how these results imply hidden regular variation in Skorokhod space of the associated risk process. Finally, we provide an application to risk theory in a reinsurance model where the $k$ largest claims are covered and we study the asymptotic behavior of the residual risk.

math.PR↗