On the optimal prediction of extreme events

Abstract

The prediction of the extremely large values of a response variable $Y$ in terms of a vector of covariates $X=(X_i)_{i=1}^d$ is a fundamental problem arising in many scientific and engineering domains. The scarcity of data in the extremes makes the optimal solution of this problem of particular importance. The optimal predictors of such events can be explicitly characterized in just a few cases and it is of fundamental practical and theoretical interest to develop optimal estimators over large classes of models and predictors. In this work, the focus is on the case where $(Y,X)$ have a multivariate regularly varying distribution and one seeks an optimal predictor expressed as a positive homogeneous function $h(X)$ of the covariates. The asymptotic prediction precision in this setting coincides with the tail-dependence coefficient $\lambda(Y,h(X))$ and it can be expressed as an integral functional of the associated angular measure of $(Y,X)$. Thus, finding asymptotically optimal homogeneous predictors amounts to solving a variational problem. We obtain a general solution to this problem, which is expressed in terms of a non-extreme conditional quantile of a tilted distribution derived from the angular measure. This leads to a general inference methodology for the optimal predictors in the peaks-over-threshold framework form extreme value theory. We establish the universal consistency for these estimators over large classes of angular measures. A general-purpose implementation of the resulting inference procedure is shown to work remarkably well against optimal oracle estimators, as well as in the challenging problem of extreme solar flare prediction.