Online selective conformal inference: adaptive scores, convergence rates and optimality

Abstract

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of Gibbs and Cand\`es (2021). For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average miscoverage getting close to a prespecified level $\alpha$ for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. OnlineSCI controls the false coverage proportion among selected times via a pathwise bound for arbitrary sequences, as well as the instantaneous error rate conditional on selection, up to a non-asymptotic remainder term, under stochastic assumptions. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {\it adaptive} capability. We show that the adaptive versions of OnlineSCI can converge to an optimal solution and provide an explicit convergence rate in several application settings, under model- and estimator-specific regularity conditions. The favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.