In-sample calibration yields conformal calibration guarantees

Abstract

Conformal prediction produces a set of predictions that has out-of-sample calibration guarantees by construction, under the assumption of exchangeability. In this work, we study the calibration properties of conformal predictive systems, which issue sets of predictive distributions for real-valued outcomes. We demonstrate that conformal predictive systems implicitly exploit prediction methods that are in-sample calibrated to construct sets that are guaranteed to contain a calibrated predictive distribution out-of-sample. This allows us to take any prediction method that is in-sample calibrated, and conformalize it to obtain a predictive system with out-of-sample calibration guarantees. While the satisfied notion of calibration is typically that prediction intervals derived from the predictive distribution have the correct marginal coverage, we show that this line of reasoning can be extended to stronger conditional notions of calibration that are common in statistical forecasting theory. Using this, we introduce two predictive systems that satisfy stronger out-of-sample calibration guarantees than existing conformal predictive systems. The first method corresponds to a binning of the data, while the second leverages isotonic distributional regression (IDR), a non-parametric distributional regression method under order constraints. We study the theoretical properties of these new predictive systems, and compare their performance in a simulation experiment. They are then applied to two case studies on European temperature forecasts and on predictions for the length of patient stay in Swiss intensive care units. Both approaches are found to outperform existing conformal predictive systems, while conformal IDR additionally provides a natural method for quantifying epistemic uncertainty of the predictions.