arXiv subjects

Matteo Sesia

Publications and source records attributed to Matteo Sesia.

At least 19 recordsLinked to original sources

arXiv2026

Conformal Prediction Regions for Continuous-Time Trajectories under Random Sampling

Uncertainty quantification for continuous-time trajectories is a prerequisite in many safety-critical engineering domains. However, a major challenge in data-driven uncertainty quantification is that calibration trajectories are sampled only at discrete, often sparse, and random intervals. Standard conformal prediction methods typically fail to provide guarantees in between sampling times. In this work, we introduce a new technique to obtain valid conformal prediction regions for continuous-time trajectories that are sampled at discrete and possibly random times. To accomplish this goal, we make three contributions: (1) we provide an algorithm that leverages regularity properties of the underlying trajectories to obtain valid prediction regions in between samples, (2) we provide methods that estimate valid bounds on the aforementioned regularity properties from an additional high-frequency calibration dataset, and (3) we introduce and compare several algorithms to deal with random sampling times. Finally, we present experiments where we validate that our methods achieve valid coverage across the entire continuous trajectory.

eess.SY↗

arXiv2026

Conformal Prediction for Regression with Clipped Outcomes

We study conformal prediction for regression using calibration data with outcomes that are doubly censored (clipped) at known fixed thresholds. We show that existing methods are unsatisfactory in this setting, as they yield intervals that may have higher marginal coverage than desired and yet lose conditional coverage precisely for the easier-to-predict cases whose outcomes are typically fully observed. This reveals that marginal coverage, the usual target of conformal prediction, may not be the ideal goal under clipping. We address this challenge by introducing a new nonconformity score and calibration methods at both ends of this trade-off: one for tight marginal coverage, and a two-step method that prioritizes conditional coverage. We characterize their finite-sample coverage and oracle-like asymptotic behavior under suitable consistency of the underlying model, and we compare them to more direct adaptations of existing approaches.

stat.ME↗

arXiv2026

Audited Conformal Prediction for Classification under Unknown Distribution Shift

We consider the problem of uncertainty quantification for a pretrained classification model deployed under unknown distribution shift. We propose Audited Conformal Prediction (ACP), a method that leverages a small labeled dataset from the target population to train an auxiliary audit model identifying inputs where the legacy model is likely to fail. By integrating the audit model's outputs into the conformal prediction framework, ACP produces prediction sets that guarantee marginal coverage while achieving substantially higher conditional coverage in practice than existing approaches. We develop and analyze two complementary integration strategies -- one targeting marginal coverage with improved conditional performance, the other providing explicit group-conditional coverage guarantees -- and establish theoretical guarantees for both. Experiments on synthetic and real-world datasets validate the method and illustrate trade-offs between prediction set size and conditional coverage.

stat.ML↗

arXiv2026

Elements of Conformal Prediction

Predictive inference is a fundamental task in statistics, traditionally addressed using parametric assumptions about the data distribution and detailed analyses of how models learn from data. In recent years, conformal prediction has emerged as an alternative framework that is well suited to modern applications involving high-dimensional data and complex machine learning models. Its appeal stems from being both distribution-free---relying mainly on symmetry assumptions such as exchangeability---and model-agnostic, treating the learning algorithm as a black box. Even under such limited assumptions, conformal prediction provides exact finite-sample guarantees, although these are typically marginal and require careful interpretation. This paper explains the core ideas of conformal prediction and reviews selected methods. Rather than offering an exhaustive survey, it aims to provide a clear conceptual entry point and a pedagogical overview of the field.

stat.ME↗

arXiv2025

Distribution-Free Selection of Low-Risk Oncology Patients for Survival Beyond a Time Horizon

We study how to select a subset of patients who are unlikely to experience an adverse event within a given time horizon, by calibrating a screening rule based on the output of any survival model. We consider two complementary frameworks. The first extends the classical idea of estimating the event rate among selected patients using a hold-out dataset, integrating it with the Learn-Then-Test method. This provides approximate high-probability guarantees that are comparable to those obtainable from simultaneous confidence bands while being often less conservative. The second takes a different perspective by reformulating the problem in terms of multiple hypotheses testing, enabling false discovery rate (FDR) control via the Benjamini-Hochberg procedure applied to selective conformal p-values. This provides approximate guarantees in expectation. We clarify the theoretical relationship between these approaches, explain how both can handle right-censored data and be made doubly robust via augmented inverse probability of censoring weighting, and compare them empirically using simulations and oncology data from the Flatiron Health Research Database. Our results reveal a practical trade-off: FDR-based screening is typically more powerful, while high-probability calibration is more conservative but offers stronger guarantees, especially when few patients are selected. We also provide practical guidance on implementation and tuning.

stat.AP↗

arXiv2025

Interpretable Multivariate Conformal Prediction with Fast Transductive Standardization

We propose a conformal prediction method for constructing tight simultaneous prediction intervals for multiple, potentially related, numerical outputs given a single input. This method can be combined with any multi-target regression model and guarantees finite-sample coverage. It is computationally efficient and yields informative prediction intervals even with limited data. The core idea is a novel \emph{coordinate-wise} standardization procedure that makes residuals across output dimensions directly comparable, estimating suitable scaling parameters using the calibration data themselves. This does not require modeling of cross-output dependence nor auxiliary sample splitting. Implementing this idea requires overcoming technical challenges associated with transductive or full conformal prediction. Experiments on simulated and real data demonstrate this method can produce tighter prediction intervals than existing baselines while maintaining valid simultaneous coverage.

stat.ME↗

arXiv2025

Conformal Inference for Open-Set and Imbalanced Classification

This paper presents a conformal prediction method for classification in highly imbalanced and open-set settings, where there are many possible classes and not all may be represented in the data. Existing approaches require a finite, known label space and typically involve random sample splitting, which works well when there is a sufficient number of observations from each class. Consequently, they have two limitations: (i) they fail to provide adequate coverage when encountering new labels at test time, and (ii) they may become overly conservative when predicting previously seen labels. To obtain valid prediction sets in the presence of unseen labels, we compute and integrate into our predictions a new family of conformal p-values that can test whether a new data point belongs to a previously unseen class. We study these p-values theoretically, establishing their optimality, and uncover an intriguing connection with the classical Good--Turing estimator for the probability of observing a new species. To make more efficient use of imbalanced data, we also develop a selective sample splitting algorithm that partitions training and calibration data based on label frequency, leading to more informative predictions. Despite breaking exchangeability, this allows maintaining finite-sample guarantees through suitable re-weighting. With both simulated and real data, we demonstrate our method leads to prediction sets with valid coverage even in challenging open-set scenarios with infinite numbers of possible labels, and produces more informative predictions under extreme class imbalance.

stat.ML↗

arXiv2025

Conformal Survival Bands for Risk Screening under Right-Censoring

We propose a method to quantify uncertainty around individual survival distribution estimates using right-censored data, compatible with any survival model. Unlike classical confidence intervals, the survival bands produced by this method offer predictive rather than population-level inference, making them useful for personalized risk screening. For example, in a low-risk screening scenario, they can be applied to flag patients whose survival band at 12 months lies entirely above 50\%, while ensuring that at least half of flagged individuals will survive past that time on average. Our approach builds on recent advances in conformal inference and integrates ideas from inverse probability of censoring weighting and multiple testing with false discovery rate control. We provide asymptotic guarantees and show promising performance in finite samples with both simulated and real data.

stat.ME↗

arXiv2025

Robust Conformal Outlier Detection under Contaminated Reference Data

Conformal prediction is a flexible framework for calibrating machine learning predictions, providing distribution-free statistical guarantees. In outlier detection, this calibration relies on a reference set of labeled inlier data to control the type-I error rate. However, obtaining a perfectly labeled inlier reference set is often unrealistic, and a more practical scenario involves access to a contaminated reference set containing a small fraction of outliers. This paper analyzes the impact of such contamination on the validity of conformal methods. We prove that under realistic, non-adversarial settings, calibration on contaminated data yields conservative type-I error control, shedding light on the inherent robustness of conformal methods. This conservativeness, however, typically results in a loss of power. To alleviate this limitation, we propose a novel, active data-cleaning framework that leverages a limited labeling budget and an outlier detection model to selectively annotate data points in the contaminated reference set that are suspected as outliers. By removing only the annotated outliers in this ``suspicious'' subset, we can effectively enhance power while mitigating the risk of inflating the type-I error rate, as supported by our theoretical analysis. Experiments on real datasets validate the conservative behavior of conformal methods under contamination and show that the proposed data-cleaning strategy improves power without sacrificing validity.

stat.ML↗

arXiv2025

Noise-Adaptive Conformal Classification with Marginal Coverage

Conformal inference provides a rigorous statistical framework for uncertainty quantification in machine learning, enabling well-calibrated prediction sets with precise coverage guarantees for any classification model. However, its reliance on the idealized assumption of perfect data exchangeability limits its effectiveness in the presence of real-world complications, such as low-quality labels -- a widespread issue in modern large-scale data sets. This work tackles this open problem by introducing an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise, leading to informative prediction sets with tight marginal coverage guarantees even in those challenging scenarios. We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets, including CIFAR-10H and BigEarthNet.

stat.ME↗

arXiv2024

Doubly Robust Conformalized Survival Analysis with Right-Censored Data

We present a conformal inference method for constructing lower prediction bounds for survival times from right-censored data, extending recent approaches designed for more restrictive type-I censoring scenarios. The proposed method imputes unobserved censoring times using a machine learning model, and then analyzes the imputed data using a survival model calibrated via weighted conformal inference. This approach is theoretically supported by an asymptotic double robustness property. Empirical studies on simulated and real data demonstrate that our method leads to relatively informative predictive inferences and is especially robust in challenging settings where the survival model may be inaccurate.

stat.ME↗

arXiv2024

Searching for local associations while controlling the false discovery rate

We introduce local conditional hypotheses that express how the relation between explanatory variables and outcomes changes across different contexts, described by covariates. By expanding upon the model-X knockoff filter, we show how to adaptively discover these local associations, all while controlling the false discovery rate. Our enhanced inferences can help explain sample heterogeneity and uncover interactions, making better use of the capabilities offered by modern machine learning models. Specifically, our method is able to leverage any model for the identification of data-driven hypotheses pertaining to different contexts. Then, it rigorously test these hypotheses without succumbing to selection bias. Importantly, our approach is efficient and does not require sample splitting. We demonstrate the effectiveness of our method through numerical experiments and by studying the genetic architecture of Waist-Hip-Ratio across different sexes in the UKBiobank.

stat.ME↗

arXiv2024

Conformal Classification with Equalized Coverage for Adaptively Selected Groups

This paper introduces a conformal inference method to evaluate uncertainty in classification by generating prediction sets with valid coverage conditional on adaptively chosen features. These features are carefully selected to reflect potential model limitations or biases. This can be useful to find a practical compromise between efficiency -- by providing informative predictions -- and algorithmic fairness -- by ensuring equalized coverage for the most sensitive groups. We demonstrate the validity and effectiveness of this method on simulated and real data sets.

stat.ML↗

arXiv2024

Structured Conformal Inference for Matrix Completion with Applications to Group Recommender Systems

We develop a conformal inference method to construct a joint confidence region for a given group of missing entries within a sparsely observed matrix, focusing primarily on entries from the same column. Our method is model-agnostic and can be combined with any ``black-box'' matrix completion algorithm to provide reliable uncertainty estimation for group-level recommendations. For example, in the context of movie recommendations, it is useful to quantify the uncertainty in the ratings assigned by all members of a group to the same movie, enabling more informed decision-making when individual preferences may conflict. Unlike existing conformal techniques, which estimate uncertainty for one individual at a time, our method provides stronger group-level guarantees by assembling a structured calibration dataset that mimics the dependencies expected in the test group. To achieve this, we introduce a generalized weighted conformalization framework that addresses the lack of exchangeability arising from structured calibration, introducing several innovations to overcome associated computational challenges. We demonstrate the practicality and effectiveness of our approach through extensive numerical experiments and an analysis of the MovieLens 100K dataset.

stat.ME↗

arXiv2024

Uncertainty in Language Models: Assessment through Rank-Calibration

Language Models (LMs) have shown promising performance in natural language generation. However, as LMs often generate incorrect or hallucinated responses, it is crucial to correctly quantify their uncertainty in responding to given inputs. In addition to verbalized confidence elicited via prompting, many uncertainty measures ($e.g.$, semantic entropy and affinity-graph-based measures) have been proposed. However, these measures can differ greatly, and it is unclear how to compare them, partly because they take values over different ranges ($e.g.$, $[0,\infty)$ or $[0,1]$). In this work, we address this issue by developing a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs. Our key tenet is that higher uncertainty (or lower confidence) should imply lower generation quality, on average. Rank-calibration quantifies deviations from this ideal relationship in a principled manner, without requiring ad hoc binary thresholding of the correctness score ($e.g.$, ROUGE or METEOR). The broad applicability and the granular interpretability of our methods are demonstrated empirically.

cs.CL↗

arXiv2024

Conformalized Adaptive Forecasting of Heterogeneous Trajectories

This paper presents a new conformal method for generating simultaneous forecasting bands guaranteed to cover the entire path of a new random trajectory with sufficiently high probability. Prompted by the need for dependable uncertainty estimates in motion planning applications where the behavior of diverse objects may be more or less unpredictable, we blend different techniques from online conformal prediction of single and multiple time series, as well as ideas for addressing heteroscedasticity in regression. This solution is both principled, providing precise finite-sample guarantees, and effective, often leading to more informative predictions than prior methods.

stat.ML↗

arXiv2024

Conformal Predictive Programming for Chance Constrained Optimization

We propose conformal predictive programming (CPP), a framework to solve chance constrained optimization problems, i.e., optimization problems with constraints that are functions of random variables. CPP utilizes samples from these random variables along with the quantile lemma - central to conformal prediction - to transform the chance constrained optimization problem into a deterministic problem with a quantile reformulation. CPP's main strength is an independent calibration step that provides a posteriori guarantees for the solution of this problem that are of conditional and marginal nature otherwise. These guarantees even apply in settings when assumptions required for obtaining standard a priori guarantees (e.g., in scenario optimization or sample average approximation) are unavailable, difficult to compute, or conservative. Another strength of CPP is that it can easily support different variants of conformal prediction which have been (or will be) proposed within the conformal prediction community. To illustrate this, we present robust CPP to deal with distribution shifts in the random variables and Mondrian CPP to deal with class conditional chance constraints. In a series of case studies, we show the validity of the aforementioned approaches, and illustrate the advantage of CPP as compared to scenario approach.

eess.SY↗

arXiv2023

A smoothed-Bayesian approach to frequency recovery from sketched data

We provide a novel statistical perspective on a classical problem at the intersection of computer science and information theory: recovering the empirical frequency of a symbol in a large discrete dataset using only a compressed representation, or sketch, obtained via random hashing. Departing from traditional algorithmic approaches, recent works have proposed Bayesian nonparametric (BNP) methods that can provide more informative frequency estimates by leveraging modeling assumptions about the distribution of the sketched data. In this paper, we propose a smoothed-Bayesian method, inspired by existing BNP approaches but designed in a frequentist framework to overcome the computational limitations of the BNP approaches when dealing with large-scale data from realistic distributions, including those with power-law tail behaviors. For sketches obtained with a single hash function, our approach is supported by rigorous frequentist properties, including unbiasedness and optimality under a squared error loss function within an intuitive class of linear estimators. For sketches with multiple hash functions, we introduce an approach based on multi-view learning to construct computationally efficient frequency estimators. We validate our method on synthetic and real data, comparing its performance to that of existing alternatives.

stat.ME↗