arXiv subjects

Ian J. Barnett

Publications and source records attributed to Ian J. Barnett.

5 recordsLinked to original sources

arXiv2026

A Confidence Interval for the $\ell_2$ Expected Calibration Error

Recent advances in machine learning have significantly improved prediction accuracy in various applications. However, ensuring the calibration of probabilistic predictions remains a significant challenge. Despite efforts to enhance model calibration, the rigorous statistical evaluation of model calibration remains less explored. In this work, we develop confidence intervals the $\ell_2$ Expected Calibration Error (ECE). We consider top-1-to-$k$ calibration, which includes both the popular notion of confidence calibration as well as full calibration. For a debiased estimator of the ECE, we show asymptotic normality, but with different convergence rates and asymptotic variances for calibrated and miscalibrated models. We develop methods to construct asymptotically valid confidence intervals for the ECE, accounting for this behavior as well as non-negativity. Our theoretical findings are supported through extensive experiments, showing that our methods produce valid confidence intervals with shorter lengths compared to those obtained by resampling-based methods.

stat.ML↗

arXiv2026

Sequential Control of False Positives in Online Change Point Detection

Online change point detection is the process of identifying distributional changes in time-ordered data in real time. In applications such as mobile health (mHealth), repeated testing is often performed as new data arrive, creating a multiple testing problem. Traditional approaches for controlling the family-wise error rate (FWER) are not well suited to this setting because the tests are highly dependent and the number of tests is not fixed in advance. In this work, we introduce a sequential family-wise error rate (sFWER), defined as the probability of at least one false positive within a moving monitoring window. We propose a simulation-based calibration procedure to estimate monitoring thresholds that control the sFWER at a desired level. Through simulation studies, we demonstrate that the proposed procedure achieves the desired error control, while commonly used alternatives are either overly conservative or fail to adequately control false alarms. Finally, we illustrate the proposed approach using passively collected smartphone data from a cohort of adolescents and young adults with affective instability.

stat.ME↗

arXiv2026

Variance component score test for multivariate change point detection with applications to mobile health

Multivariate change point detection is the process of identifying distributional shifts in time-ordered data across multiple features. This task is particularly challenging when the number of features is large relative to the number of observations. This problem is often present in mobile health, where behavioral changes in at-risk patients must be detected in real time in order to prompt timely interventions. We propose a variance component score test (VC*) for detecting changes in feature means and/or variances using only pre-change point data to estimate distributional parameters. Through simulation studies, we show that VC* has higher power than existing methods. Moreover, we demonstrate that reducing bias by using only pre-change point days to estimate parameters outweighs the increased estimator variances in most scenarios. Lastly, we apply VC* and competing methods to passively collected smartphone data in adolescents and young adults with affective instability.

stat.ME↗

arXiv2024

Entropy-Based Strategies for Multi-Bracket Pools

Much work in the parimutuel betting literature has discussed estimating event outcome probabilities or developing optimal wagering strategies, particularly for horse race betting. Some betting pools, however, involve betting not just on a single event, but on a tuple of events. For example, pick six betting in horse racing, March Madness bracket challenges, and predicting a randomly drawn bitstring each involve making a series of individual forecasts. Although traditional optimal wagering strategies work well when the size of the tuple is very small (e.g., betting on the winner of a horse race), they are intractable for more general betting pools in higher dimensions (e.g., March Madness bracket challenges). Hence we pose the multi-brackets problem: supposing we wish to predict a tuple of events and that we know the true probabilities of each potential outcome of each event, what is the best way to tractably generate a set of $n$ predicted tuples? The most general version of this problem is extremely difficult, so we begin with a simpler setting. In particular, we generate $n$ independent predicted tuples according to a distribution having optimal entropy. This entropy-based approach is tractable, scalable, and performs well.

cs.GT↗

arXiv2022

Sparse Neural Additive Model: Interpretable Deep Learning with Feature Selection via Group Sparsity

Interpretable machine learning has demonstrated impressive performance while preserving explainability. In particular, neural additive models (NAM) offer the interpretability to the black-box deep learning and achieve state-of-the-art accuracy among the large family of generalized additive models. In order to empower NAM with feature selection and improve the generalization, we propose the sparse neural additive models (SNAM) that employ the group sparsity regularization (e.g. Group LASSO), where each feature is learned by a sub-network whose trainable parameters are clustered as a group. We study the theoretical properties for SNAM with novel techniques to tackle the non-parametric truth, thus extending from classical sparse linear models such as the LASSO, which only works on the parametric truth. Specifically, we show that SNAM with subgradient and proximal gradient descents provably converges to zero training loss as $t\to\infty$, and that the estimation error of SNAM vanishes asymptotically as $n\to\infty$. We also prove that SNAM, similar to LASSO, can have exact support recovery, i.e. perfect feature selection, with appropriate regularization. Moreover, we show that the SNAM can generalize well and preserve the `identifiability', recovering each feature's effect. We validate our theories via extensive experiments and further testify to the good accuracy and efficiency of SNAM.

stat.ML↗