arXiv subjects

Christopher T. Franck

Publications and source records attributed to Christopher T. Franck.

13 recordsLinked to original sources

arXiv2026

Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Function

Machine-generated probability predictions are essential in modern classification tasks such as image classification. A model is well calibrated when its predicted probabilities correspond to observed event frequencies. Despite the need for multicategory recalibration methods, existing methods are limited to (i) comparing calibration between two or more models rather than directly assessing the calibration of a single model, (ii) requiring under-the-hood model access, e.g., accessing logit-scale predictions within the layers of a neural network, and (iii) providing output which is difficult for human analysts to understand. To overcome (i)-(iii), we propose Multicategory Linear Log Odds (MCLLO) recalibration, which (i) includes a likelihood ratio hypothesis test to assess calibration, (ii) does not require under-the-hood access to models and is thus applicable on a wide range of classification problems, and (iii) can be easily interpreted. We demonstrate the effectiveness of the MCLLO method through simulations and three real-world case studies involving image classification via convolutional neural network, obesity analysis via random forest, and ecology via regression modeling. We compare MCLLO to four comparator recalibration techniques utilizing both our hypothesis test and the existing calibration metric Expected Calibration Error to show that our method works well alone and in concert with other methods.

stat.ML↗

arXiv2025

Gaussian Process Assisted Meta-learning for Image Classification and Object Detection Models

Collecting operationally realistic data to inform machine learning models can be costly. Before collecting new data, it is helpful to understand where a model is deficient. For example, object detectors trained on images of rare objects may not be good at identification in poorly represented conditions. We offer a way of informing subsequent data acquisition to maximize model performance by leveraging the toolkit of computer experiments and metadata describing the circumstances under which the training data was collected (e.g., season, time of day, location). We do this by evaluating the learner as the training data is varied according to its metadata. A Gaussian process (GP) surrogate fit to that response surface can inform new data acquisitions. This meta-learning approach offers improvements to learner performance as compared to data with randomly selected metadata, which we illustrate on both classic learning examples, and on a motivating application involving the collection of aerial images in search of airplanes.

stat.ML↗

arXiv2025

Modular Jump Gaussian Processes

Gaussian processes (GPs) furnish accurate nonlinear predictions with well-calibrated uncertainty. However, the typical GP setup has a built-in stationarity assumption, making it ill-suited for modeling data from processes with sudden changes, or "jumps" in the output variable. The "jump GP" (JGP) was developed for modeling data from such processes, combining local GPs and latent "level" variables under a joint inferential framework. But joint modeling can be fraught with difficulty. We aim to simplify by suggesting a more modular setup, eschewing joint inference but retaining the main JGP themes: (a) learning optimal neighborhood sizes that locally respect manifolds of discontinuity; and (b) a new cluster-based (latent) feature to capture regions of distinct output levels on both sides of the manifold. We show that each of (a) and (b) separately leads to dramatic improvements when modeling processes with jumps. In tandem (but without requiring joint inference) that benefit is compounded, as illustrated on real and synthetic benchmark examples from the recent literature.

stat.ME↗

arXiv2025

Monitoring the calibration of probability forecasts with an application to concept drift detection involving image classification

Machine learning approaches for image classification have led to impressive advances in that field. For example, convolutional neural networks are able to achieve remarkable image classification accuracy across a wide range of applications in industry, defense, and other areas. While these machine learning models boast impressive accuracy, a related concern is how to assess and maintain calibration in the predictions these models make. A classification model is said to be well calibrated if its predicted probabilities correspond with the rates events actually occur. While there are many available methods to assess machine learning calibration and recalibrate faulty predictions, less effort has been spent on developing approaches that continually monitor predictive models for potential loss of calibration as time passes. We propose a cumulative sum-based approach with dynamic limits that enable detection of miscalibration in both traditional process monitoring and concept drift applications. This enables early detection of operational context changes that impact image classification performance in the field. The proposed chart can be used broadly in any situation where the user needs to monitor probability predictions over time for potential lapses in calibration. Importantly, our method operates on probability predictions and event outcomes and does not require under-the-hood access to the machine learning model.

stat.ML↗

arXiv2025

Scale-Location-Truncated Beta Regression: Expanding Beta Regression to Accommodate 0 and 1

Beta regression is frequently used when the outcome variable y is bounded within a specific interval, transformed to the (0, 1) domain if necessary. However, standard beta regression cannot handle data observed at the boundary values of 0 or 1, as the likelihood function takes on values of either 0 or infinity. To address this issue, we propose the Scale-Location-Truncated beta (SLTB) regression model, which extends the beta distribution's domain to the [0, 1] interval. By using scale-location transformation and truncation, SLTB distribution allows positive finite mass to the boundary values, offering a flexible approach for handling values at 0 and 1. In this paper, we demonstrate the effectiveness of the SLTB regression model in comparison to standard beta regression models and other approaches like the Zero-One Inflated Beta (ZOIB) mixture model and XBX regression. Using empirical and simulated data, we compare the performance including predictive accuracy of the SLTB regression model with other methods, particularly in cases with observed boundary data values for y. The SLTB model is shown to offer great flexibility, supporting both linear and nonlinear relationships. Additionally, we implement the SLTB model within maximum likelihood and Bayesian frameworks, employing both hierarchical and non-hierarchical models. These comprehensive implementations demonstrate the broad applicability of SLTB model for modeling data with bounded values in a variety of contexts.

stat.ME↗

arXiv2025

BRcal: An R Package to Boldness-Recalibrate Probability Predictions

When probability predictions are too cautious for decision making, boldness-recalibration enables responsible emboldening while maintaining the probability of calibration required by the user. We formulate boldness-recalibration as a nonlinear optimization of boldness with a nonlinear inequality constraint on calibration. We further show that recalibration based on the maximized linear log odds likelihood also maximizes the posterior probability of calibration. We introduce BRcal, an R package implementing boldness-recalibration and supporting methodology as recently proposed. The BRcal package provides direct control of the calibration-boldness tradeoff and visualizes how different calibration levels change individual predictions. We present a new real world case study involving housing foreclosure predictions. The BRcal package is available on the Comprehensive R Archive Network (CRAN) (https://cran.r-project.org/web/packages/BRcal/index.html) and on Github (https://github.com/apguthrie/BRcal).

stat.ME↗

arXiv2024

Chapter 10: Quantitative Models of Discounting

This chapter provides a tutorial that the reader can follow towards analyzing discounting data. Previous chapters have already described the breadth of outcomes associated with discounting (Odum et al. 2020) and other background information (Odum 2011). We focus on delay discounting, where indifference points describe the value of a delayed reward that a participant would be willing to accept in order to have the reward immediately for a variety of delays. This chapter describes the two-stage approach to analyzing discounting data, as this is the simplest approach that is also statistically defensible. The two-stage approach first quantifies the discounting rate of each participant individually, and second stage analyzes these rates as a function of relevant variables (e.g., between smokers and non-smokers)

stat.AP↗

arXiv2024

Flexible cost-penalized Bayesian model selection: developing inclusion paths with an application to diagnosis of heart disease

We propose a Bayesian model selection approach that allows medical practitioners to select among predictor variables while taking their respective costs into account. Medical procedures almost always incur costs in time and/or money. These costs might exceed their usefulness for modeling the outcome of interest. We develop Bayesian model selection that uses flexible model priors to penalize costly predictors a priori and select a subset of predictors useful relative to their costs. Our approach (i) gives the practitioner control over the magnitude of cost penalization, (ii) enables the prior to scale well with sample size, and (iii) enables the creation of our proposed inclusion path visualization, which can be used to make decisions about individual candidate predictors using both probabilistic and visual tools. We demonstrate the effectiveness of our inclusion path approach and the importance of being able to adjust the magnitude of the prior's cost penalization through a dataset pertaining to heart disease diagnosis in patients at the Cleveland Clinic Foundation, where several candidate predictors with various costs were recorded for patients, and through simulated data.

stat.ME↗

arXiv2024

Bayesian Model Selection with Latent Group-Based Effects and Variances with the R Package slgf

Linear modeling is ubiquitous, but performance can suffer when the model is misspecified. We have recently demonstrated that latent groupings in the levels of categorical predictors can complicate inference in a variety of fields including bioinformatics, agriculture, industry, engineering, and medicine. Here we present the R package slgf which enables the user to easily implement our recently-developed approach to detect group-based regression effects, latent interactions, and/or heteroscedastic error variance through Bayesian model selection. We focus on the scenario in which the levels of a categorical predictor exhibit two latent groups. We treat the detection of this grouping structure as an unsupervised learning problem by searching the space of possible groupings of factor levels. First we review the suspected latent grouping factor (SLGF) method. Next, using both observational and experimental data, we illustrate the usage of slgf in the context of several common linear model layouts: one-way analysis of variance (ANOVA), analysis of covariance (ANCOVA), a two-way replicated layout, and a two-way unreplicated layout. We have selected data that reveal the shortcomings of classical analyses to emphasize the advantage our method can provide when a latent grouping structure is present.

stat.ME↗

arXiv2024

Predicting competitions by combining conditional logistic regression and subjective Bayes: An Academy Awards case study

Predicting the outcome of elections, sporting events, entertainment awards, and other competitions has long captured the human imagination. Such prediction is growing in sophistication in these areas, especially in the rapidly growing field of data-driven journalism intended for a general audience as the availability of historical information rapidly balloons. Providing statistical methodology to probabilistically predict competition outcomes faces two main challenges. First, a suitably general modeling approach is necessary to assign probabilities to competitors. Second, the modeling framework must be able to accommodate expert opinion, which is usually available but difficult to fully encapsulate in typical data sets. We overcome these challenges with a combined conditional logistic regression/subjective Bayes approach. To illustrate the method, we re-analyze data from a recent Time.com piece in which the authors attempted to predict the 2019 Best Picture Academy Award winner using standard logistic regression. Towards engaging and educating a broad readership, we discuss strategies to deploy the proposed method via an online application.

stat.AP↗

arXiv2024

Boldness-Recalibration for Binary Event Predictions

Probability predictions are essential to inform decision making across many fields. Ideally, probability predictions are (i) well calibrated, (ii) accurate, and (iii) bold, i.e., spread out enough to be informative for decision making. However, there is a fundamental tension between calibration and boldness, since calibration metrics can be high when predictions are overly cautious, i.e., non-bold. The purpose of this work is to develop a Bayesian model selection-based approach to assess calibration, and a strategy for boldness-recalibration that enables practitioners to responsibly embolden predictions subject to their required level of calibration. Specifically, we allow the user to pre-specify their desired posterior probability of calibration, then maximally embolden predictions subject to this constraint. We demonstrate the method with a case study on hockey home team win probabilities and then verify the performance of our procedures via simulation. We find that very slight relaxation of calibration probability (e.g., from 0.99 to 0.95) can often substantially embolden predictions when they are well calibrated and accurate (e.g., widening hockey predictions range from .26-.78 to .10-.91).

stat.ME↗

arXiv2020

Assessing Bayes factor surfaces using interactive visualization and computer surrogate modeling

Bayesian model selection provides a natural alternative to classical hypothesis testing based on p-values. While many papers mention that Bayesian model selection is frequently sensitive to prior specification on the parameters, there are few practical strategies to assess and report this sensitivity. This article has two goals. First, we aim educate the broader statistical community about the extent of potential sensitivity through visualization of the Bayes factor surface. The Bayes factor surface shows the value a Bayes factor takes (usually on the log scale) as a function of user-specified hyperparameters. We provide interactive visualization through an R shiny application that allows the user to explore sensitivity in Bayes factor over a range of hyperparameter settings in a familiar regression setting. We compare the surface with three automatic procedures. Second, we suggest surrogate modeling via Gaussian processes (GPs) to visualize the Bayes factor surface in situations where computation of Bayes factors is expensive. That is, we treat Bayes factor calculation as a computer simulation experiment. In this context, we provide a fully reproducible example using accessible GP libraries to augment an important study of the influence of outliers in empirical finance. We suggest Bayes factor surfaces are valuable for scientific reporting since they (i) increase transparency, making potential instability in Bayes factors easy to visualize, (ii) generalize to simple and more complicated examples, and (iii) provide a path for researchers to assess the impact of prior choice on modeling decisions in a wide variety research areas.

stat.ME↗

arXiv2019

Detection of latent heteroscedasticity and group-based regression effects in linear models via Bayesian model selection

Standard linear modeling approaches make potentially simplistic assumptions regarding the structure of categorical effects that may obfuscate more complex relationships governing data. For example, recent work focused on the two-way unreplicated layout has shown that hidden groupings among the levels of one categorical predictor frequently interact with the ungrouped factor. We extend the notion of a "latent grouping factor" to linear models in general. The proposed work allows researchers to determine whether an apparent grouping of the levels of a categorical predictor reveals a plausible hidden structure given the observed data. Specifically, we offer Bayesian model selection-based approaches to reveal latent group-based heteroscedasticity, regression effects, and/or interactions. Failure to account for such structures can produce misleading conclusions. Since the presence of latent group structures is frequently unknown a priori to the researcher, we use fractional Bayes factor methods and mixture $g$-priors to overcome lack of prior information.

stat.ME↗