arXiv subjects

Jonathan P Williams

Publications and source records attributed to Jonathan P Williams.

17 recordsLinked to original sources

arXiv2025

Model-free generalized fiducial inference

Conformal prediction (CP) was developed to provide finite-sample probabilistic prediction guarantees. While CP algorithms are a relatively general-purpose approach to uncertainty quantification, with finite-sample guarantees, they lack versatility. Namely, the CP approach does not {\em prescribe} how to quantify the degree to which a data set provides evidence in support of (or against) an arbitrary event from a general class of events. In this paper, tools are offered from imprecise probability theory to build a formal connection between CP and generalized fiducial (GF) inference. These new insights establish a more general inferential lens from which CP can be understood, and demonstrate the pragmatism of fiducial ideas. The formal connection establishes a context in which epistemically-derived GF probability matches aleatoric/frequentist probability. Beyond this fact, it is illustrated how tools from imprecise probability theory, namely lower and upper probability functions, can be applied in the context of the imprecise GF distribution to provide posterior-like, prescriptive inference that is not possible within the CP framework alone. In addition to the primary CP generalization that is contributed, fundamental connections are synthesized between this new model-free GF and three other areas of contemporary research: nonparametric predictive inference (NPI), conformal predictive systems/distributions, and inferential models (IMs).

stat.ML↗

arXiv2024

Conformal Prediction for Astronomy Data with Measurement Error

Astronomers often deal with data where the covariates and the dependent variable are measured with heteroscedastic non-Gaussian error. For instance, while TESS and Kepler datasets provide a wealth of information, addressing the challenges of measurement errors and systematic biases is critical for extracting reliable scientific insights and improving machine learning models' performance. Although techniques have been developed for estimating regression parameters for these data, few techniques exist to construct prediction intervals with finite sample coverage guarantees. To address this issue, we tailor the conformal prediction approach to our application. We empirically demonstrate that this method gives finite sample control over Type I error probabilities under a variety of assumptions on the measurement errors in the observed data. Further, we demonstrate how the conformal prediction method could be used for constructing prediction intervals for unobserved exoplanet masses using established broken power-law relationships between masses and radii found in the literature.

astro-ph.IM↗

arXiv2024

Decision theory via model-free generalized fiducial inference

Building on the recent development of the model-free generalized fiducial (MFGF) paradigm (Williams, 2023) for predictive inference with finite-sample frequentist validity guarantees, in this paper, we develop an MFGF-based approach to decision theory. Beyond the utility of the new tools we contribute to the field of decision theory, our work establishes a formal connection between decision theories from the perspectives of fiducial inference, conformal prediction, and imprecise probability theory. In our paper, we establish pointwise and uniform consistency of an {\em MFGF upper risk function} as an approximation to the true risk function via the derivation of nonasymptotic concentration bounds, and our work serves as the foundation for future investigations of the properties of the MFGF upper risk from the perspective of new decision-theoretic, finite-sample validity criterion, as in Martin (2021).

math.ST↗

arXiv2024

A Statistical Primer on Classical Period-Finding Techniques in Astronomy

The aim of our paper is to investigate the properties of the classical phase-dispersion minimization (PDM), analysis of variance (AOV), string-length (SL), and Lomb-Scargle (LS) power statistics from a statistician's perspective. We confirm that when the data are perturbations of a constant function, i.e. under the null hypothesis of no period in the data, a scaled version of the PDM statistic follows a beta distribution, the AOV statistic follows an F distribution, and the LS power follows a chi-squared distribution with two degrees of freedom. However, the SL statistic does not have a closed-form distribution. We further verify these theoretical distributions through simulations and demonstrate that the extreme values of these statistics (over a range of trial periods), often used for period estimation and determination of the false alarm probability (FAP), follow different distributions than those derived for a single period. We emphasize that multiple-testing considerations are needed to correctly derive FAP bounds. Though, in fact, multiple-testing controls are built into the FAP bound for these extreme-value statistics, e.g. the FAP bound derived specifically for the maximum LS power statistic over a range of trial periods. Additionally, we find that all of these methods are robust to heteroscedastic noise aimed to mimic the degradation or miscalibration of an instrument over time. Finally, we examine the ability of these statistics to detect a non-constant periodic function via simulating data that mimics a well-detached binary system, and we find that the AOV statistic has the most power to detect the correct period, which agrees with what has been observed in practice.

astro-ph.EP↗

arXiv2024

Bayesian hidden Markov models for latent variable labeling assignments in conflict research: application to the role ceasefires play in conflict dynamics

A crucial challenge for solving problems in conflict research is in leveraging the semi-supervised nature of the data that arise. Observed response data such as counts of battle deaths over time indicate latent processes of interest such as intensity and duration of conflicts, but defining and labeling instances of these unobserved processes requires nuance and imprecision. The availability of such labels, however, would make it possible to study the effect of intervention-related predictors - such as ceasefires - directly on conflict dynamics (e.g., latent intensity) rather than through an intermediate proxy like observed counts of battle deaths. Motivated by this problem and the new availability of the ETH-PRIO Civil Conflict Ceasefires data set, we propose a Bayesian autoregressive (AR) hidden Markov model (HMM) framework as a sufficiently flexible machine learning approach for semi-supervised regime labeling with uncertainty quantification. We motivate our approach by illustrating the way it can be used to study the role that ceasefires play in shaping conflict dynamics. This ceasefires data set is the first systematic and globally comprehensive data on ceasefires, and our work is the first to analyze this new data and to explore the effect of ceasefires on conflict dynamics in a comprehensive and cross-country manner.

stat.AP↗

arXiv2023

The EAS approach to variable selection for multivariate response data in high-dimensional settings

In this paper, we develop an {\em epsilon admissible subsets} (EAS) model selection approach for performing group variable selection in the high-dimensional multivariate regression setting. This EAS strategy is designed to estimate a posterior-like, generalized fiducial distribution over a parsimonious class of models in the setting of correlated predictors and/or in the absence of a sparsity assumption. The effectiveness of our approach, to this end, is demonstrated empirically in simulation studies, and is compared to other state-of-the-art model/variable selection procedures. Furthermore, assuming a matrix-Normal linear model we show that the EAS strategy achieves {\em strong model selection consistency} in the high-dimensional setting if there does exist a sparse, true data generating set of predictors. In contrast to Bayesian approaches for model selection, our generalized fiducial approach completely avoids the problem of simultaneously having to specify arbitrary prior distributions for model parameters and penalize model complexity; our approach allows for inference directly on the model complexity. \textcolor{black}{Implementation of the method is illustrated through yeast data to identify significant cell-cycle regulating transcription factors.

stat.ME↗

arXiv2022

Generalized Fiducial Inference on Differentiable Manifolds

We introduce a novel approach to inference on parameters that take values in a Riemannian manifold embedded in a Euclidean space. Parameter spaces of this form are ubiquitous across many fields, including chemistry, physics, computer graphics, and geology. This new approach uses generalized fiducial inference to obtain a posterior-like distribution on the manifold, without needing to know a parameterization that maps the constrained space to an unconstrained Euclidean space. The proposed methodology, called the constrained generalized fiducial distribution (CGFD), is obtained by using mathematical tools from Riemannian geometry. A Bernstein-von Mises-type result for the CGFD, which provides intuition for how the desirable asymptotic qualities of the unconstrained generalized fiducial distribution are inherited by the CGFD, is provided. To demonstrate the practical use of the CGFD, we provide three proof-of-concept examples: inference for data from a multivariate normal density with the mean parameters on a sphere, a linear logspline density estimation problem, and a reimagined approach to the AR(1) model, all of which exhibit desirable coverages via simulation. We discuss two Markov chain Monte Carlo algorithms for the exploration of these constrained parameter spaces and adapt them for the CGFD.

stat.ME↗

arXiv2022

A penalized complexity prior for deep Bayesian transfer learning with application to materials informatics

A key task in the emerging field of materials informatics is to use machine learning to predict a material's properties and functions. A fast and accurate predictive model allows researchers to more efficiently identify or construct a material with desirable properties. As in many fields, deep learning is one of the state-of-the art approaches, but fully training a deep learning model is not always feasible in materials informatics due to limitations on data availability, computational resources, and time. Accordingly, there is a critical need in the application of deep learning to materials informatics problems to develop efficient transfer learning algorithms. The Bayesian framework is natural for transfer learning because the model trained from the source data can be encoded in the prior distribution for the target task of interest. However, the Bayesian perspective on transfer learning is relatively unaccounted for in the literature, and is complicated for deep learning because the parameter space is large and the interpretations of individual parameters are unclear. Therefore, rather than subjective prior distributions for individual parameters, we propose a new Bayesian transfer learning approach based on the penalized complexity prior on the Kullback-Leibler divergence between the predictive models of the source and target tasks. We show via simulations that the proposed method outperforms other transfer learning methods across a variety of settings. The new method is then applied to a predictive materials science problem where we show improved precision for estimating the band gap of a material based on its structural properties.

stat.AP↗

arXiv2021

Conformal prediction for text infilling and part-of-speech prediction

Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new conformal prediction-enhanced bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (BiLSTM) algorithms for POS tagging and a new conformal prediction-enhanced BERT algorithm for text infilling. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.

stat.ML↗

arXiv2021

yaglm: a Python package for fitting and tuning generalized linear models that supports structured, adaptive and non-convex penalties

The yaglm package aims to make the broader ecosystem of modern generalized linear models accessible to data analysts and researchers. This ecosystem encompasses a range of loss functions (e.g. linear, logistic, quantile regression), constraints (e.g. positive, isotonic) and penalties. Beyond the basic lasso/ridge, the package supports structured penalties such as the nuclear norm as well as the group, exclusive, fused, and generalized lasso. It also supports more accurate adaptive and non-convex (e.g. SCAD) versions of these penalties that often come with strong statistical guarantees at limited additional computational expense. yaglm comes with a variety of tuning parameter selection methods including: cross-validation, information criteria that have favorable model selection properties, and degrees of freedom estimators. While several solvers are built in (e.g. FISTA), a key design choice allows users to employ their favorite state of the art optimization algorithms. Designed to be user friendly, the package automatically creates tuning parameter grids, supports tuning with fast path algorithms along with parallelization, and follows a unified scikit-learn compatible API.

stat.CO↗

arXiv2021

Discussion of "A Gibbs sampler for a class of random convex polytopes"

An exciting new algorithmic breakthrough has been advanced for how to carry out inferences in a Dempster-Shafer (DS) formulation of a categorical data generating model. The developed sampling mechanism, which draws on theory for directed graphs, is a clever and remarkable achievement, as this has been an open problem for many decades. In this discussion, I comment on important contributions, central questions, and prevailing matters of the article.

stat.CO↗

arXiv2020

Generalized fiducial factor: an alternative to the Bayes factor for forensic identification of source problems

One formulation of forensic identification of source problems is to determine the source of trace evidence, for instance, glass fragments found on a suspect for a crime. The current state of the science is to compute a Bayes factor (BF) comparing the marginal distribution of measurements of trace evidence under two competing propositions for whether or not the unknown source evidence originated from a specific source. The obvious problem with such an approach is the ability to tailor the prior distributions (placed on the features/parameters of the statistical model for the measurements of trace evidence) in favor of the defense or prosecution, which is further complicated by the fact that the typical number of measurements of trace evidence is typically sufficiently small that prior choice/specification has a strong influence on the value of the BF. To remedy this problem of prior specification and choice, we develop an alternative to the BF, within the framework of generalized fiducial inference (GFI), that we term a {\em generalized fiducial factor} (GFF). Furthermore, we demonstrate empirically, on the synthetic and real Netherlands Forensic Institute (NFI) casework data, deficiencies in the BF and classical/frequentist likelihood ratio (LR) approaches.

stat.ME↗

arXiv2019

The EAS approach for graphical selection consistency in vector autoregression models

As evidenced by various recent and significant papers within the frequentist literature, along with numerous applications in macroeconomics, genomics, and neuroscience, there continues to be substantial interest to understand the theoretical estimation properties of high-dimensional vector autoregression (VAR) models. To date, however, while Bayesian VAR (BVAR) models have been developed and studied empirically (primarily in the econometrics literature) there exist very few theoretical investigations of the repeated sampling properties for BVAR models in the literature. In this direction, we construct methodology via the $\varepsilon$-$admissible$ subsets (EAS) approach for posterior-like inference based on a generalized fiducial distribution of relative model probabilities over all sets of active/inactive components (graphs) of the VAR transition matrix. We provide a mathematical proof of $pairwise$ and $strong$ graphical selection consistency for the EAS approach for stable VAR(1) models which is robust to model misspecification, and demonstrate numerically that it is an effective strategy in high-dimensional settings.

stat.ME↗

arXiv2018

A Bayesian Approach to Multi-State Hidden Markov Models: Application to Dementia Progression

People are living longer than ever before, and with this arises new complications and challenges for humanity. Among the most pressing of these challenges is of understanding the role of aging in the development of dementia. This paper is motivated by the Mayo Clinic Study of Aging data for 4742 subjects since 2004, and how it can be used to draw inference on the role of aging in the development of dementia. We construct a hidden Markov model (HMM) to represent progression of dementia from states associated with the buildup of amyloid plaque in the brain, and the loss of cortical thickness. A hierarchical Bayesian approach is taken to estimate the parameters of the HMM with a truly time-inhomogeneous infinitesimal generator matrix, and response functions of the continuous-valued biomarker measurements are cut-point agnostic. A Bayesian approach with these features could be useful in many disease progression models. Additionally, an approach is illustrated for correcting a common bias in delayed enrollment studies, in which some or all subjects are not observed at baseline. Standard software is incapable of accounting for this critical feature, so code to perform the estimation of the model described below is made available online.

stat.ME↗

arXiv2018

An exposition of the false confidence theorem

A recent paper presents the "false confidence theorem" (FCT) which has potentially broad implications for statistical inference using Bayesian posterior uncertainty. This theorem says that with arbitrarily large (sampling/frequentist) probability, there exists a set which does \textit{not} contain the true parameter value, but which has arbitrarily large posterior probability. Since the use of Bayesian methods has become increasingly popular in applications of science, engineering, and business, it is critically important to understand when Bayesian procedures lead to problematic statistical inferences or interpretations. In this paper, we consider a number of examples demonstrating the paradoxical nature of false confidence to begin to understand the contexts in which the FCT does (and does not) play a meaningful role in statistical inference. Our examples illustrate that models involving marginalization to non-linear, not one-to-one functions of multiple parameters play a key role in more extreme manifestations of false confidence.

stat.ME↗

arXiv2018

Non-penalized variable selection in high-dimensional linear model settings via generalized fiducial inference

Standard penalized methods of variable selection and parameter estimation rely on the magnitude of coefficient estimates to decide which variables to include in the final model. However, coefficient estimates are unreliable when the design matrix is collinear. To overcome this challenge an entirely new perspective on variable selection is presented within a generalized fiducial inference framework. This new procedure is able to effectively account for linear dependencies among subsets of covariates in a high-dimensional setting where $p$ can grow almost exponentially in $n$, as well as in the classical setting where $p \le n$. It is shown that the procedure very naturally assigns small probabilities to subsets of covariates which include redundancies by way of explicit $L_{0}$ minimization. Furthermore, with a typical sparsity assumption, it is shown that the proposed method is consistent in the sense that the probability of the true sparse subset of covariates converges in probability to 1 as $n \to \infty$, or as $n \to \infty$ and $p \to \infty$. Very reasonable conditions are needed, and little restriction is placed on the class of possible subsets of covariates to achieve this consistency result.

stat.ME↗

arXiv2010

SMAObservations of GOODS 850-11 and GOODS 850-13 - First Examples of Multiple Submillimeter Sources Resolved by an Interferometer

We carried out extremely sensitive Submillimeter Array (SMA) 340 GHz continuum imaging on two submillimeter galaxies (SMGs): GOODS 850-11 and GOODS 850-13. The observations reach sub-mJy rms sensitivities and, interestingly, resolve both sources into multiple, physically unrelated SMGs. GOODS 850-11 is resolved into two sources at different redshifts. GOODS 850-13 is resolved into three sources, two with different spectroscopic redshifts and one only with a photometric redshift. All the SMA sources have fluxes in the 3-5 mJy range and all are detected at 1.4 GHz. Three of them are detected by Chandra, and one is a previously unknown X-ray SMG. This is the first time that single-dish SMGs are resolved into multiple unrelated sources and also the first time that the SMA has discovered new SMGs. Our results show that identifications of SMGs at any wavelengths other than the submillimeter itself can be misleading, since such identifications usually only pick up one of the real counterparts. Using simulations that mimic our SCUBA and SMA observations, we find that the number of triple systems detected in our SMA survey is much higher than that expected from the current best-determined number counts. We tentatively attribute this to clustering. We also predict that ALMA will find ~1/3 of >5 mJy 850 um SCUBA sources to be multiple systems. Based on our SMA observations and simulations, we suggest that large samples of existing SMGs should be imaged with sensitive interferometric observations, even if the SMGs were previously thought to be securely identified.

astro-ph.CO↗