arXiv subjects

Howard D. Bondell

Publications and source records attributed to Howard D. Bondell.

10 recordsLinked to original sources

arXiv2026

Expectation-Propagation for Bayesian Empirical Likelihood Inference

Bayesian inference typically relies on specifying a parametric model that approximates the data-generating process. However, misspecified models can yield poor convergence rates and unreliable posterior calibration. Bayesian empirical likelihood offers a semi-parametric alternative by replacing the parametric likelihood with a profile empirical likelihood defined through moment constraints, thereby avoiding explicit distributional assumptions. Despite these advantages, Bayesian empirical likelihood faces substantial computational challenges, including the need to solve a constrained optimization problem for each likelihood evaluation and difficulties with non-convex posterior support, particularly in small-sample settings. This paper introduces a variational approach based on expectation-propagation to approximate the Bayesian empirical-likelihood posterior, balancing computational cost and accuracy without altering the target posterior via adjustments such as pseudo-observations. Empirically, we show that our approach can achieve a superior cost-accuracy trade-off relative to existing methods, including Hamiltonian Monte Carlo and variational Bayes. Theoretically, we show that the approximation and the Bayesian empirical-likelihood posterior are asymptotically equivalent.

stat.ME↗

arXiv2024

The R2D2 Prior for Generalized Linear Mixed Models

In Bayesian analysis, the selection of a prior distribution is typically done by considering each parameter in the model. While this can be convenient, in many scenarios it may be desirable to place a prior on a summary measure of the model instead. In this work, we propose a prior on the model fit, as measured by a Bayesian coefficient of determination ($R^2)$, which then induces a prior on the individual parameters. We achieve this by placing a beta prior on $R^2$ and then deriving the induced prior on the global variance parameter for generalized linear mixed models. We derive closed-form expressions in many scenarios and present several approximation strategies when an analytic form is not possible and/or to allow for easier computation. In these situations, we suggest approximating the prior by using a generalized beta prime distribution and provide a simple default prior construction scheme. This approach is quite flexible and can be easily implemented in standard Bayesian software. Lastly, we demonstrate the performance of the method on simulated and real-world data, where the method particularly shines in high-dimensional settings, as well as modeling random effects.

stat.ME↗

arXiv2023

Spatial regression modeling via the R2D2 framework

Spatially dependent data arises in many applications, and Gaussian processes are a popular modelling choice for these scenarios. While Bayesian analyses of these problems have proven to be successful, selecting prior distributions for these complex models remains a difficult task. In this work, we propose a principled approach for setting prior distributions on model variance components by placing a prior distribution on a measure of model fit. In particular, we derive the distribution of the prior coefficient of determination. Placing a beta prior distribution on this measure induces a generalized beta prime prior distribution on the global variance of the linear predictor in the model. This method can also be thought of as shrinking the fit towards the intercept-only (null) model. We derive an efficient Gibbs sampler for the majority of the parameters and use Metropolis-Hasting updates for the others. Finally, the method is applied to a marine protection area data set. We estimate the effect of marine policies on biodiversity and conclude that no-take restrictions lead to a slight increase in biodiversity and that the majority of the variance in the linear predictor comes from the spatial effect.\vspace{12pt}

stat.ME↗

arXiv2023

Cold PAWS: Unsupervised class discovery and addressing the cold-start problem for semi-supervised learning

In many machine learning applications, labeling datasets can be an arduous and time-consuming task. Although research has shown that semi-supervised learning techniques can achieve high accuracy with very few labels within the field of computer vision, little attention has been given to how images within a dataset should be selected for labeling. In this paper, we propose a novel approach based on well-established self-supervised learning, clustering, and manifold learning techniques that address this challenge of selecting an informative image subset to label in the first instance, which is known as the cold-start or unsupervised selective labelling problem. We test our approach using several publicly available datasets, namely CIFAR10, Imagenette, DeepWeeds, and EuroSAT, and observe improved performance with both supervised and semi-supervised learning strategies when our label selection strategy is used, in comparison to random sampling. We also obtain superior performance for the datasets considered with a much simpler approach compared to other methods in the literature.

cs.CV↗

arXiv2022

On Robust Probabilistic Principal Component Analysis using Multivariate $t$-Distributions

Probabilistic principal component analysis (PPCA) is a probabilistic reformulation of principal component analysis (PCA), under the framework of a Gaussian latent variable model. To improve the robustness of PPCA, it has been proposed to change the underlying Gaussian distributions to multivariate $t$-distributions. Based on the representation of $t$-distribution as a scale mixture of Gaussian distributions, a hierarchical model is used for implementation. However, in the existing literature, the hierarchical model implemented does not yield the equivalent interpretation. In this paper, we present two sets of equivalent relationships between the high-level multivariate $t$-PPCA framework and the hierarchical model used for implementation. In doing so, we clarify a current misrepresentation in the literature, by specifying the correct correspondence. In addition, we discuss the performance of different multivariate $t$ robust PPCA methods both in theory and simulation studies, and propose a novel Monte Carlo expectation-maximization (MCEM) algorithm to implement one general type of such models.

stat.ME↗

arXiv2020

Conditional Density Estimation via Weighted Logistic Regressions

Compared to the conditional mean as a simple point estimator, the conditional density function is more informative to describe the distributions with multi-modality, asymmetry or heteroskedasticity. In this paper, we propose a novel parametric conditional density estimation method by showing the connection between the general density and the likelihood function of inhomogeneous Poisson process models. The maximum likelihood estimates can be obtained via weighted logistic regressions, and the computation can be significantly relaxed by combining a block-wise alternating maximization scheme and local case-control sampling. We also provide simulation studies for illustration.

stat.ME↗

arXiv2020

Nonparametric Conditional Density Estimation In A Deep Learning Framework For Short-Term Forecasting

Short-term forecasting is an important tool in understanding environmental processes. In this paper, we incorporate machine learning algorithms into a conditional distribution estimator for the purposes of forecasting tropical cyclone intensity. Many machine learning techniques give a single-point prediction of the conditional distribution of the target variable, which does not give a full accounting of the prediction variability. Conditional distribution estimation can provide extra insight on predicted response behavior, which could influence decision-making and policy. We propose a technique that simultaneously estimates the entire conditional distribution and flexibly allows for machine learning techniques to be incorporated. A smooth model is fit over both the target variable and covariates, and a logistic transformation is applied on the model output layer to produce an expression of the conditional density function. We provide two examples of machine learning models that can be used, polynomial regression and deep learning models. To achieve computational efficiency we propose a case-control sampling approximation to the conditional distribution. A simulation study for four different data distributions highlights the effectiveness of our method compared to other machine learning-based conditional distribution estimation techniques. We then demonstrate the utility of our approach for forecasting purposes using tropical cyclone data from the Atlantic Seaboard. This paper gives a proof of concept for the promise of our method, further computational developments can fully unlock its insights in more complex forecasting and other applications.

stat.ML↗

arXiv2020

Bayesian Regression Using a Prior on the Model Fit: The R2-D2 Shrinkage Prior

Prior distributions for high-dimensional linear regression require specifying a joint distribution for the unobserved regression coefficients, which is inherently difficult. We instead propose a new class of shrinkage priors for linear regression via specifying a prior first on the model fit, in particular, the coefficient of determination, and then distributing through to the coefficients in a novel way. The proposed method compares favourably to previous approaches in terms of both concentration around the origin and tail behavior, which leads to improved performance both in posterior contraction and in empirical performance. The limiting behavior of the proposed prior is $1/x$, both around the origin and in the tails. This behavior is optimal in the sense that it simultaneously lies on the boundary of being an improper prior both in the tails and around the origin. None of the existing shrinkage priors obtain this behavior in both regions simultaneously. We also demonstrate that our proposed prior leads to the same near-minimax posterior contraction rate as the spike-and-slab prior.

stat.ME↗

arXiv2019

Deep Distribution Regression

Due to their flexibility and predictive performance, machine-learning based regression methods have become an important tool for predictive modeling and forecasting. However, most methods focus on estimating the conditional mean or specific quantiles of the target quantity and do not provide the full conditional distribution, which contains uncertainty information that might be crucial for decision making. In this article, we provide a general solution by transforming a conditional distribution estimation problem into a constrained multi-class classification problem, in which tools such as deep neural networks. We propose a novel joint binary cross-entropy loss function to accomplish this goal. We demonstrate its performance in various simulation studies comparing to state-of-the-art competing methods. Additionally, our method shows improved accuracy in a probabilistic solar energy forecasting problem.

stat.ML↗

arXiv2016

Variable selection via penalized credible regions with Dirichlet-Laplace global-local shrinkage priors

The method of Bayesian variable selection via penalized credible regions separates model fitting and variable selection. The idea is to search for the sparsest solution within the joint posterior credible regions. Although the approach was successful, it depended on the use of conjugate normal priors. More recently, improvements in the use of global-local shrinkage priors have been made for high-dimensional Bayesian variable selection. In this paper, we incorporate global-local priors into the credible region selection framework. The Dirichlet-Laplace (DL) prior is adapted to linear regression. Posterior consistency for the normal and DL priors are shown, along with variable selection consistency. We further introduce a new method to tune hyperparameters in prior distributions for linear regression. We propose to choose the hyperparameters to minimize a discrepancy between the induced distribution on R-square and a prespecified target distribution. Prior elicitation on R-square is more natural, particularly when there are a large number of predictor variables in which elicitation on that scale is not feasible. For a normal prior, these hyperparameters are available in closed form to minimize the Kullback-Leibler divergence between the distributions.

stat.ME↗