arXiv subjects

Robert B. Gramacy

Publications and source records attributed to Robert B. Gramacy.

At least 19 recordsLinked to original sources

arXiv2026

Vecchia approximated Bayesian heteroskedastic Gaussian processes

Many computer simulations are stochastic and exhibit input dependent noise. In such situations, heteroskedastic Gaussian processes (hetGPs) make ideal surrogates as they estimate a latent, non-constant variance. However, existing hetGP implementations are unable to deal with large simulation campaigns and use point-estimates for all unknown quantities, including latent variances. This limits applicability to small experiments and undercuts uncertainty. We propose a Bayesian hetGP using elliptical slice sampling (ESS) for posterior variance integration, and the Vecchia approximation to circumvent computational bottlenecks. We show good performance for our upgraded hetGP capability, compared to alternatives, on a benchmark example and a motivating corpus of more than 9-million lake temperature simulations. An open source implementation is provided as bhetGP on CRAN.

stat.ME↗

arXiv2025

Gaussian Process Assisted Meta-learning for Image Classification and Object Detection Models

Collecting operationally realistic data to inform machine learning models can be costly. Before collecting new data, it is helpful to understand where a model is deficient. For example, object detectors trained on images of rare objects may not be good at identification in poorly represented conditions. We offer a way of informing subsequent data acquisition to maximize model performance by leveraging the toolkit of computer experiments and metadata describing the circumstances under which the training data was collected (e.g., season, time of day, location). We do this by evaluating the learner as the training data is varied according to its metadata. A Gaussian process (GP) surrogate fit to that response surface can inform new data acquisitions. This meta-learning approach offers improvements to learner performance as compared to data with randomly selected metadata, which we illustrate on both classic learning examples, and on a motivating application involving the collection of aerial images in search of airplanes.

stat.ML↗

arXiv2025

Bayesian Statistical Inversion for High-Dimensional Computer Model Output and Spatially Distributed Counts

Data collected by the Interstellar Boundary Explorer (IBEX) satellite, recording heliospheric energetic neutral atoms (ENAs), exhibit a phenomenon that has caused space scientists to revise hypotheses about the physical processes, and computer simulations under those models, in play at the boundary of our solar system. Evaluating the fit of these computer models involves tuning their parameters to observational data from IBEX. This would be a classic (Bayesian) inverse problem if not for three challenges: (1) the computer simulations are slow, limiting the size of campaigns of runs; so (2) surrogate modeling is essential, but outputs are high-resolution images, thwarting conventional methods; and (3) IBEX observations are counts, whereas most inverse problem techniques assume Gaussian field data. To fill that gap we propose a novel approach to Bayesian inverse problems coupling a Poisson response with a sparse Gaussian process surrogate using the Vecchia approximation. We demonstrate the capabilities of our proposed framework, which compare favorably to alternatives, through multiple simulated examples in terms of recovering "true" computer model parameters and accurate out-of-sample prediction. We then apply this new technology to IBEX satellite data and associated computer models developed at Los Alamos National Laboratory.

stat.AP↗

arXiv2025

Robust Wrapped Gaussian Process Inference for Noisy Angular Data

Angular data are commonly encountered in settings with a directional or orientational component. Regressing an angular response on real-valued features requires intrinsically capturing the circular or spherical manifold the data lie on, or using an appropriate extrinsic transformation. A popular example of the latter is the technique of distributional wrapping, in which functions are "wrapped" around the unit circle via a modulo-$2π$ transformation. This approach enables flexible, non-linear models like Gaussian processes (GPs) to properly account for circular structure. While straightforward in concept, the need to infer the latent unwrapped distribution along with its wrapping behavior makes inference difficult in noisy response settings, as misspecification of one can severely hinder estimation of the other. However, applications such as radiowave analysis (Shangguan et al., 2015) and biomedical engineering (Kurz and Hanebeck, 2015) encounter radial data where wrapping occurs in only one direction. We therefore propose a novel wrapped GP (WGP) model formulation that recognizes monotonic wrapping behavior for more accurate inference in these situations. This is achieved by estimating the locations where wrapping occurs and partitioning the input space accordingly. We also specify a more robust Student's t response likelihood, and take advantage of an elliptical slice sampling (ESS) algorithm for rejection-free sampling from the latent GP space. We showcase our model's preferable performance on simulated examples compared to existing WGP methodologies. We then apply our method to the problem of localizing radiofrequency identification (RFID) tags, in which we model the relationship between frequency and phase angle to infer how far away an RFID tag is from an antenna.

stat.AP↗

arXiv2025

Modular Jump Gaussian Processes

Gaussian processes (GPs) furnish accurate nonlinear predictions with well-calibrated uncertainty. However, the typical GP setup has a built-in stationarity assumption, making it ill-suited for modeling data from processes with sudden changes, or "jumps" in the output variable. The "jump GP" (JGP) was developed for modeling data from such processes, combining local GPs and latent "level" variables under a joint inferential framework. But joint modeling can be fraught with difficulty. We aim to simplify by suggesting a more modular setup, eschewing joint inference but retaining the main JGP themes: (a) learning optimal neighborhood sizes that locally respect manifolds of discontinuity; and (b) a new cluster-based (latent) feature to capture regions of distinct output levels on both sides of the manifold. We show that each of (a) and (b) separately leads to dramatic improvements when modeling processes with jumps. In tandem (but without requiring joint inference) that benefit is compounded, as illustrated on real and synthetic benchmark examples from the recent literature.

stat.ME↗

arXiv2025

Monotonic warpings for additive and deep Gaussian processes

Gaussian processes (GPs) are canonical as surrogates for computer experiments because they enjoy a degree of analytic tractability. But that breaks when the response surface is constrained, say to be monotonic. Here, we provide a mono-GP construction for a single input that is highly efficient even though the calculations are non-analytic. Key ingredients include transformation of a reference process and elliptical slice sampling. We then show how mono-GP may be deployed effectively in two ways. One is additive, extending monotonicity to more inputs; the other is as a prior on injective latent warping variables in a deep Gaussian process for (non-monotonic, multi-input) non-stationary surrogate modeling. We provide illustrative and benchmarking examples throughout, showing that our methods yield improved performance over the state-of-the-art on examples from those two classes of problems.

stat.CO↗

arXiv2025

Synthesizing data products, mathematical models, and observational measurements for lake temperature forecasting

We present a novel forecasting framework for lake water temperature, which is crucial for managing lake ecosystems and drinking water resources. The General Lake Model (GLM) has been previously used for this purpose, but, similar to many process-based simulation models, it: requires a large number of inputs, many of which are stochastic; presents challenges for uncertainty quantification (UQ); and can exhibit model bias. To address these issues, we propose a Gaussian process (GP) surrogate-based forecasting approach that efficiently handles large, high-dimensional data and accounts for input-dependent variability and systematic GLM bias. We validate the proposed approach and compare it with other forecasting methods, including a climatological model and raw GLM simulations. Our results demonstrate that our bias-corrected GP surrogate (GPBC) can outperform competing approaches in terms of forecast accuracy and UQ up to two weeks into the future.

stat.AP↗

arXiv2025

Modernizing full posterior inference for surrogate modeling of categorical-output simulation experiments

Gaussian processes (GPs) are powerful tools for nonlinear classification in which latent GPs are combined with link functions. But GPs do not scale well to large training data. This is compounded for classification where the latent GPs require Markov chain Monte Carlo integration. Consequently, fully Bayesian, sampling-based approaches had been largely abandoned. Instead, maximization-based alternatives, such as Laplace/variational inference (VI) combined with low rank approximations, are preferred. Though feasible for large training data sets, such schemes sacrifice uncertainty quantification and modeling fidelity, two aspects that are important to our work on surrogate modeling of computer simulation experiments. Here we are motivated by a large scale simulation of binary black hole (BBH) formation. We propose an alternative GP classification framework which uses elliptical slice sampling for Bayesian posterior integration and Vecchia approximation for computational thrift. We demonstrate superiority over VI-based alternatives for BBH simulations and other benchmark classification problems. We then extend our setup to warped inputs for "deep" nonstationary classification.

stat.CO↗

arXiv2024

Voronoi Candidates for Bayesian Optimization

Bayesian optimization (BO) offers an elegant approach for efficiently optimizing black-box functions. However, acquisition criteria demand their own challenging inner-optimization, which can induce significant overhead. Many practical BO methods, particularly in high dimension, eschew a formal, continuous optimization of the acquisition function and instead search discretely over a finite set of space-filling candidates. Here, we propose to use candidates which lie on the boundary of the Voronoi tessellation of the current design points, so they are equidistant to two or more of them. We discuss strategies for efficient implementation by directly sampling the Voronoi boundary without explicitly generating the tessellation, thus accommodating large designs in high dimension. On a battery of test problems optimized via Gaussian processes with expected improvement, our proposed approach significantly improves the execution time of a multi-start continuous search without a loss in accuracy.

stat.ML↗

arXiv2024

Nonstationary Gaussian Process Surrogates

We provide a survey of nonstationary surrogate models which utilize Gaussian processes (GPs) or variations thereof, including nonstationary kernel adaptations, partition and local GPs, and spatial warpings through deep Gaussian processes. We also overview publicly available software implementations and conclude with a bake-off involving an 8-dimensional satellite drag computer experiment. Code for this example is provided in a public git repository.

stat.ME↗

arXiv2024

Augmenting a simulation campaign for hybrid computer model and field data experiments

The Kennedy and O'Hagan (KOH) calibration framework uses coupled Gaussian processes (GPs) to meta-model an expensive simulator (first GP), tune its ``knobs" (calibration inputs) to best match observations from a real physical/field experiment and correct for any modeling bias (second GP) when predicting under new field conditions (design inputs). There are well-established methods for placement of design inputs for data-efficient planning of a simulation campaign in isolation, i.e., without field data: space-filling, or via criterion like minimum integrated mean-squared prediction error (IMSPE). Analogues within the coupled GP KOH framework are mostly absent from the literature. Here we derive a closed form IMSPE criterion for sequentially acquiring new simulator data for KOH. We illustrate how acquisitions space-fill in design space, but concentrate in calibration space. Closed form IMSPE precipitates a closed-form gradient for efficient numerical optimization. We demonstrate that our KOH-IMSPE strategy leads to a more efficient simulation campaign on benchmark problems, and conclude with a showcase on an application to equilibrium concentrations of rare earth elements for a liquid-liquid extraction reaction.

stat.ME↗

arXiv2024

Contour Location for Reliability in Airfoil Simulation Experiments using Deep Gaussian Processes

Bayesian deep Gaussian processes (DGPs) outperform ordinary GPs as surrogate models of complex computer experiments when response surface dynamics are non-stationary, which is especially prevalent in aerospace simulations. Yet DGP surrogates have not been deployed for the canonical downstream task in that setting: reliability analysis through contour location (CL). In that context, we are motivated by a simulation of an RAE-2822 transonic airfoil which demarcates efficient and inefficient flight conditions. Level sets separating passable versus failable operating conditions are best learned through strategic sequential designs. There are two limitations to modern CL methodology which hinder DGP integration in this setting. First, derivative-based optimization underlying acquisition functions is thwarted by sampling-based Bayesian (i.e., MCMC) inference, which is essential for DGP posterior integration. Second, canonical acquisition criteria, such as entropy, are famously myopic to the extent that optimization may even be undesirable. Here we tackle both of these limitations at once, proposing a hybrid criterion that explores along the Pareto front of entropy and (predictive) uncertainty, requiring evaluation only at strategically located "triangulation" candidates. We showcase DGP CL performance in several synthetic benchmark exercises and on the RAE-2822 airfoil.

stat.ME↗

arXiv2023

Robust expected improvement for Bayesian optimization

Bayesian Optimization (BO) links Gaussian Process (GP) surrogates with sequential design toward optimizing expensive-to-evaluate black-box functions. Example design heuristics, or so-called acquisition functions, like expected improvement (EI), balance exploration and exploitation to furnish global solutions under stringent evaluation budgets. However, they fall short when solving for robust optima, meaning a preference for solutions in a wider domain of attraction. Robust solutions are useful when inputs are imprecisely specified, or where a series of solutions is desired. A common mathematical programming technique in such settings involves an adversarial objective, biasing a local solver away from ``sharp'' troughs. Here we propose a surrogate modeling and active learning technique called robust expected improvement (REI) that ports adversarial methodology into the BO/GP framework. After describing the methods, we illustrate and draw comparisons to several competitors on benchmark synthetic exercises and real problems of varying complexity.

cs.LG↗

arXiv2022

Traditional kriging versus modern Gaussian processes for large-scale mining data

The canonical technique for nonlinear modeling of spatial/point-referenced data is known as kriging in geostatistics, and as Gaussian Process (GP) regression for surrogate modeling and statistical learning. This article reviews many similarities shared between kriging and GPs, but also highlights some important differences. One is that GPs impose a process that can be used to automate kernel/variogram inference, thus removing the human from the loop. The GP framework also suggests a probabilistically valid means of scaling to handle a large corpus of training data, i.e., an alternative to so-called ordinary kriging. Finally, recent GP implementations are tailored to make the most of modern computing architectures such as multi-core workstations and multi-node supercomputers. We argue that such distinctions are important even in classically geostatistical settings. To back that up, we present out-of-sample validation exercises using two, real, large-scale borehole data sets involved in the mining of gold and other minerals. We pit classic kriging against the modern GPs in several variations and conclude that the latter can more economical (fewer human and compute resources), more accurate and offer better uncertainty quantification. We go on to show how the fully generative modeling apparatus provided by GPs can gracefully accommodate left-censoring of small measurements, as commonly occurs in mining data and other borehole assays.

stat.AP↗

arXiv2022

Vecchia-approximated Deep Gaussian Processes for Computer Experiments

Deep Gaussian processes (DGPs) upgrade ordinary GPs through functional composition, in which intermediate GP layers warp the original inputs, providing flexibility to model non-stationary dynamics. Two DGP regimes have emerged in recent literature. A "big data" regime, prevalent in machine learning, favors approximate, optimization-based inference for fast, high-fidelity prediction. A "small data" regime, preferred for computer surrogate modeling, deploys posterior integration for enhanced uncertainty quantification (UQ). We aim to bridge this gap by expanding the capabilities of Bayesian DGP posterior inference through the incorporation of the Vecchia approximation, allowing linear computational scaling without compromising accuracy or UQ. We are motivated by surrogate modeling of simulation campaigns with upwards of 100,000 runs - a size too large for previous fully-Bayesian implementations - and demonstrate prediction and UQ superior to that of "big data" competitors. All methods are implemented in the "deepgp" package on CRAN.

stat.CO↗

arXiv2022

Multi-output calibration of a honeycomb seal via on-site surrogates

We consider large-scale industrial computer model calibration, combining multi-output simulation with limited physical observation, involved in the development of a honeycomb seal. Toward that end, we adopt a localized sampling and emulation strategy called "on-site surrogates (OSSs)", designed to cope with the amalgamated challenges of high-dimensional inputs, large-scale simulation campaigns, and nonstationary response surfaces. In previous applications, OSSs were one-at-a-time affairs for multiple outputs leading to dissonance in calibration efforts for a common parameter set across outputs for the honeycomb. We demonstrate that a principal-components representation, adapted from ordinary Gaussian process surrogate modeling to the OSS setting, can resolve this tension. With a two-pronged - optimization and fully Bayesian - approach, we show how pooled information across outputs can reduce uncertainty and enhance efficiency in calibrated parameters and prediction for the honeycomb relative to the previous, "data-poor" univariate analog.

stat.ME↗

arXiv2022

Triangulation candidates for Bayesian optimization

Bayesian optimization involves "inner optimization" over a new-data acquisition criterion which is non-convex/highly multi-modal, may be non-differentiable, or may otherwise thwart local numerical optimizers. In such cases it is common to replace continuous search with a discrete one over random candidates. Here we propose using candidates based on a Delaunay triangulation of the existing input design. We detail the construction of these "tricands" and demonstrate empirically how they outperform both numerically optimized acquisitions and random candidate-based alternatives, and are well-suited for hybrid schemes, on benchmark synthetic and real simulation experiments.

stat.CO↗

arXiv2021

Sensitivity Prewarping for Local Surrogate Modeling

In the continual effort to improve product quality and decrease operations costs, computational modeling is increasingly being deployed to determine feasibility of product designs or configurations. Surrogate modeling of these computer experiments via local models, which induce sparsity by only considering short range interactions, can tackle huge analyses of complicated input-output relationships. However, narrowing focus to local scale means that global trends must be re-learned over and over again. In this article, we propose a framework for incorporating information from a global sensitivity analysis into the surrogate model as an input rotation and rescaling preprocessing step. We discuss the relationship between several sensitivity analysis methods based on kernel regression before describing how they give rise to a transformation of the input variables. Specifically, we perform an input warping such that the "warped simulator" is equally sensitive to all input directions, freeing local models to focus on local dynamics. Numerical experiments on observational data and benchmark test functions, including a high-dimensional computer simulator from the automotive industry, provide empirical validation.

stat.ML↗