arXiv subjects

Rui Tuo

Publications and source records attributed to Rui Tuo.

36 records · Page 2Linked to original sources

arXiv2022

Kernel Packet: An Exact and Scalable Algorithm for Gaussian Process Regression with Matérn Correlations

We develop an exact and scalable algorithm for one-dimensional Gaussian process regression with Matérn correlations whose smoothness parameter $ν$ is a half-integer. The proposed algorithm only requires $\mathcal{O}(ν^3 n)$ operations and $\mathcal{O}(νn)$ storage. This leads to a linear-cost solver since $ν$ is chosen to be fixed and usually very small in most applications. The proposed method can be applied to multi-dimensional problems if a full grid or a sparse grid design is used. The proposed method is based on a novel theory for Matérn correlation functions. We find that a suitable rearrangement of these correlation functions can produce a compactly supported function, called a "kernel packet". Using a set of kernel packets as basis functions leads to a sparse representation of the covariance matrix that results in the proposed algorithm. Simulation studies show that the proposed algorithm, when applicable, is significantly superior to the existing alternatives in both the computational time and predictive accuracy.

stat.ML↗

arXiv2021

High-Dimensional Non-Parametric Density Estimation in Mixed Smooth Sobolev Spaces

Density estimation plays a key role in many tasks in machine learning, statistical inference, and visualization. The main bottleneck in high-dimensional density estimation is the prohibitive computational cost and the slow convergence rate. In this paper, we propose novel estimators for high-dimensional non-parametric density estimation called the adaptive hyperbolic cross density estimators, which enjoys nice convergence properties in the mixed smooth Sobolev spaces. As modifications of the usual Sobolev spaces, the mixed smooth Sobolev spaces are more suitable for describing high-dimensional density functions in some applications. We prove that, unlike other existing approaches, the proposed estimator does not suffer the curse of dimensionality under Integral Probability Metric, including Hölder Integral Probability Metric, where Total Variation Metric and Wasserstein Distance are special cases. Applications of the proposed estimators to generative adversarial networks (GANs) and goodness of fit test for high-dimensional data are discussed to illustrate the proposed estimator's good performance in high-dimensional problems. Numerical experiments are conducted and illustrate the efficiency of our proposed method.

cs.LG↗

arXiv2021

High-Dimensional Simulation Optimization via Brownian Fields and Sparse Grids

High-dimensional simulation optimization is notoriously challenging. We propose a new sampling algorithm that converges to a global optimal solution and suffers minimally from the curse of dimensionality. The algorithm consists of two stages. First, we take samples following a sparse grid experimental design and approximate the response surface via kernel ridge regression with a Brownian field kernel. Second, we follow the expected improvement strategy -- with critical modifications that boost the algorithm's sample efficiency -- to iteratively sample from the next level of the sparse grid. Under mild conditions on the smoothness of the response surface and the simulation noise, we establish upper bounds on the convergence rate for both noise-free and noisy simulation samples. These upper bounds deteriorate only slightly in the dimension of the feasible set, and they can be improved if the objective function is known to be of a higher-order smoothness. Extensive numerical experiments demonstrate that the proposed algorithm dramatically outperforms typical alternatives in practice.

stat.ML↗

arXiv2021

A Reproducing Kernel Hilbert Space Approach to Functional Calibration of Computer Models

This paper develops a frequentist solution to the functional calibration problem, where the value of a calibration parameter in a computer model is allowed to vary with the value of control variables in the physical system. The need of functional calibration is motivated by engineering applications where using a constant calibration parameter results in a significant mismatch between outputs from the computer model and the physical experiment. Reproducing kernel Hilbert spaces (RKHS) are used to model the optimal calibration function, defined as the functional relationship between the calibration parameter and control variables that gives the best prediction. This optimal calibration function is estimated through penalized least squares with an RKHS-norm penalty and using physical data. An uncertainty quantification procedure is also developed for such estimates. Theoretical guarantees of the proposed method are provided in terms of prediction consistency and consistency of estimating the optimal calibration function. The proposed method is tested using both real and synthetic data and exhibits more robust performance in prediction and uncertainty quantification than the existing parametric functional calibration method and a state-of-art Bayesian method.

stat.ME↗

arXiv2021

Gaussian process aided function comparison using noisy scattered data

This work proposes a nonparametric method to compare the underlying mean functions given two noisy datasets. The motivation for the work stems from an application of comparing wind turbine power curves. Comparing wind turbine data presents new problems, namely the need to identify the regions of difference in the input space and to quantify the extent of difference that is statistically significant. Our proposed method, referred to as funGP, estimates the underlying functions for different data samples using Gaussian process models. We build a confidence band using the probability law of the estimated function differences under the null hypothesis. Then, the confidence band is used for the hypothesis test as well as for identifying the regions of difference. This identification of difference regions is a distinct feature, as existing methods tend to conduct an overall hypothesis test stating whether two functions are different. Understanding the difference regions can lead to further practical insights and help devise better control and maintenance strategies for wind turbines. The merit of funGP is demonstrated by using three simulation studies and four real wind turbine datasets.

stat.ME↗

arXiv2020

Kriging prediction with isotropic Matérn correlations: Robustness and experimental design

This work investigates the prediction performance of the kriging predictors. We derive some error bounds for the prediction error in terms of non-asymptotic probability under the uniform metric and $L_p$ metrics when the spectral densities of both the true and the imposed correlation functions decay algebraically. The Matérn family is a prominent class of correlation functions of this kind. Our analysis shows that, when the smoothness of the imposed correlation function exceeds that of the true correlation function, the prediction error becomes more sensitive to the space-filling property of the design points. In particular, the kriging predictor can still reach the optimal rate of convergence, if the experimental design scheme is quasi-uniform. Lower bounds of the kriging prediction error are also derived under the uniform metric and $L_p$ metrics. An accurate characterization of this error is obtained, when an oversmoothed correlation function and a space-filling design is used.

math.ST↗

arXiv2020

Generalization Guarantees for Sparse Kernel Approximation with Entropic Optimal Features

Despite their success, kernel methods suffer from a massive computational cost in practice. In this paper, in lieu of commonly used kernel expansion with respect to $N$ inputs, we develop a novel optimal design maximizing the entropy among kernel features. This procedure results in a kernel expansion with respect to entropic optimal features (EOF), improving the data representation dramatically due to features dissimilarity. Under mild technical assumptions, our generalization bound shows that with only $O(N^{\frac{1}{4}})$ features (disregarding logarithmic factors), we can achieve the optimal statistical accuracy (i.e., $O(1/\sqrt{N})$). The salient feature of our design is its sparsity that significantly reduces the time and space cost. Our numerical experiments on benchmark datasets verify the superiority of EOF over the state-of-the-art in kernel approximation.

cs.LG↗

arXiv2020

On the Improved Rates of Convergence for Matérn-type Kernel Ridge Regression, with Application to Calibration of Computer Models

Kernel ridge regression is an important nonparametric method for estimating smooth functions. We introduce a new set of conditions, under which the actual rates of convergence of the kernel ridge regression estimator under both the L_2 norm and the norm of the reproducing kernel Hilbert space exceed the standard minimax rates. An application of this theory leads to a new understanding of the Kennedy-O'Hagan approach for calibrating model parameters of computer simulation. We prove that, under certain conditions, the Kennedy-O'Hagan calibration estimator with a known covariance function converges to the minimizer of the norm of the residual function in the reproducing kernel Hilbert space.

math.ST↗

arXiv2019

Effective Model Calibration via Sensible Variable Identification and Adjustment, with Application to Composite Fuselage Simulation

Estimation of model parameters of computer simulators, also known as calibration, is an important topic in many engineering applications. In this paper, we consider the calibration of computer model parameters with the help of engineering design knowledge. We introduce the concept of sensible (calibration) variables. Sensible variables are model parameters which are sensitive in the engineering modeling, and whose optimal values differ from the engineering design values.We propose an effective calibration method to identify and adjust the sensible variables with limited physical experimental data. The methodology is applied to a composite fuselage simulation problem.

stat.AP↗

arXiv2019

On Prediction Properties of Kriging: Uniform Error Bounds and Robustness

Kriging based on Gaussian random fields is widely used in reconstructing unknown functions. The kriging method has pointwise predictive distributions which are computationally simple. However, in many applications one would like to predict for a range of untried points simultaneously. In this work we obtain some error bounds for the (simple) kriging predictor under the uniform metric. It works for a scattered set of input points in an arbitrary dimension, and also covers the case where the covariance function of the Gaussian process is misspecified. These results lead to a better understanding of the rate of convergence of kriging under the Gaussian or the Matérn correlation functions, the relationship between space-filling designs and kriging models, and the robustness of the Matérn correlation functions.

math.ST↗

arXiv2018

Differentially Private Change-Point Detection

The change-point detection problem seeks to identify distributional changes at an unknown change-point k* in a stream of data. This problem appears in many important practical settings involving personal data, including biosurveillance, fault detection, finance, signal detection, and security systems. The field of differential privacy offers data analysis tools that provide powerful worst-case privacy guarantees. We study the statistical problem of change-point detection through the lens of differential privacy. We give private algorithms for both online and offline change-point detection, analyze these algorithms theoretically, and provide empirical validation of our results.

math.ST↗

arXiv2017

Deterministic Sampling of Expensive Posteriors Using Minimum Energy Designs

Markov chain Monte Carlo (MCMC) methods require a large number of samples to approximate a posterior distribution, which can be costly when the likelihood or prior is expensive to evaluate. The number of samples can be reduced if we can avoid repeated samples and those that are close to each other. This is the idea behind deterministic sampling methods such as Quasi-Monte Carlo (QMC). However, the existing QMC methods aim at sampling from a uniform hypercube, which can miss the high probability regions of the posterior distribution and thus the approximation can be poor. Minimum energy design (MED) is a recently proposed deterministic sampling method, which makes use of the posterior evaluations to obtain a weighted space-filling design in the region of interest. However, the existing implementation of MED is inefficient because it requires several global optimizations and thus numerous evaluations of the posterior. In this article, we develop an efficient algorithm that can generate MED with few posterior evaluations. We also make several improvements to the MED criterion to make it perform better in high dimensions. The advantages of MED over MCMC and QMC are illustrated using an example of calibrating a friction drilling process.

stat.CO↗

arXiv2017

Adjustments to Computer Models via Projected Kernel Calibration

Identification of model parameters in computer simulations is an important topic in computer experiments. We propose a new method, called the projected kernel calibration method, to estimate these model parameters. The proposed method is proven to be asymptotic normal and semi-parametric efficient. As a frequentist method, the proposed method is as efficient as the $L_2$ calibration method proposed by Tuo and Wu [Ann. Statist. 43 (2015) 2331-2352]. On the other hand, the proposed method has a natural Bayesian version, which the $L_2$ method does not have. This Bayesian version allows users to calculate the credible region of the calibration parameters without using a large sample approximation. We also show that, the inconsistency problem of the calibration method proposed by Kennedy and O'Hagan [J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 (2001) 425-464] can be rectified by a simple modification of the kernel matrix.

stat.ME↗

arXiv2017

Prediction based on the Kennedy-O'Hagan calibration model: asymptotic consistency and other properties

Kennedy and O'Hagan (2001) propose a model for calibrating some unknown parameters in a computer model and estimating the discrepancy between the computer output and physical response. This model is known to have certain identifiability issues. Tuo and Wu (2016) show that there are examples for which the Kennedy-O'Hagan method renders unreasonable results in calibration. In spite of its unstable performance in calibration, the Kennedy-O'Hagan approach has a more robust behavior in predicting the physical response. In this work, we present some theoretical analysis to show the consistency of predictor based on their calibration model in the context of radial basis functions.

math.ST↗

arXiv2017

A finite element method for elliptic problems with observational boundary data

In this paper we propose a finite element method for solving elliptic equations with the observational Dirichlet boundary data which may subject to random noises. The method is based on the weak formulation of Lagrangian multiplier. We show the convergence of the random finite element error in expectation and, when the noise is sub-Gaussian, in the Orlicz 2- norm which implies the probability that the finite element error estimates are violated decays exponentially. Numerical examples are included.

math.NA↗

arXiv2017

Stochastic Convergence of A Nonconforming Finite Element Method for the Thin Plate Spline Smoother for Observational Data

The thin plate spline smoother is a classical model for fnding a smooth function from the knowledge of its observation at scattered locations which may have random noises. We consider a nonconforming Morley finite element method to approximate the model. We prove the stochastic convergence of the finite element method which characterizes the tail property of the probability distribution function of the finite element error. We also propose a self-consistent iterative algorithm to determine the smoothing parameter based on our theoretical analysis. Numerical examples are included to confirm the theoretical analysis and to show the competitive performance of the self- consistent algorithm for finding the smoothing parameter.

math.NA↗

arXiv2015

A theoretical framework for calibration in computer models: parametrization, estimation and convergence properties

Calibration parameters in deterministic computer experiments are those attributes that cannot be measured or available in physical experiments. Kennedy and O'Hagan \cite{kennedy2001bayesian} suggested an approach to estimate them by using data from physical experiments and computer simulations. A theoretical framework is given which allows us to study the issues of parameter identifiability and estimation. We define the $L_2$-consistency for calibration as a justification for calibration methods. It is shown that a simplified version of the original KO method leads to asymptotically $L_2$-inconsistent calibration. This $L_2$-inconsistency can be remedied by modifying the original estimation procedure. A novel calibration method, called the $L_2$ calibration, is proposed and proven to be $L_2$-consistent and enjoys optimal convergence rate. A numerical example and some mathematical analysis are used to illustrate the source of the $L_2$-inconsistency problem.

stat.ME↗

arXiv2015

Efficient Calibration for Imperfect Computer Models

Many computer models contain unknown parameters which need to be estimated using physical observations. Kennedy and O'Hagan (2001) shows that the calibration method based on Gaussian process models proposed by Kennedy and O'Hagan (2001) may lead to unreasonable estimate for imperfect computer models. In this work, we extend their study to calibration problems with stochastic physical data. We propose a novel method, called the $L_2$ calibration, and show its semiparametric efficiency. The conventional method of the ordinary least squares is also studied. Theoretical analysis shows that it is consistent but not efficient. Numerical examples show that the proposed method outperforms the existing ones.

stat.ME↗