arXiv subjects

Nai-Yuan Chiang

Publications and source records attributed to Nai-Yuan Chiang.

7 recordsLinked to original sources

arXiv2026

Practical Efficient Global Optimization is No-regret

Efficient global optimization (EGO) is one of the most widely used noise-free Bayesian optimization algorithms.It comprises the Gaussian process (GP) surrogate model and expected improvement (EI) acquisition function. In practice, when EGO is applied, a scalar matrix of a small positive value (also called a nugget or jitter) is usually added to the covariance matrix of the deterministic GP to improve numerical stability. We refer to this EGO with a positive nugget as the practical EGO. Despite its wide adoption and empirical success, to date, cumulative regret bounds for practical EGO have yet to be established. In this paper, we present for the first time the cumulative regret upper bound of practical EGO. In particular, we show that practical EGO has sublinear cumulative regret bounds and thus is a no-regret algorithm for commonly used kernels including the squared exponential (SE) and Matérn kernels ($ν>\frac{1}{2}$). Moreover, we analyze the effect of the nugget on the regret bound and discuss the theoretical implication on its choice. Numerical experiments are conducted to support and validate our findings.

stat.ML↗

arXiv2026

Convergence Rates of Constrained Expected Improvement

Constrained Bayesian optimization (CBO) methods have seen significant success in black-box optimization with constraints. One of the most commonly used CBO methods is the constrained expected improvement (CEI) algorithm. CEI is a natural extension of expected improvement (EI) when constraints are incorporated. However, the theoretical convergence rate of CEI has not been established. In this work, we study the convergence rate of CEI by analyzing its simple regret upper bound. First, we show that when the objective function $f$ and constraint function $c$ are assumed to each lie in a reproducing kernel Hilbert space (RKHS), CEI achieves the convergence rates of $\mathcal{O} \left(t^{-\frac{1}{2}}\log^{\frac{d+1}{2}}(t) \right) \ \text{and }\ \mathcal{O}\left(t^{\frac{-ν}{2ν+d}} \log^{\fracν{2ν+d}}(t)\right)$ for the commonly used squared exponential and Matérn kernels ($ν>\frac{1}{2}$), respectively. Second, we show that when $f$ is assumed to be sampled from Gaussian processes (GPs), CEI achieves similar convergence rates with a high probability. Numerical experiments are performed to validate the theoretical analysis.

stat.ML↗

arXiv2025

A Constrained Multi-Fidelity Bayesian Optimization Method

Recently, multi-fidelity Bayesian optimization (MFBO) has been successfully applied to many engineering design optimization problems, where the cost of high-fidelity simulations and experiments can be prohibitive. However, challenges remain for constrained optimization problems using the MFBO framework, particularly in efficiently identifying the feasible region defined by the constraints. In this paper, we propose a constrained multi-fidelity Bayesian optimization (CMFBO) method with novel acquisition functions. Specifically, we design efficient acquisition functions that 1) have analytically closed-form expressions; 2) are straightforward to implement; and 3) do not require feasible initial samples, an important feature often missing in commonly used acquisition functions such as expected constrained improvement (ECI). We demonstrate the effectiveness of our algorithms on synthetic test problems using different combinations of acquisition functions. Then, we apply the proposed method to a data-driven inertial confinement fusion (ICF) design problem, and a high-current joint design problem using finite element simulations with computational contact mechanics.

math.NA↗

arXiv2025

On the convergence rate of noisy Bayesian Optimization with Expected Improvement

Expected improvement (EI) is one of the most widely used acquisition functions in Bayesian optimization (BO). Despite its proven success in applications for decades, important open questions remain on the theoretical convergence behaviors and rates for EI. In this paper, we contribute to the convergence theory of EI in three novel and critical areas. First, we consider objective functions that fit under the Gaussian process (GP) prior assumption, whereas existing works mostly focus on functions in the reproducing kernel Hilbert space (RKHS). Second, we establish for the first time the asymptotic error bound and its corresponding rate for GP-EI with noisy observations under the GP prior assumption. Third, by investigating the exploration and exploitation properties of the non-convex EI function, we establish improved error bounds of GP-EI for both the noise-free and noisy cases.

stat.ML↗

arXiv2024

On Improved Regret Bounds In Bayesian Optimization with Gaussian Noise

Bayesian optimization (BO) with Gaussian process (GP) surrogate models is a powerful black-box optimization method. Acquisition functions are a critical part of a BO algorithm as they determine how the new samples are selected. Some of the most widely used acquisition functions include upper confidence bound (UCB) and Thompson sampling (TS). The convergence analysis of BO algorithms has focused on the cumulative regret under both the Bayesian and frequentist settings for the objective. In this paper, we establish new pointwise bounds on the prediction error of GP under the frequentist setting with Gaussian noise. Consequently, we prove improved convergence rates of cumulative regret bound for both GP-UCB and GP-TS. Of note, the new prediction error bound under Gaussian noise can be applied to general BO algorithms and convergence analysis, e.g., the asymptotic convergence of expected improvement (EI) with noise.

cs.LG↗

arXiv2021

A Hybrid Direct-Iterative Method for Solving KKT Linear Systems

We propose a solution strategy for linear systems arising in interior method optimization, which is suitable for implementation on hardware accelerators such as graphical processing units (GPUs). The current gold standard for solving these systems is the LDL^T factorization. However, LDL^T requires pivoting during factorization, which substantially increases communication cost and degrades performance on GPUs. Our novel approach solves a large indefinite system by solving multiple smaller positive definite systems, using an iterative solve for the Schur complement and an inner direct solve (via Cholesky factorization) within each iteration. Cholesky is stable without pivoting, thereby reducing communication and allowing reuse of the symbolic factorization. We demonstrate the practicality of our approach and show that on large systems it can efficiently utilize GPUs and outperform LDL^T factorization of the full system.

math.OC↗

arXiv2019

Efficient Propagation of Uncertainties in Manufacturing Supply Chains: Time Buckets, L-leap and Multilevel Monte Carlo

Uncertainty propagation of large scale discrete supply chains can be prohibitive when a large number of events occur during the simulated period and discrete event simulations (DES) are costly. We present a time bucket method to approximate and accelerate the DES of supply chains. Its stochastic version, which we call the L(logistic)-leap method, can be viewed as an extension of the leap methods, e.g., tau-leap, D-leap, developed in the chemical engineering community for the acceleration of stochastic DES of chemical reactions. The L-leap method instantaneously updates the system state vector at discrete time points and the production rates and policies of a supply chain are assumed to be stationary during each time bucket. We propose to use Multilevel Monte Carlo (MLMC) to efficiently propagate the uncertainties in a supply chain network, where the levels are naturally defined by the sizes of the time buckets of the simulations. We demonstrate the efficiency and accuracy of our methods using four numerical examples derived from a real world manufacturing material flow. In these examples, our multilevel L-leap approach can be faster than the standard Monte Carlo (MC) method by one or two orders of magnitudes without compromising the accuracy.

stat.CO↗