Function Optimization with Posterior Gaussian Derivative Process

Abstract

We propose a Bayesian optimization algorithm for differentiable functions with available first and second partial derivatives. The objective function is modeled as a Gaussian process, inducing a Gaussian derivative process. Given initial function evaluations, we obtain the posterior of the derivative process and construct a posterior for stationary points by setting the derivative to zero. A recursive importance-resampling scheme with prior constraints on the gradient norm and Hessian definiteness drives the algorithm toward the true optima. We prove almost sure convergence as the number of stages tends to infinity, relying on novel results establishing almost sure uniform convergence of the Gaussian process and its derivative process posteriors to the true function and its derivatives under fixed-domain infill asymptotics; rates of convergence are also derived. We also give a Bayesian characterization of the number of optima. The method is demonstrated on five diverse problems, including finding maxima, minima, saddle points, and inconclusive cases, ranging from one-dimensional to 100-dimensional nonlinear least-squares. On a real-world Poisson regression (AIDS deaths data), our procedure achieves a substantially smaller gradient norm at the MLE than Fisher scoring, BFGS, simulated annealing, and multi-start quasi-Newton. The posterior simulation nature allows exploration of neighborhoods of solutions from other methods, yielding more accurate results. Code is available at https://github.com/Sourabh-Bhattacharya/FUNCTION_OPT_GDP.