arXiv subjects

Paz Fink Shustin

Publications and source records attributed to Paz Fink Shustin.

6 recordsLinked to original sources

arXiv2026

Nonlinear Dimensionality Reduction Techniques for Bayesian Optimization

Bayesian optimisation (BO) enables sample-efficient global optimisation of expensive black-box functions but remains challenging in high dimensions. We investigate nonlinear dimensionality reduction to a sequence of low-dimensional latent-space BO (LSBO) problems. Early LSBO used linear random and supervised embeddings; building on Grosnit et al., we employ variational autoencoders (VAEs), deep metric loss for structured latent manifolds, and retraining to adapt the encoder-decoder pair to newly sampled regions. We couple LSBO with sequential domain reduction (SDR) directly in latent space (SDR-LSBO), narrowing search domains as evidence accumulates. Implemented in GPU-accelerated BoTorch with Matérn-5/2 Gaussian-process surrogates, our methods improve benchmark optimisation quality, and retraining can enhance BO performance. Comparisons with adaptive supervised linear random embeddings demonstrate the effectiveness of VAE-based BO for nonlinear low-dimensional structures. We analyse BO-VAE with a fixed pretrained representation, decomposing ambient-space simple regret into latent BO error and a fixed VAE-induced representation gap. Under a PAC-Bayes-certified reconstruction condition and standard fixed-prior assumptions for expected improvement with a Matérn-5/2 kernel, latent BO error vanishes as the evaluation budget increases, whereas the representation gap remains fixed and may impose a non-vanishing error floor. Visualisations empirically assess accessibility of the ambient optimum through the learned decoder. To our knowledge, this is the first study combining SDR with VAE-based LSBO. Our analysis clarifies metric shaping and retraining choices critical for scalable latent-space BO. For reproducibility, source code is available at https://github.com/L-Lok/Nonlinear-Dimensionality-Reduction-Techniques-for-Bayesian-Optimization.git.

math.OC↗

arXiv2025

PCENet: High Dimensional Surrogate Modeling for Learning Uncertainty

Learning data representations under uncertainty is an important task that emerges in numerous scientific computing and data analysis applications. However, uncertainty quantification techniques are computationally intensive and become prohibitively expensive for high-dimensional data. In this study, we introduce a dimensionality reduction surrogate modeling (DRSM) approach for representation learning and uncertainty quantification that aims to deal with data of moderate to high dimensions. The approach involves a two-stage learning process: 1) employing a variational autoencoder to learn a low-dimensional representation of the input data distribution; and 2) harnessing polynomial chaos expansion (PCE) formulation to map the low dimensional distribution to the output target. The model enables us to (a) capture the system dynamics efficiently in the low-dimensional latent space, (b) learn under uncertainty, a representation of the data and a mapping between input and output distributions, (c) estimate this uncertainty in the high-dimensional data system, and (d) match high-order moments of the output distribution; without any prior statistical assumptions on the data. Numerical results are presented to illustrate the performance of the proposed method.

cs.LG↗

arXiv2024

Dimensionality Reduction Techniques for Global Bayesian Optimisation

Bayesian Optimisation (BO) is a state-of-the-art global optimisation technique for black-box problems where derivative information is unavailable, and sample efficiency is crucial. However, improving the general scalability of BO has proved challenging. Here, we explore Latent Space Bayesian Optimisation (LSBO), that applies dimensionality reduction to perform BO in a reduced-dimensional subspace. While early LSBO methods used (linear) random projections (Wang et al., 2013), we employ Variational Autoencoders (VAEs) to manage more complex data structures and general DR tasks. Building on Grosnit et. al. (2021), we analyse the VAE-based LSBO framework, focusing on VAE retraining and deep metric loss. We suggest a few key corrections in their implementation, originally designed for tasks such as molecule generation, and reformulate the algorithm for broader optimisation purposes. Our numerical results show that structured latent manifolds improve BO performance. Additionally, we examine the use of the Matérn-$\frac{5}{2}$ kernel for Gaussian Processes in this LSBO context. We also integrate Sequential Domain Reduction (SDR), a standard global optimization efficiency strategy, into BO. SDR is included in a GPU-based environment using \textit{BoTorch}, both in the original and VAE-generated latent spaces, marking the first application of SDR within LSBO.

math.OC↗

arXiv2021

Semi-Infinite Linear Regression and Its Applications

Finite linear least squares is one of the core problems of numerical linear algebra, with countless applications across science and engineering. Consequently, there is a rich and ongoing literature on algorithms for solving linear least squares problems. In this paper, we explore a variant in which the system's matrix has one infinite dimension (i.e., it is a quasimatrix). We call such problems semi-infinite linear regression problems. As we show, the semi-infinite case arises in several applications, such as supervised learning and function approximation, and allows for novel interpretations of existing algorithms. We explore semi-infinite linear regression rigorously and algorithmically. To that end, we give a formal framework for working with quasimatrices, and generalize several algorithms designed for the finite problem to the infinite case. Finally, we suggest the use of various sampling methods for obtaining an approximate solution.

math.NA↗

arXiv2021

Error Inhibiting Schemes for Initial Boundary Value Heat Equation

Finite Difference (FD) schemes are widely used in science and engineering for approximating solutions of partial differential equations (PDEs). Error analysis of FD schemes relies on estimating the truncation error at each time step. This approach usually leads to a global error whose order is of the same order of the truncation error. For classical FD schemes the global error is indeed of the same order as the truncation error. A particular class of FD schemes is the Block Finite Difference (BFD) schemes, in which the grid is divided into blocks. The structure of such schemes is similar to the structure of the Discontinuous Galerkin (DG) method, and allows inhabitation of the truncation errors. Recently, much effort was devoted to design BFD schemes whose global error converges faster than the truncation error. In this paper, we elaborate the approach presented in arXiv:1711.07926 for the heat equation with periodic boundary conditions. We generalize this methodology to design BFD schemes for the heat equation with Dirichlet or Neumann boundary conditions, whose global error converges faster than the truncation error. Such schemes are henceforth called Error Inhibiting Schemes. We provide an explicit error analysis, including proofs of stability and convergence of the proposed schemes. We illustrate our approach using several numerical examples, which demonstrate the efficiency of our method in comparison to standard FD schemes.

math.NA↗

arXiv2021

Gauss-Legendre Features for Gaussian Process Regression

Gaussian processes provide a powerful probabilistic kernel learning framework, which allows learning high quality nonparametric regression models via methods such as Gaussian process regression. Nevertheless, the learning phase of Gaussian process regression requires massive computations which are not realistic for large datasets. In this paper, we present a Gauss-Legendre quadrature based approach for scaling up Gaussian process regression via a low rank approximation of the kernel matrix. We utilize the structure of the low rank approximation to achieve effective hyperparameter learning, training and prediction. Our method is very much inspired by the well-known random Fourier features approach, which also builds low-rank approximations via numerical integration. However, our method is capable of generating high quality approximation to the kernel using an amount of features which is poly-logarithmic in the number of training points, while similar guarantees will require an amount that is at the very least linear in the number of training points when random Fourier features. Furthermore, the structure of the low-rank approximation that our method builds is subtly different from the one generated by random Fourier features, and this enables much more efficient hyperparameter learning. The utility of our method for learning with low-dimensional datasets is demonstrated using numerical experiments.

math.NA↗