arXiv subjects

Olivier Zahm

Publications and source records attributed to Olivier Zahm.

At least 19 recordsLinked to original sources

arXiv2026

Certified Bayesian optimal experimental design: from Expected Information Gain to Signal-to-Noise Ratio

Optimal experimental design is often formulated as the maximization of the Expected Information Gain (EIG), which measures the expected reduction in uncertainty about a parameter of interest after observing the data. For nonlinear forward models and nonGaussian priors, computing the EIG typically requires costly nested Monte Carlo estimators or sophisticated density-approximation techniques. In this work, we derive new upper and lower bounds on the EIG using dimensional logarithmic Sobolev inequalities. The proposed bounds are expressed in terms of signal-to-noise ratios (SNR), preserving the simple matrix-based structure of the EIG in the linear-Gaussian setting and depending solely on covariance matrices and Fisher information matrices. We introduce two families of bounds: conservative bounds, designed to retain the entire information content, and incremental bounds, that prioritize early information acquisition for sequential experimental design. When model gradients are available, we further derive computable approximations of the Fisher-information terms that require only a small number of gradient evaluations, yielding substantial computational savings. The tightness of these bounds is controlled by the nonlinearity of the forward model, and in the linear-Gaussian setting, these bounds recover the exact value of the EIG. By eliminating the nested-sampling bottleneck, the proposed framework offers a tractable and theoretically grounded alternative to direct EIG estimation for large-scale Bayesian experimental design, while retaining the interpretability of the SNR-based bounds.

cs.IT↗

arXiv2026

Gradient-free Riemannian Langevin Sampler

We address the problem of efficiently sampling multimodal probability distributions, where standard Markov Chain Monte Carlo methods often suffer from poor mixing and mode trapping. To mitigate these issues, we propose Gradient-free Riemannian Langevin Sampler (GRiLS), a novel proposal that improves exploration without requiring gradient evaluations of the target density. Our approach introduces a Riemannian metric which reshapes the local geometry in order to facilitate transitions across modes. The resulting gradient-free MCMC algorithm is particularly suitable for complex, computationally expensive targets where derivatives are unavailable or impractical. The GRiLS proposal requires knowing the mean and covariance of the target density, which we estimate using an ensemble of interacting particles. Empirical results on multimodal benchmarks demonstrate that GRiLS achieves improved mixing compared to existing gradient-based and gradient-free MCMC approaches.

cs.LG↗

arXiv2026

A new gradient-free active subspace estimation method with application to rare event probability estimation

To reduce the cost of estimating the probability of a rare event involving a very large number of random parameters, we propose a new strategy for dimension reduction coupled with a surrogate model for the expensive part of the algorithm. To this end, we extend the Ordinary Kriging Active Subspace (OK-AS) method into a sequential version. Our approach consists of iteratively re-estimating the active subspace using a Kriging surrogate trained in a rotated coordinate system until the active subspace stabilises. This method allows for a reduction in prediction error and a better approximation of the active subspace on a benchmark of test problems. Furthermore, we integrate our algorithm into an efficient pre-existing approach for estimating the probability of a rare event. This approach is based on learning the active subspace associated with the random event whose probability is to be estimated. The sequential learning of an importance sampling density is necessary and corresponds to the expensive part of this strategy. To circumvent this issue, we integrate our sequential OK-AS version into the estimation of the importance sampling density. The numerical results indicate that our method allows for reducing the cost required to obtain a precise estimate of the rare event probability.

stat.CO↗

arXiv2025

Sharp detection of low-dimensional structure in probability measures via dimensional logarithmic Sobolev inequalities

Identifying low-dimensional structure in high-dimensional probability measures is an essential pre-processing step for efficient sampling. We introduce a method for identifying and approximating a target measure $π$ as a perturbation of a given reference measure $μ$ along a few significant directions of $\mathbb{R}^{d}$. The reference measure can be a Gaussian or a nonlinear transformation of a Gaussian, as commonly arising in generative modeling. Our method extends prior work on minimizing majorizations of the Kullback--Leibler divergence to identify optimal approximations within this class of measures. Our main contribution unveils a connection between the \emph{dimensional} logarithmic Sobolev inequality (LSI) and approximations with this ansatz. Specifically, when the target and reference are both Gaussian, we show that minimizing the dimensional LSI is equivalent to minimizing the KL divergence restricted to this ansatz. For general non-Gaussian measures, the dimensional LSI produces majorants that uniformly improve on previous majorants for gradient-based dimension reduction. We further demonstrate the applicability of this analysis to the squared Hellinger distance, where analogous reasoning shows that the dimensional Poincaré inequality offers improved bounds.

stat.ML↗

arXiv2025

Coupled Input-Output Dimension Reduction: Application to Goal-oriented Bayesian Experimental Design and Global Sensitivity Analysis

We introduce a new method to jointly reduce the dimension of the input and output space of a function between high-dimensional spaces. Choosing a reduced input subspace influences which output subspace is relevant and vice versa. Conventional methods focus on reducing either the input or output space, even though both are often reduced simultaneously in practice. Our coupled approach naturally supports goal-oriented dimension reduction, where either an input or output quantity of interest is prescribed. We consider, in particular, goal-oriented sensor placement and goal-oriented sensitivity analysis, which can be viewed as dimension reduction where the most important output or, respectively, input components are chosen. Both applications present difficult combinatorial optimization problems with expensive objectives such as the expected information gain and Sobol' indices. By optimizing gradient-based bounds, we can determine the most informative sensors and most influential parameters as the largest diagonal entries of some diagnostic matrices, thus bypassing the combinatorial optimization and objective evaluation.

stat.ML↗

arXiv2025

Optimal Riemannian metric for Poincaré inequalities and how to ideally precondition Langevin dynamics

Poincaré inequality is a fundamental property that rises naturally in different branches of mathematics. The associated Poincaré constant plays a central role in many applications since it governs the convergence of various practical algorithms. For instance, the convergence rate of the Langevin dynamics is exactly given by the Poincaré constant. This paper investigates a Riemannian version of Poincaré inequality where a positive definite weighting matrix field (\emph{i.e.} a Riemannian metric) is introduced to improve the Poincaré constant, and therefore the performances of the associated algorithm. Assuming the underlying measure is a \emph{moment measure}, we show that an optimal metric exists and the resulting Poincaré constant is 1. We demonstrate that such optimal metric is necessarily a \emph{Stein kernel}, offering a novel perspective on these complex but central mathematical objects that are hard to obtain in practice. We further discuss how to numerically obtain the optimal metric by deriving an implementable optimization algorithm. The resulting method is illustrated in a few simple but nontrivial examples, where solutions are revealed to be rather sophisticated. We also demonstrate how to design efficient Langevin-based sampling schemes by utilizing the precomputed optimal metric as a preconditioner.

math.PR↗

arXiv2024

Sequential transport maps using SoS density estimation and $α$-divergences

Transport-based density estimation methods are receiving growing interest because of their ability to efficiently generate samples from the approximated density. We further invertigate the sequential transport maps framework proposed from arXiv:2106.04170 arXiv:2303.02554, which builds on a sequence of composed Knothe-Rosenblatt (KR) maps. Each of those maps are built by first estimating an intermediate density of moderate complexity, and then by computing the exact KR map from a reference density to the precomputed approximate density. In our work, we explore the use of Sum-of-Squares (SoS) densities and $α$-divergences for approximating the intermediate densities. Combining SoS densities with $α$-divergence interestingly yields convex optimization problems which can be efficiently solved using semidefinite programming. The main advantage of $α$-divergences is to enable working with unnormalized densities, which provides benefits both numerically and theoretically. In particular, we provide a new convergence analyses of the sequential transport maps based on information geometric properties of $α$-divergences. The choice of intermediate densities is also crucial for the efficiency of the method. While tempered (or annealed) densities are the state-of-the-art, we introduce diffusion-based intermediate densities which permits to approximate densities known from samples only. Such intermediate densities are well-established in machine learning for generative modeling. Finally we propose low-dimensional maps (or lazy maps) for dealing with high-dimensional problems and numerically demonstrate our methods on Bayesian inference problems and unsupervised learning tasks.

stat.ML↗

arXiv2024

Continuous Gaussian mixture solution for linear Bayesian inversion with application to Laplace priors

We focus on Bayesian inverse problems with Gaussian likelihood, linear forward model, and priors that can be formulated as a Gaussian mixture. Such a mixture is expressed as an integral of Gaussian density functions weighted by a mixing density over the mixing variables. Within this framework, the corresponding posterior distribution also takes the form of a Gaussian mixture, and we derive the closed-form expression for its posterior mixing density. To sample from the posterior Gaussian mixture, we propose a two-step sampling method. First, we sample the mixture variables from the posterior mixing density, and then we sample the variables of interest from Gaussian densities conditioned on the sampled mixing variables. However, the posterior mixing density is relatively difficult to sample from, especially in high dimensions. Therefore, we propose to replace the posterior mixing density by a dimension-reduced approximation, and we provide a bound in the Hellinger distance for the resulting approximate posterior. We apply the proposed approach to a posterior with Laplace prior, where we introduce two dimension-reduced approximations for the posterior mixing density. Our numerical experiments indicate that samples generated via the proposed approximations have very low correlation and are close to the exact posterior.

stat.CO↗

arXiv2024

On the representation and learning of monotone triangular transport maps

Transportation of measure provides a versatile approach for modeling complex probability distributions, with applications in density estimation, Bayesian inference, generative modeling, and beyond. Monotone triangular transport maps$\unicode{x2014}$approximations of the Knothe$\unicode{x2013}$Rosenblatt (KR) rearrangement$\unicode{x2014}$are a canonical choice for these tasks. Yet the representation and parameterization of such maps have a significant impact on their generality and expressiveness, and on properties of the optimization problem that arises in learning a map from data (e.g., via maximum likelihood estimation). We present a general framework for representing monotone triangular maps via invertible transformations of smooth functions. We establish conditions on the transformation such that the associated infinite-dimensional minimization problem has no spurious local minima, i.e., all local minima are global minima; and we show for target distributions satisfying certain tail conditions that the unique global minimizer corresponds to the KR map. Given a sample from the target, we then propose an adaptive algorithm that estimates a sparse semi-parametric approximation of the underlying KR map. We demonstrate how this framework can be applied to joint and conditional density estimation, likelihood-free inference, and structure learning of directed graphical models, with stable generalization performance across a range of sample sizes.

stat.ML↗

arXiv2024

Principal Feature Detection via $Φ$-Sobolev Inequalities

We investigate the approximation of high-dimensional target measures as low-dimensional updates of a dominating reference measure. This approximation class replaces the associated density with the composition of: (i) a feature map that identifies the leading principal components or features of the target measure, relative to the reference, and (ii) a low-dimensional profile function. When the reference measure satisfies a subspace $ϕ$-Sobolev inequality, we construct a computationally tractable approximation that yields certifiable error guarantees with respect to the Amari $α$-divergences. Our construction proceeds in two stages. First, for any feature map and any $α$-divergence, we obtain an analytical expression for the optimal profile function. Second, for linear feature maps, the principal features are obtained from eigenvectors of a matrix involving gradients of the log-density. Neither step requires explicit access to normalizing constants. Notably, by leveraging the $ϕ$-Sobolev inequalities, we demonstrate that these features universally certify approximation errors across the range of $α$-divergences $α\in (0,1]$. We then propose an application to Bayesian inverse problems and provide an analogous construction with approximation guarantees that hold in expectation over the data. We conclude with an extension of the proposed dimension reduction strategy to nonlinear feature maps.

stat.CO↗

arXiv2023

Self-reinforced polynomial approximation methods for concentrated probability densities

Transport map methods offer a powerful statistical learning tool that can couple a target high-dimensional random variable with some reference random variable using invertible transformations. This paper presents new computational techniques for building the Knothe--Rosenblatt (KR) rearrangement based on general separable functions. We first introduce a new construction of the KR rearrangement -- with guaranteed invertibility in its numerical implementation -- based on approximating the density of the target random variable using tensor-product spectral polynomials and downward closed sparse index sets. Compared to other constructions of KR arrangements based on either multi-linear approximations or nonlinear optimizations, our new construction only relies on a weighted least square approximation procedure. Then, inspired by the recently developed deep tensor trains (Cui and Dolgov, Found. Comput. Math. 22:1863--1922, 2022), we enhance the approximation power of sparse polynomials by preconditioning the density approximation problem using compositions of maps. This is particularly suitable for high-dimensional and concentrated probability densities commonly seen in many applications. We approximate the complicated target density by a composition of self-reinforced KR rearrangements, in which previously constructed KR rearrangements -- based on the same approximation ansatz -- are used to precondition the density approximation problem for building each new KR rearrangement. We demonstrate the efficiency of our proposed methods and the importance of using the composite map on several inverse problems governed by ordinary differential equations (ODEs) and partial differential equations (PDEs).

math.NA↗

arXiv2023

Prior normalization for certified likelihood-informed subspace detection of Bayesian inverse problems

Markov chain Monte Carlo (MCMC) methods form one of the algorithmic foundations of Bayesian inverse problems. The recent development of likelihood-informed subspace (LIS) methods offers a viable route to designing efficient MCMC methods for exploring high-dimensional posterior distributions via exploiting the intrinsic low-dimensional structure of the underlying inverse problem. However, existing LIS methods and the associated performance analysis often assume that the prior distribution is Gaussian. This assumption is limited for inverse problems aiming to promote sparsity in the parameter estimation, as heavy-tailed priors, e.g., Laplace distribution or the elastic net commonly used in Bayesian LASSO, are often needed in this case. To overcome this limitation, we consider a prior normalization technique that transforms any non-Gaussian (e.g. heavy-tailed) priors into standard Gaussian distributions, which makes it possible to implement LIS methods to accelerate MCMC sampling via such transformations. We also rigorously investigate the integration of such transformations with several MCMC methods for high-dimensional problems. Finally, we demonstrate various aspects of our theoretical claims on two nonlinear inverse problems.

math.NA↗

arXiv2023

Scalable conditional deep inverse Rosenblatt transports using tensor-trains and gradient-based dimension reduction

We present a novel offline-online method to mitigate the computational burden of the characterization of posterior random variables in statistical learning. In the offline phase, the proposed method learns the joint law of the parameter random variables and the observable random variables in the tensor-train (TT) format. In the online phase, the resulting order-preserving conditional transport can characterize the posterior random variables given newly observed data in real time. Compared with the state-of-the-art normalizing flow techniques, the proposed method relies on function approximation and is equipped with a thorough performance analysis. The function approximation perspective also allows us to further extend the capability of transport maps in challenging problems with high-dimensional observations and high-dimensional parameters. On the one hand, we present novel heuristics to reorder and/or reparametrize the variables to enhance the approximation power of TT. On the other hand, we integrate the TT-based transport maps and the parameter reordering/reparametrization into layered compositions to further improve the performance of the resulting transport maps. We demonstrate the efficiency of the proposed method on various statistical learning tasks in ordinary differential equations (ODEs) and partial differential equations (PDEs).

stat.ML↗

arXiv2023

Learning non-Gaussian graphical models via Hessian scores and triangular transport

Undirected probabilistic graphical models represent the conditional dependencies, or Markov properties, of a collection of random variables. Knowing the sparsity of such a graphical model is valuable for modeling multivariate distributions and for efficiently performing inference. While the problem of learning graph structure from data has been studied extensively for certain parametric families of distributions, most existing methods fail to consistently recover the graph structure for non-Gaussian data. Here we propose an algorithm for learning the Markov structure of continuous and non-Gaussian distributions. To characterize conditional independence, we introduce a score based on integrated Hessian information from the joint log-density, and we prove that this score upper bounds the conditional mutual information for a general class of distributions. To compute the score, our algorithm SING estimates the density using a deterministic coupling, induced by a triangular transport map, and iteratively exploits sparse structure in the map to reveal sparsity in the graph. For certain non-Gaussian datasets, we show that our algorithm recovers the graph structure even with a biased approximation to the density. Among other examples, we apply SING to learn the dependencies between the states of a chaotic dynamical system with local interactions.

stat.ML↗

arXiv2022

Gradient-based data and parameter dimension reduction for Bayesian models: an information theoretic perspective

We consider the problem of reducing the dimensions of parameters and data in non-Gaussian Bayesian inference problems. Our goal is to identify an "informed" subspace of the parameters and an "informative" subspace of the data so that a high-dimensional inference problem can be approximately reformulated in low-to-moderate dimensions, thereby improving the computational efficiency of many inference techniques. To do so, we exploit gradient evaluations of the log-likelihood function. Furthermore, we use an information-theoretic analysis to derive a bound on the posterior error due to parameter and data dimension reduction. This bound relies on logarithmic Sobolev inequalities, and it reveals the appropriate dimensions of the reduced variables. We compare our method with classical dimension reduction techniques, such as principal component analysis and canonical correlation analysis, on applications ranging from mechanics to image processing.

stat.CO↗

arXiv2022

Nonlinear dimension reduction for surrogate modeling using gradient information

We introduce a method for the nonlinear dimension reduction of a high-dimensional function $u:\mathbb{R}^d\rightarrow\mathbb{R}$, $d\gg1$. Our objective is to identify a nonlinear feature map $g:\mathbb{R}^d\rightarrow\mathbb{R}^m$, with a prescribed intermediate dimension $m\ll d$, so that $u$ can be well approximated by $f\circ g$ for some profile function $f:\mathbb{R}^m\rightarrow\mathbb{R}$. We propose to build the feature map by aligning the Jacobian $\nabla g$ with the gradient $\nabla u$, and we theoretically analyze the properties of the resulting $g$. Once $g$ is built, we construct $f$ by solving a gradient-enhanced least squares problem. Our practical algorithm makes use of a sample $\{x^{(i)},u(x^{(i)}),\nabla u(x^{(i)})\}_{i=1}^N$ and builds both $g$ and $f$ on adaptive downward-closed polynomial spaces, using cross validation to avoid overfitting. We numerically evaluate the performance of our algorithm across different benchmarks, and explore the impact of the intermediate dimension $m$. We show that building a nonlinear feature map $g$ can permit more accurate approximation of $u$ than a linear $g$, for the same input data set.

math.NA↗

arXiv2022

Certified dimension reduction in nonlinear Bayesian inverse problems

We propose a dimension reduction technique for Bayesian inverse problems with nonlinear forward operators, non-Gaussian priors, and non-Gaussian observation noise. The likelihood function is approximated by a ridge function, i.e., a map which depends non-trivially only on a few linear combinations of the parameters. We build this ridge approximation by minimizing an upper bound on the Kullback--Leibler divergence between the posterior distribution and its approximation. This bound, obtained via logarithmic Sobolev inequalities, allows one to certify the error of the posterior approximation. Computing the bound requires computing the second moment matrix of the gradient of the log-likelihood function. In practice, a sample-based approximation of the upper bound is then required. We provide an analysis that enables control of the posterior approximation error due to this sampling. Numerical and theoretical comparisons with existing methods illustrate the benefits of the proposed methodology.

math.PR↗

arXiv2021

Data-Free Likelihood-Informed Dimension Reduction of Bayesian Inverse Problems

Identifying a low-dimensional informed parameter subspace offers a viable path to alleviating the dimensionality challenge in the sampled-based solution to large-scale Bayesian inverse problems. This paper introduces a novel gradient-based dimension reduction method in which the informed subspace does not depend on the data. This permits an online-offline computational strategy where the expensive low-dimensional structure of the problem is detected in an offline phase, meaning before observing the data. This strategy is particularly relevant for multiple inversion problems as the same informed subspace can be reused. The proposed approach allows controlling the approximation error (in expectation over the data) of the posterior distribution. We also present sampling strategies that exploit the informed subspace to draw efficiently samples from the exact posterior distribution. The method is successfully illustrated on two numerical examples: a PDE-based inverse problem with a Gaussian process prior and a tomography problem with Poisson data and a Besov-$\mathcal{B}^2_{11}$ prior.

stat.CO↗