arXiv subjects

Ilja Klebanov

Publications and source records attributed to Ilja Klebanov.

At least 19 recordsLinked to original sources

arXiv2026

Verifiable Regularity Criterion for Conditional Expectation Operators and Conditional Mean Embeddings with Applications to Nonparametric Regression, Bayesian Inverse Problems, and Koopman Operators

Conditional expectation operators (CEOs) and their associated conditional mean embeddings (CMEs) play a central role across applied mathematics and machine learning, appearing in nonparametric regression, Bayesian inverse problems, and Koopman operator theory. A fundamental question is when a CEO maps a function space on $\mathcal{Y}$ into a prescribed function space on $\mathcal{X}$, particularly a reproducing kernel Hilbert space (RKHS). We show that such mapping properties are characterized by the regularity of the Radon--Nikodym density of the conditional law, and establish a simple, verifiable sufficient condition under which the CEO is bounded and Hilbert--Schmidt. For RKHSs norm-equivalent to Sobolev spaces, this condition reduces to Sobolev regularity of the conditional density. The result yields a direct route to validate CME representations and error bounds for Galerkin-type and CME-based estimators. We verify the regularity condition in three settings: nonparametric regression, Bayesian inverse problems, and Koopman operator theory for stochastic dynamical systems. We show in each case that classical regularity results on the underlying probabilistic model imply the required mapping properties. The resulting framework offers a unified perspective on conditional expectation operators across probability, operator theory, kernel methods, and stochastic dynamics.

math.DS↗

arXiv2026

Lattice Rules Meet Kernel Cubature

Rank-1 lattice rules are a class of equally weighted quasi-Monte Carlo methods that achieve essentially linear convergence rates for functions in a reproducing kernel Hilbert space (RKHS) characterized by square-integrable first-order mixed partial derivatives. In this work, we explore the impact of replacing the equal weights in lattice rules with optimized cubature weights derived using the reproducing kernel. We establish a theoretical result demonstrating a doubled convergence rate in the one-dimensional case and provide numerical investigations of convergence rates in higher dimensions. We also present numerical results for an uncertainty quantification problem involving an elliptic partial differential equation with a random coefficient.

math.NA↗

arXiv2026

Mixture-Weighted Ensemble Kalman Filter with Quasi-Monte Carlo Transport

The Bootstrap Particle Filter (BPF) and the Ensemble Kalman Filter (EnKF) are two widely used methods for sequential Bayesian filtering: the BPF is asymptotically exact but can suffer from weight degeneracy, while the EnKF scales well in high dimension (typically with localization) yet is exact only in the linear-Gaussian case. We combine these approaches by retaining the EnKF transport step and adding a principled importance-sampling correction. Our first contribution is a general importance-sampling theory for mixture targets and proposals, including variance comparisons between individual- and mixture-based estimators. We then interpret the stochastic EnKF analysis as sampling from explicit Gaussian-mixture proposals obtained by conditioning on the current or previous ensemble, which leads to six self-normalized IS-EnKF schemes. We embed these updates into a broader class of ensemble-based filters and prove consistency and error bounds, including weight-variance comparisons and sufficient conditions ensuring finite-variance importance weights. As a second contribution, we construct transported quasi-Monte Carlo (TQMC) point sets for the Gaussian-mixture laws arising in prediction and analysis, yielding TQMC-enhanced variants that can substantially reduce sampling error without changing the filtering pipeline. Numerical experiments on benchmark models compare the proposed mixture-weighted and TQMC-enhanced filters, showing improved filtering accuracy relative to BPF, EnKF, and the standard weighted EnKF, and that the weighted schemes eliminate the EnKF error plateau often caused by analysis-target mismatch.

stat.ME↗

arXiv2026

Error Bounds for Importance Sampling with Estimated Proposal Distributions

Importance sampling with data-driven proposal distributions is widely used in practice. A common workflow first generates an auxiliary sample of size $N$ from an approximation of the target distribution, constructs a density estimate $\hat q$ such as a kernel density estimator (KDE), and then draws $n$ importance samples from this learned proposal. Despite its practical relevance, the theoretical properties of this hierarchical procedure remain poorly understood, since classical importance sampling theory assumes a fixed proposal. We address this gap by deriving non-asymptotic error bounds for standard, defensive, and self-normalized importance sampling estimators with random proposals. Our results separate the Monte Carlo error, scaling as $n^{-1/2}$, from the proposal approximation error measured through the mean integrated absolute and squared errors (MIAE and MISE) of $\hat q$. To obtain explicit convergence rates in $(N,n)$, we establish MIAE and MISE bounds for KDEs constructed from geometrically ergodic Markov chains in stationary and non-stationary regimes. Combining these results yields quantitative guarantees for importance sampling with KDE-based proposals. Our theory provides practical guidance for selecting defensive mixture weights in a nonparametric importance sampling framework.

math.ST↗

arXiv2025

Classification of small-ball modes and maximum a posteriori estimators in metric spaces

A mode, or `most likely point', for a probability measure $μ$ can be defined in various ways via the asymptotic behaviour of the $μ$-mass of balls as their radius tends to zero. Such points are of intrinsic interest in the local theory of measures on metric spaces and also arise naturally in the study of Bayesian inverse problems and diffusion processes. Building upon special cases already proposed in the literature, this paper develops a systematic framework for defining modes through small-ball probabilities. We propose `common-sense' axioms that such definitions should obey, including appropriate treatment of discrete and absolutely continuous measures, as well as symmetry and invariance properties. We show that there are exactly ten such definitions consistent with these axioms, and that they are partially but not totally ordered in strength, forming a complete, distributive lattice. We also show how this classification simplifies for well-behaved $μ$.

math.PR↗

arXiv2024

Deterministic Fokker-Planck Transport -- With Applications to Sampling, Variational Inference, Kernel Mean Embeddings & Sequential Monte Carlo

The Fokker-Planck equation can be reformulated as a continuity equation, which naturally suggests using the associated velocity field in particle flow methods. While the resulting probability flow ODE offers appealing properties - such as defining a gradient flow of the Kullback-Leibler divergence between the current and target densities with respect to the 2-Wasserstein distance - it relies on evaluating the current probability density, which is intractable in most practical applications. By closely examining the drawbacks of approximating this density via kernel density estimation, we uncover opportunities to turn these limitations into advantages in contexts such as variational inference, kernel mean embeddings, and sequential Monte Carlo.

stat.ML↗

arXiv2024

Coherent set identification via direct low rank maximum likelihood estimation

We analyze connections between two low rank modeling approaches from the last decade for treating dynamical data. The first one is the coherence problem (or coherent set approach), where groups of states are sought that evolve under the action of a stochastic transition matrix in a way maximally distinguishable from other groups. The second one is a low rank factorization approach for stochastic matrices, called Direct Bayesian Model Reduction (DBMR), which estimates the low rank factors directly from observed data. We show that DBMR results in a low rank model that is a projection of the full model, and exploit this insight to infer bounds on a quantitative measure of coherence within the reduced model. Both approaches can be formulated as optimization problems, and we also prove a bound between their respective objectives. On a broader scope, this work relates the two classical loss functions of nonnegative matrix factorization, namely the Frobenius norm and the generalized Kullback--Leibler divergence, and suggests new links between likelihood-based and projection-based estimation of probabilistic models.

cs.IT↗

arXiv2024

Graph Convex Hull Bounds as generalized Jensen Inequalities

Jensen's inequality is ubiquitous in measure and probability theory, statistics, machine learning, information theory and many other areas of mathematics and data science. It states that, for any convex function $f\colon K \to \mathbb{R}$ defined on a convex domain $K \subseteq \mathbb{R}^{d}$ and any random variable $X$ taking values in $K$, $\mathbb{E}[f(X)] \geq f(\mathbb{E}[X])$. In this paper, sharp upper and lower bounds on $\mathbb{E}[f(X)]$, termed ``graph convex hull bounds'', are derived for arbitrary functions $f$ on arbitrary domains $K$, thereby extensively generalizing Jensen's inequality. The derivation of these bounds necessitates the investigation of the convex hull of the graph of $f$, which can be challenging for complex functions. On the other hand, once these inequalities are established, they hold, just like Jensen's inequality, for \emph{any} $K$-valued random variable $X$. Therefore, these bounds are of particular interest in cases where $f$ is relatively simple and $X$ is complicated or unknown. Both finite- and infinite-dimensional domains and codomains of $f$ are covered as well as analogous bounds for conditional expectations and Markov operators.

math.ST↗

arXiv2023

Transporting Higher-Order Quadrature Rules: Quasi-Monte Carlo Points and Sparse Grids for Mixture Distributions

Integration against, and hence sampling from, high-dimensional probability distributions is of essential importance in many application areas and has been an active research area for decades. One approach that has drawn increasing attention in recent years has been the generation of samples from a target distribution $\mathbb{P}_{\mathrm{tar}}$ using transport maps: if $\mathbb{P}_{\mathrm{tar}} = T_\# \mathbb{P}_{\mathrm{ref}}$ is the pushforward of an easily-sampled probability distribution $\mathbb{P}_{\mathrm{ref}}$ under the transport map $T$, then the application of $T$ to $\mathbb{P}_{\mathrm{ref}}$-distributed samples yields $\mathbb{P}_{\mathrm{tar}}$-distributed samples. This paper proposes the application of transport maps not just to random samples, but also to quasi-Monte Carlo points, higher-order nets, and sparse grids in order for the transformed samples to inherit the original convergence rates that are often better than $N^{-1/2}$, $N$ being the number of samples/quadrature nodes. Our main result is the derivation of an explicit transport map for the case that $\mathbb{P}_{\mathrm{tar}}$ is a mixture of simple distributions, e.g.\ a Gaussian mixture, in which case application of the transport map $T$ requires the solution of an \emph{explicit} ODE with \emph{closed-form} right-hand side. Mixture distributions are of particular applicability and interest since many methods proceed by first approximating $\mathbb{P}_{\mathrm{tar}}$ by a mixture and then sampling from that mixture (often using importance reweighting). Hence, this paper allows for the sampling step to provide a better convergence rate than $N^{-1/2}$ for all such methods.

math.NA↗

arXiv2022

Maximum a posteriori estimators in $\ell^p$ are well-defined for diagonal Gaussian priors

We prove that maximum a posteriori estimators are well-defined for diagonal Gaussian priors $μ$ on $\ell^p$ under common assumptions on the potential $Φ$. Further, we show connections to the Onsager--Machlup functional and provide a corrected and strongly simplified proof in the Hilbert space case $p=2$, previously established by Dashti et al (2013) and Kretschmann (2019). These corrections do not generalize to the setting $1 \leq p < \infty$, which requires a novel convexification result for the difference between the Cameron--Martin norm and the $p$-norm.

math.ST↗

arXiv2021

Γ-convergence of Onsager-Machlup functionals. Part I: With applications to maximum a posteriori estimation in Bayesian inverse problems

The Bayesian solution to a statistical inverse problem can be summarised by a mode of the posterior distribution, i.e. a MAP estimator. The MAP estimator essentially coincides with the (regularised) variational solution to the inverse problem, seen as minimisation of the Onsager-Machlup functional of the posterior measure. An open problem in the stability analysis of inverse problems is to establish a relationship between the convergence properties of solutions obtained by the variational approach and by the Bayesian approach. To address this problem, we propose a general convergence theory for modes that is based on the $Γ$-convergence of Onsager-Machlup functionals, and apply this theory to Bayesian inverse problems with Gaussian and edge-preserving Besov priors. Part II of this paper considers more general prior distributions.

math.ST↗

arXiv2021

Γ-convergence of Onsager-Machlup functionals. Part II: Infinite product measures on Banach spaces

We derive Onsager-Machlup functionals for countable product measures on weighted $\ell^p$ subspaces of the sequence space $\mathbb{R}^{\mathbb{N}}$. Each measure in the product is a shifted and scaled copy of a reference probability measure on $\mathbb{R}$ that admits a sufficiently regular Lebesgue density. We study the equicoercivity and $Γ$-convergence of sequences of Onsager-Machlup functionals associated to convergent sequences of measures within this class. We use these results to establish analogous results for probability measures on separable Banach or Hilbert spaces, including Gaussian, Cauchy, and Besov measures with summability parameter $1 \leq p \leq 2$. Together with Part I of this paper, this provides a basis for analysis of the convergence of maximum a posteriori estimators in Bayesian inverse problems and most likely paths in transition path theory.

math.ST↗

arXiv2020

The linear conditional expectation in Hilbert space

The linear conditional expectation (LCE) provides a best linear (or rather, affine) estimate of the conditional expectation and hence plays an important rôle in approximate Bayesian inference, especially the Bayes linear approach. This article establishes the analytical properties of the LCE in an infinite-dimensional Hilbert space context. In addition, working in the space of affine Hilbert--Schmidt operators, we establish a regularisation procedure for this LCE. As an important application, we obtain a simple alternative derivation and intuitive justification of the conditional mean embedding formula, a concept widely used in machine learning to perform the conditioning of random variables by embedding them into reproducing kernel Hilbert spaces.

math.ST↗

arXiv2020

Markov Chain Importance Sampling -- a highly efficient estimator for MCMC

Markov chain (MC) algorithms are ubiquitous in machine learning and statistics and many other disciplines. Typically, these algorithms can be formulated as acceptance rejection methods. In this work we present a novel estimator applicable to these methods, dubbed Markov chain importance sampling (MCIS), which efficiently makes use of rejected proposals. For the unadjusted Langevin algorithm, it provides a novel way of correcting the discretization error. Our estimator satisfies a central limit theorem and improves on error per CPU cycle, often to a large extent. As a by-product it enables estimating the normalizing constant, an important quantity in Bayesian machine learning and statistics.

stat.ML↗

arXiv2020

Objective Priors in the Empirical Bayes Framework

When dealing with Bayesian inference the choice of the prior often remains a debatable question. Empirical Bayes methods offer a data-driven solution to this problem by estimating the prior itself from an ensemble of data. In the nonparametric case, the maximum likelihood estimate is known to overfit the data, an issue that is commonly tackled by regularization. However, the majority of regularizations are ad hoc choices which lack invariance under reparametrization of the model and result in inconsistent estimates for equivalent models. We introduce a non-parametric, transformation invariant estimator for the prior distribution. Being defined in terms of the missing information similar to the reference prior, it can be seen as an extension of the latter to the data-driven setting. This implies a natural interpretation as a trade-off between choosing the least informative prior and incorporating the information provided by the data, a symbiosis between the objective and empirical Bayes methodologies.

stat.ME↗

arXiv2020

A Rigorous Theory of Conditional Mean Embeddings

Conditional mean embeddings (CMEs) have proven themselves to be a powerful tool in many machine learning applications. They allow the efficient conditioning of probability distributions within the corresponding reproducing kernel Hilbert spaces (RKHSs) by providing a linear-algebraic relation for the kernel mean embeddings of the respective joint and conditional probability distributions. Both centred and uncentred covariance operators have been used to define CMEs in the existing literature. In this paper, we develop a mathematically rigorous theory for both variants, discuss the merits and problems of each, and significantly weaken the conditions for applicability of CMEs. In the course of this, we demonstrate a beautiful connection to Gaussian conditioning in Hilbert spaces.

math.ST↗

arXiv2018

Adaptive Convolutions

When smoothing a function $f$ via convolution with some kernel, it is often desirable to adapt the amount of smoothing locally to the variation of $f$. For this purpose, the constant smoothing coefficient of regular convolutions needs to be replaced by an adaptation function $μ$. This function is matrix-valued which allows for different degrees of smoothing in different directions. The aim of this paper is twofold. The first is to provide a theoretical framework for such adaptive convolutions. The second purpose is to derive a formula for the automatic choice of the adaptation function $μ= μ_f$ in dependence of the function $f$ to be smoothed. This requires the notion of the \emph{local variation} of $f$, the quantification of which relies on certain phase space transformations of $f$. The derivation is guided by meaningful axioms which, among other things, guarantee invariance of adaptive convolutions under shifting and scaling of $f$.

math.FA↗

arXiv2018

Axiomatic Approach to Variable Kernel Density Estimation

Variable kernel density estimation allows the approximation of a probability density by the mean of differently stretched and rotated kernels centered at given sampling points $y_n\in\mathbb{R}^d,\ n=1,\dots,N$. Up to now, the choice of the corresponding bandwidth matrices $h_n$ has relied mainly on asymptotic arguments, like the minimization of the asymptotic mean integrated squared error (AMISE), which work well for large numbers of sampling points. However, in practice, one is often confronted with small to moderately sized sample sets far below the asymptotic regime, which highly restricts the usability of such methods. As an alternative to this asymptotic reasoning we suggest an axiomatic approach which guarantees invariance of the density estimate under linear transformations of the original density (and the sampling points) as well as under splitting of the density into several `well-separated' parts. In order to still ensure proper asymptotic behavior of the estimate, we \emph{postulate} the typical dependence $h_n\propto N^{-1/(d+4)}$. Further, we derive a new bandwidths selection rule which satisfies these axioms and performs considerably better than conventional ones in an artificially intricate two-dimensional example as well as in a real life example.

math.ST↗