arXiv subjects

Stephan Eckstein

Publications and source records attributed to Stephan Eckstein.

At least 19 recordsLinked to original sources

arXiv2026

Gromov-Wasserstein Quantization and Clustering: Structure, Rates, and Algorithms

Clustering is a fundamental class of data analysis techniques with the most important representatives being centroid-based methods like $k$-means. Such methods are strongly connected to quantization problems, which aim to approximate general probability measures with discrete ones. For example, $k$-means corresponds to quantization with respect to the Wasserstein distance. While Wasserstein quantization clusters points within a fixed space, this paper studies Gromov-Wasserstein (GW) quantization, which additionally aims at clustering the ambient geometry of the space. We show existence of solutions to the GW quantization problem and give a characterization that justifies an analogue to the $k$-means algorithm (Lloyd's algorithm) to approximate them numerically. We further calculate the quantization rate for usual Euclidean geometries that are used in the GW context, and relate it to standard Wasserstein quantization rates. Finally, numerical experiments show that GW quantization opens up many modeling possibilities beyond normal clustering methods (e.g., for geodesic distances of 3D shapes or structured pruning of neural networks) and that the introduced algorithm leads to useful numerical solutions with approximation quality often in line with theoretically optimal rates.

math.OC↗

arXiv2026

Partial Fusion of Neural Networks: Efficient Tradeoffs Between Ensembles and Weight Aggregation

Ensembles of neural networks typically outperform individual networks but incur large computational costs, whereas weight aggregation produces less costly, yet also less accurate, aggregate models. We introduce partial fusion of networks, which interpolates between ensembles and weight aggregation and thus allows for a flexible tradeoff between computational cost and performance. A direct way to achieve this is to extend existing weight aggregation methods based on neuron-level similarity between different networks, where partial fusion then only aggregates weights of neurons which are most similar. We showcase one particular method to jointly identify which neurons are most similar and match them via partial optimal transport. Further, we consider the more general perspective of weight aggregation and partial fusion as generalized pruning of ensemble models, where neurons cannot just be deleted, but also linearly combined. Finally, we show that generalized pruning applied to a single network yields similar benefits as partial fusion by allowing for a tradeoff between isolating, deleting, and linearly combining neurons based on similarity. Our code is available at https://github.com/Fabian-Mor/partial_fusion_nn.

cs.LG↗

arXiv2026

The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought

Existing expressivity results for transformers typically rely on hardmax attention, high precision, and other architectural modifications that disconnect them from the models used in practice. We bridge this gap by analyzing standard transformer decoders with softmax attention and rounding of activations and attention weights, while allowing depth and width to grow logarithmically with the context length. As an intermediate step, we construct hardmax transformers with ternary activations and well-separated attention scores that simulate Turing machines using Chain-of-Thought (CoT). This lets us convert the constructions to equivalent softmax transformers without the unrealistic parameter magnitudes or activation precision that prior approaches would require. Using the same technique, we analyze a recently proposed summarized CoT paradigm and show that it simulates Turing machines more efficiently, with model size scaling logarithmically in a space bound rather than a time bound. We empirically test predictions made by our results on a Sudoku reasoning task and find better alignment with learnability than for prior high-precision results. Our code is available at https://github.com/moritzbroe/transformer-expressivity.

cs.LG↗

arXiv2025

Fast Wasserstein rates for estimating probability distributions of probabilistic graphical models

Using i.i.d. data to estimate a high-dimensional distribution in Wasserstein distance is a fundamental instance of the curse of dimensionality. We explore how structural knowledge about the data-generating process which gives rise to the distribution can be used to overcome this curse. More precisely, we work with the set of distributions of probabilistic graphical models for a given directed acyclic graph. It turns out that this knowledge is only helpful if it can be quantified, which we formalize via smoothness conditions on the transition kernels in the disintegration corresponding to the graph. In this case, we prove that the rate of estimation is governed by the local structure of the graph, more precisely by dimensions corresponding to single nodes together with their parent nodes. The precise rate depends on the exact notion of smoothness assumed for the kernels, where either weak (Wasserstein-Lipschitz) or strong (bidirectional Total-Variation-Lipschitz) conditions lead to different results. We prove sharpness under the strong condition and show that this condition covers, as a special case, distributions having a positive Lipschitz density.

math.ST↗

arXiv2025

Sparse Regularized Optimal Transport without Curse of Dimensionality

Entropic optimal transport -- the optimal transport problem regularized by KL diver\-gence -- is highly successful in statistical applications. Thanks to the smoothness of the entropic coupling, its sample complexity avoids the curse of dimensionality suffered by unregularized optimal transport. The flip side of smoothness is overspreading: the entropic coupling always has full support, whereas the unregularized coupling that it approximates is usually sparse, even given by a map. Regularizing optimal transport by less-smooth $f$-divergences such as Tsallis divergence (i.e., $L^p$-regularization) is known to allow for sparse approximations, but is often thought to suffer from the curse of dimensionality as the couplings have limited differentiability and the dual is not strongly concave. We refute this conventional wisdom and show, for a broad family of divergences, that the key empirical quantities converge at the parametric rate, independently of the dimension. More precisely, we provide central limit theorems for the optimal cost, the optimal coupling, and the dual potentials induced by i.i.d.\ samples from the marginals. These results are obtained by a powerful yet elementary approach that is of broader interest for Z-estimation in function classes that are not Donsker.

math.ST↗

arXiv2025

Exponential convergence of general iterative proportional fitting procedures

Motivated by the success of Sinkhorn's algorithm for entropic optimal transport, we study convergence properties of iterative proportional fitting procedures (IPFP) used to solve more general information projection problems. We establish exponential convergence guarantees for the IPFP whenever the set of probability measures which is projected onto is defined through constraints arising from linear function spaces. This unifies and extends recent results from multi-marginal, adapted and martingale optimal transport. The proofs are based on strong convexity arguments for the dual problem, and the key contribution is to illuminate the role of the geometric interplay between the subspaces defining the constraints. In this regard, we show that the larger the angle (in the sense of Friedrichs) between the linear function spaces, the better the rate of contraction of the IPFP.

math.OC↗

arXiv2024

Time-Causal VAE: Robust Financial Time Series Generator

We build a time-causal variational autoencoder (TC-VAE) for robust generation of financial time series data. Our approach imposes a causality constraint on the encoder and decoder networks, ensuring a causal transport from the real market time series to the fake generated time series. Specifically, we prove that the TC-VAE loss provides an upper bound on the causal Wasserstein distance between market distributions and generated distributions. Consequently, the TC-VAE loss controls the discrepancy between optimal values of various dynamic stochastic optimization problems under real and generated distributions. To further enhance the model's ability to approximate the latent representation of the real market distribution, we integrate a RealNVP prior into the TC-VAE framework. Finally, extensive numerical experiments show that TC-VAE achieves promising results on both synthetic and real market data. This is done by comparing real and generated distributions according to various statistical distances, demonstrating the effectiveness of the generated data for downstream financial optimization tasks, as well as showcasing that the generated data reproduces stylized facts of real financial market data.

cs.LG↗

arXiv2024

Optimal nonparametric estimation of the expected shortfall risk

We address the problem of estimating the expected shortfall risk of a financial loss using a finite number of i.i.d. data. It is well known that the classical plug-in estimator suffers from poor statistical performance when faced with (heavy-tailed) distributions that are commonly used in financial contexts. Further, it lacks robustness, as the modification of even a single data point can cause a significant distortion. We propose a novel procedure for the estimation of the expected shortfall and prove that it recovers the best possible statistical properties (dictated by the central limit theorem) under minimal assumptions and for all finite numbers of data. Further, this estimator is adversarially robust: even if a (small) proportion of the data is maliciously modified, the procedure continuous to optimally estimate the true expected shortfall risk. We demonstrate that our estimator outperforms the classical plug-in estimator through a variety of numerical experiments across a range of standard loss distributions.

q-fin.RM↗

arXiv2023

Hilbert's projective metric for functions of bounded growth and exponential convergence of Sinkhorn's algorithm

Motivated by the entropic optimal transport problem in unbounded settings, we study versions of Hilbert's projective metric for spaces of integrable functions of bounded growth. These versions of Hilbert's metric originate from cones which are relaxations of the cone of all non-negative functions, in the sense that they include all functions having non-negative integral values when multiplied with certain test functions. We show that kernel integral operators are contractions with respect to suitable specifications of such metrics even for kernels which are not bounded away from zero, provided that the decay to zero of the kernel is controlled. As an application to entropic optimal transport, we show exponential convergence of Sinkhorn's algorithm in settings where the marginal distributions have sufficiently light tails compared to the growth of the cost function.

math.PR↗

arXiv2023

Estimating the Rate-Distortion Function by Wasserstein Gradient Descent

In the theory of lossy compression, the rate-distortion (R-D) function $R(D)$ describes how much a data source can be compressed (in bit-rate) at any given level of fidelity (distortion). Obtaining $R(D)$ for a given data source establishes the fundamental performance limit for all compression algorithms. We propose a new method to estimate $R(D)$ from the perspective of optimal transport. Unlike the classic Blahut--Arimoto algorithm which fixes the support of the reproduction distribution in advance, our Wasserstein gradient descent algorithm learns the support of the optimal reproduction distribution by moving particles. We prove its local convergence and analyze the sample complexity of our R-D estimator based on a connection to entropic optimal transport. Experimentally, we obtain comparable or tighter bounds than state-of-the-art neural network methods on low-rate sources while requiring considerably less tuning and computation effort. We also highlight a connection to maximum-likelihood deconvolution and introduce a new class of sources that can be used as test cases with known solutions to the R-D problem.

cs.IT↗

arXiv2023

Optimal transport and Wasserstein distances for causal models

In this paper, we introduce a variant of optimal transport adapted to the causal structure given by an underlying directed graph $G$. Different graph structures lead to different specifications of the optimal transport problem. For instance, a fully connected graph yields standard optimal transport, a linear graph structure corresponds to causal optimal transport between the distributions of two discrete-time stochastic processes, and an empty graph leads to a notion of optimal transport related to CO-OT, Gromov-Wasserstein distances and factored OT. We derive different characterizations of $G$-causal transport plans and introduce Wasserstein distances between causal models that respect the underlying graph structure. We show that average treatment effects are continuous with respect to $G$-causal Wasserstein distances and small perturbations of structural causal models lead to small deviations in $G$-causal Wasserstein distance. We also introduce an interpolation between causal models based on $G$-causal Wasserstein distance and compare it to standard Wasserstein interpolation.

math.ST↗

arXiv2022

Stability and Sample Complexity of Divergence Regularized Optimal Transport

We study stability and sample complexity properties of divergence regularized optimal transport (DOT). First, we obtain quantitative stability results for optimizers of DOT measured in Wasserstein distance, which are applicable to a wide class of divergences and simultaneously improve known results for entropic optimal transport. Second, we study the case of sample complexity, where the DOT problem is approximated using empirical measures of the marginals. We show that divergence regularization can improve the corresponding convergence rate compared to unregularized optimal transport. To this end, we prove upper bounds which exploit both the regularity of cost function and divergence functional, as well as the intrinsic dimension of the marginals. Along the way, we establish regularity properties of dual optimizers of DOT, as well as general limit theorems for empirical measures with suitable classes of test functions.

math.OC↗

arXiv2022

Convergence Rates for Regularized Optimal Transport via Quantization

We study the convergence of divergence-regularized optimal transport as the regularization parameter vanishes. Sharp rates for general divergences including relative entropy or $L^{p}$ regularization, general transport costs and multi-marginal problems are obtained. A novel methodology using quantization and martingale couplings is suitable for non-compact marginals and achieves, in particular, the sharp leading-order term of entropically regularized 2-Wasserstein distance for all marginals with finite $(2+\delta)$-moment.

math.OC↗

arXiv2022

Dimensionality Reduction and Wasserstein Stability for Kernel Regression

In a high-dimensional regression framework, we study consequences of the naive two-step procedure where first the dimension of the input variables is reduced and second, the reduced input variables are used to predict the output variable with kernel regression. In order to analyze the resulting regression errors, a novel stability result for kernel regression with respect to the Wasserstein distance is derived. This allows us to bound errors that occur when perturbed input data is used to fit the regression function. We apply the general stability result to principal component analysis (PCA). Exploiting known estimates from the literature on both principal component analysis and kernel regression, we deduce convergence rates for the two-step procedure. The latter turns out to be particularly useful in a semi-supervised setting.

stat.ML↗

arXiv2022

Computational methods for adapted optimal transport

Adapted optimal transport (AOT) problems are optimal transport problems for distributions of a time series where couplings are constrained to have a temporal causal structure. In this paper, we develop computational tools for solving AOT problems numerically. First, we show that AOT problems are stable with respect to perturbations in the marginals and thus arbitrary AOT problems can be approximated by sequences of linear programs. We further study entropic methods to solve AOT problems. We show that any entropically regularized AOT problem converges to the corresponding unregularized problem if the regularization parameter goes to zero. The proof is based on a novel method - even in the non-adapted case - to easily obtain smooth approximations of a given coupling with fixed marginals. Finally, we show tractability of the adapted version of Sinkhorn's algorithm. We give explicit solutions for the occurring projections and prove that the procedure converges to the optimizer of the entropic AOT problem.

math.PR↗

arXiv2021

Quantitative Stability of Regularized Optimal Transport and Convergence of Sinkhorn's Algorithm

We study the stability of entropically regularized optimal transport with respect to the marginals. Lipschitz continuity of the value and H\"older continuity of the optimal coupling in $p$-Wasserstein distance are obtained under general conditions including quadratic costs and unbounded marginals. The results for the value extend to regularization by an arbitrary divergence. As an application, we show convergence of Sinkhorn's algorithm in Wasserstein sense, including for quadratic cost. Two techniques are presented: The first compares an optimal coupling with its so-called shadow, a coupling induced on other marginals by an explicit construction. The second transforms one set of marginals by a change of coordinates and thus reduces the comparison of differing marginals to the comparison of differing cost functions under the same marginals.

math.OC↗

arXiv2021

Martingale transport with homogeneous stock movements

We study a variant of the martingale optimal transport problem in a multi-period setting to derive robust price bounds of a financial derivative. On top of marginal and martingale constraints, we introduce a time-homogeneity assumption, which restricts the variability of the forward-looking transitions of the martingale across time. We provide a dual formulation in terms of superhedging and discuss relaxations of the time-homogeneity assumption by adding market frictions. In financial terms, the introduced time-homogeneity corresponds to a time-consistency condition for call prices, given the state of the stock. The time homogeneity assumption leads to improved price bounds as market data from many time points can be incorporated effectively. The approach is illustrated with two numerical examples.

q-fin.MF↗

arXiv2021

Limits of random walks with distributionally robust transition probabilities

We consider a nonlinear random walk which, in each time step, is free to choose its own transition probability within a neighborhood (w.r.t. Wasserstein distance) of the transition probability of a fixed Lévy process. In analogy to the classical framework we show that, when passing from discrete to continuous time via a scaling limit, this nonlinear random walk gives rise to a nonlinear semigroup. We explicitly compute the generator of this semigroup and corresponding PDE as a perturbation of the generator of the initial Lévy process.

math.PR↗