arXiv subjects

Lukas Weigand

Publications and source records attributed to Lukas Weigand.

4 recordsLinked to original sources

arXiv2025

Explainable Learning Based Regularization of Inverse Problems

Machine learning techniques for the solution of inverse problems have become an attractive approach in the last decade, while their theoretical foundations are still in their infancy. In this chapter we want to pursue the study of regularization properties, robustness, convergence rates, and structure of regularizers for inverse problems obtained from different learning paradigms. For this sake we study simple architectures that are explainable in the sense that they allow for a theoretical analysis also in the infinite-dimensional limit. In particular we will advance the study of spectral architectures with new results on convergence rates highlighting the role of the smoothness in the training data set, and a study of adversarial robustness. We can show that adversarial training is actually a convergent regularization method. Moreover, we discuss extensions to frame systems and CNN-type architectures for variational regularizers, where we obtain some results on their structure by carefully designed numerical experiments.

math.NA↗

arXiv2025

Adversarial flows: A gradient flow characterization of adversarial attacks

A popular method to perform adversarial attacks on neuronal networks is the so-called fast gradient sign method and its iterative variant. In this paper, we interpret this method as an explicit Euler discretization of a differential inclusion, where we also show convergence of the discretization to the associated gradient flow. To do so, we consider the concept of p-curves of maximal slope in the case $p=\infty$. We prove existence of $\infty$-curves of maximum slope and derive an alternative characterization via differential inclusions. Furthermore, we also consider Wasserstein gradient flows for potential energies, where we show that curves in the Wasserstein space can be characterized by a representing measure on the space of curves in the underlying Banach space, which fulfill the differential inclusion. The application of our theory to the finite-dimensional setting is twofold: On the one hand, we show that a whole class of normalized gradient descent methods (in particular signed gradient descent) converge, up to subsequences, to the flow, when sending the step size to zero. On the other hand, in the distributional setting, we show that the inner optimization task of adversarial training objective can be characterized via $\infty$-curves of maximum slope on an appropriate optimal transport space.

cs.LG↗

arXiv2025

Analysis of mean-field models arising from self-attention dynamics in transformer architectures with layer normalization

The aim of this paper is to provide a mathematical analysis of transformer architectures using a self-attention mechanism with layer normalization. In particular, observed patterns in such architectures resembling either clusters or uniform distributions pose a number of challenging mathematical questions. We focus on a special case that admits a gradient flow formulation in the spaces of probability measures on the unit sphere under a special metric, which allows us to give at least partial answers in a rigorous way. The arising mathematical problems resemble those recently studied in aggregation equations, but with additional challenges emerging from restricting the dynamics to the sphere and the particular form of the interaction energy. We provide a rigorous framework for studying the gradient flow, which also suggests a possible metric geometry to study the general case (i.e. one that is not described by a gradient flow). We further analyze the stationary points of the induced self-attention dynamics. The latter are related to stationary points of the interaction energy in the Wasserstein geometry, and we further discuss energy minimizers and maximizers in different parameter settings.

math.AP↗

arXiv2024

Analysis of Primal-Dual Langevin Algorithms

We analyze a recently proposed class of algorithms for the problem of sampling from probability distributions $μ^\ast$ in $\mathbb{R}^d$ with a Lebesgue density of the form $μ^\ast(x) \propto \exp(-f(Kx)-g(x))$, where $K$ is a linear operator and $f,g$ convex and non-smooth. The method is a generalization of the primal-dual hybrid gradient optimization algorithm to a sampling scheme. We give the iteration's continuous time limit, a stochastic differential equation in the joint primal-dual variable, and its mean field limit Fokker-Planck equation. Under mild conditions, the scheme converges to a unique stationary state in continuous and discrete time. Contrary to purely primal overdamped Langevin diffusion, the stationary state in continuous time does not have $μ^\ast$ as its primal marginal. Thus, further analysis is carried out to bound the bias induced by the partial dualization, and potentially correct for it in the diffusion. Time discretizations of the diffusion lead to implementable algorithms, but, as is typical in Langevin Monte Carlo methods, introduce further bias. We prove bounds for these discretization errors, which allow to give convergence results relating the produced samples to the target. We demonstrate our findings numerically first on small-scale examples in which we can exactly verify the theoretical results, and subsequently on typical examples of larger scale from Bayesian imaging inverse problems.

math.OC↗