arXiv subjects

Leon Bungert

Publications and source records attributed to Leon Bungert.

At least 19 recordsLinked to original sources

arXiv2026

Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multi-particle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like $\sqrt{{\log(\beta+1)}/{\beta}}\exp(Ct)+\exp(-ct)$ in terms of the temperature parameter $\beta^{-1}\to 0$ and inference time $t\geq 0$. For the proof, we establish Lyapunov-type estimates for the zero-temperature equation, identify its limit as $t\to\infty$, and employ a stability estimate in Wasserstein space together with a quantitative Laplace principle to couple the two equations. Our result implies that for time scales of order $\log\beta$ the token distribution concentrates at the identified limiting distribution. Numerical experiments confirm this and, beyond that, complement our theory by showing that for finite $\beta$ and large $t$ the dynamics enter a different terminal phase, dominated by the spectrum of the value matrix.

math.AP↗

arXiv2026

Duality for the Adversarial Total Variation

Adversarial training of binary classifiers can be reformulated as regularized risk minimization involving a nonlocal total variation. Building on this perspective, we establish a characterization of the subdifferential of this total variation using duality techniques. To achieve this, we derive a dual representation of the nonlocal total variation and a related integration of parts formula, involving a nonlocal gradient and divergence. We provide such duality statements both in the space of continuous functions vanishing at infinity on proper metric spaces and for the space of essentially bounded functions on Euclidean domains. Furthermore, under some additional conditions we provide characterizations of the subdifferential in these settings.

math.AP↗

arXiv2026

Sparse Training of Neural Networks based on Multilevel Mirror Descent

We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern updates. The key idea is to combine sparsity-inducing Bregman iterations with adaptive freezing of the network structure to enable efficient exploration of the sparse parameter space while maintaining sparsity. We provide convergence guaranties by embedding our method in a multilevel optimization framework. Furthermore, we empirically show that our algorithm can produce highly sparse and accurate models on standard benchmarks. We also show that the theoretical number of FLOPs compared to SGD training can be reduced from 38% for standard Bregman iterations to 6% for our method while maintaining test accuracy.We additionally show a training time reduction by about 50%, when using a sparsity-aware CPU implementation of our method.

cs.LG↗

arXiv2025

Convergence of graph Dirichlet energies and graph Laplacians on intersecting manifolds of varying dimensions

We study $\Gamma$-convergence of graph Dirichlet energies and spectral convergence of graph Laplacians on unions of intersecting manifolds of potentially different dimensions. Our investigation is motivated by problems of machine learning, as real-world data often consist of parts or classes with different intrinsic dimensions. An important challenge is to understand which machine learning methods adapt to such varied dimensionalities. We investigate the standard unnormalized and the normalized graph Dirichlet energies. We show that the unnormalized energy and its associated graph Laplacian asymptotically only sees the variations within the manifold of the highest dimension. On the other hand, we prove that the normalized Dirichlet energy converges to a (tensorized) Dirichlet energy on the union of manifolds that adapts to all dimensions simultaneously. We also establish the related spectral convergence and present a few numerical experiments to illustrate our findings.

math.AP↗

arXiv2025

Consensus-based optimization for closed-box adversarial attacks and a connection to evolution strategies

Consensus-based optimization (CBO) has established itself as an efficient gradient-free optimization scheme, with attractive mathematical properties, such as mean-field convergence results for non-convex loss functions. In this work, we study CBO in the context of closed-box adversarial attacks, which are imperceptible input perturbations that aim to fool a classifier, without accessing its gradient. Our contribution is to establish a connection between the so-called consensus hopping as introduced by Riedl et al. and natural evolution strategies (NES) commonly applied in the context of adversarial attacks and to rigorously relate both methods to gradient-based optimization schemes. Beyond that, we provide a comprehensive experimental study that shows that despite the conceptual similarities, CBO can outperform NES and other evolutionary strategies in certain scenarios.

math.OC↗

arXiv2025

Introduction to Nonlinear Spectral Analysis

These notes are meant as an introduction to the theory of nonlinear spectral theory. We will discuss the variational form of nonlninear eigenvalue problems and the corresponding non-linear Euler--Lagrange equations, as well as connections with gradient flows. For the latter ones, we will give precise conditions for finite time extinction and discuss convergence rates. We will use this theory to study asymptotic behaviour of nonlinear PDEs and present applications in $L^\infty$ variational problems. Finally we will discuss numerical methods for solving gradient flows and computing nonlinear eigenfunctions based on a nonlinear power method. Our main tools are convex analysis and calculus of variations, necessary background on which will be provided. It is expected that the reader is familiar with Hilbert spaces; familiarity with Banach spaces is beneficial but not strictly necessary. The notes are based on the lectures taught by the authors at the universities of Bonn and Cambridge in 2022.

math.SP↗

arXiv2025

Meshless Shape Optimization using Neural Networks and Partial Differential Equations on Graphs

Shape optimization involves the minimization of a cost function defined over a set of shapes, often governed by a partial differential equation (PDE). In the absence of closed-form solutions, one relies on numerical methods to approximate the solution. The level set method -- when coupled with the finite element method -- is one of the most versatile numerical shape optimization approaches but still suffers from the limitations of most mesh-based methods. In this work, we present a fully meshless level set framework that leverages neural networks to parameterize the level set function and employs the graph Laplacian to approximate the underlying PDE. Our approach enables precise computations of geometric quantities such as surface normals and curvature, and allows tackling optimization problems within the class of convex shapes.

math.NA↗

arXiv2025

MirrorCBO: A consensus-based optimization method in the spirit of mirror descent

In this work we propose MirrorCBO, a consensus-based optimization (CBO) method which generalizes standard CBO in the same way that mirror descent generalizes gradient descent. For this we apply the CBO methodology to a swarm of dual particles and retain the primal particle positions by applying the inverse of the mirror map, which we parametrize as the subdifferential of a strongly convex function $\phi$. In this way, we combine the advantages of a derivative-free non-convex optimization algorithm with those of mirror descent. As a special case, the method extends CBO to optimization problems with convex constraints. Assuming bounds on the Bregman distance associated to $\phi$, we provide asymptotic convergence results for MirrorCBO with explicit exponential rate. Another key contribution is an exploratory numerical study of this new algorithm across different application settings, focusing on (i) sparsity-inducing optimization, and (ii) constrained optimization, demonstrating the competitive performance of MirrorCBO. We observe empirically that the method can also be used for optimization on (non-convex) submanifolds of Euclidean space, can be adapted to mirrored versions of other recent CBO variants, and that it inherits from mirror descent the capability to select desirable minimizers, like sparse ones. We also include an overview of recent CBO approaches for constrained optimization and compare their performance to MirrorCBO.

math.OC↗

arXiv2024

Convergence rates of the fractional to the local Dirichlet problem

We prove non-asymptotic rates of convergence in the $W^{s,2}(\mathbb R^d)$-norm for the solution of the fractional Dirichlet problem to the solution of the local Dirichlet problem as $s\uparrow 1$. For regular enough boundary values we get a rate of order $\sqrt{1-s}$, while for less regular data the rate is of order $\sqrt{(1-s)|\log(1-s)|}$. We also obtain results when the right hand side depends on $s$, and our error estimates are true for all $s\in(0,1)$. The proofs use variational arguments to deduce rates in the fractional Sobolev norm from energy estimates between the fractional and the standard Dirichlet energy.

math.AP↗

arXiv2024

Convergence rates for Poisson learning to a Poisson equation with measure data

In this paper we prove discrete to continuum convergence rates for Poisson Learning, a graph-based semi-supervised learning algorithm that is based on solving the graph Poisson equation with a source term consisting of a linear combination of Dirac deltas located at labeled points and carrying label information. The corresponding continuum equation is a Poisson equation with measure data in a Euclidean domain $\Omega \subset \mathbb{R}^d$. The singular nature of these equations is challenging and requires an approach with several distinct parts: (1) We prove quantitative error estimates when convolving the measure data of a Poisson equation with (approximately) radial function supported on balls. (2) We use quantitative variational techniques to prove discrete to continuum convergence rates on random geometric graphs with bandwidth $\varepsilon>0$ for bounded source terms. (3) We show how to regularize the graph Poisson equation via mollification with the graph heat kernel, and we study fine asymptotics of the heat kernel on random geometric graphs. Combining these three pillars we obtain $L^1$ convergence rates that scale, up to logarithmic factors, like $O(\varepsilon^{\frac{1}{d+2}})$ for general data distributions, and $O(\varepsilon^{\frac{2-\sigma}{d+4}})$ for uniformly distributed data, where $\sigma>0$. These rates are valid with high probability if $\varepsilon\gg\left({\log n}/{n}\right)^q$ where $n$ denotes the number of vertices of the graph and $q \approx \frac{1}{3d}$.

math.AP↗

arXiv2024

A mean curvature flow arising in adversarial training

We connect adversarial training for binary classification to a geometric evolution equation for the decision boundary. Relying on a perspective that recasts adversarial training as a regularization problem, we introduce a modified training scheme that constitutes a minimizing movements scheme for a nonlocal perimeter functional. We prove that the scheme is monotone and consistent as the adversarial budget vanishes and the perimeter localizes, and as a consequence we rigorously show that the scheme approximates a weighted mean curvature flow. This highlights that the efficacy of adversarial training may be due to locally minimizing the length of the decision boundary. In our analysis, we introduce a variety of tools for working with the subdifferential of a supremal-type nonlocal total variation and its regularity properties.

math.AP↗

arXiv2023

The Infinity Laplacian eigenvalue problem: reformulation and a numerical scheme

In this work, we present an alternative formulation of the higher eigenvalue problem associated to the infinity Laplacian, which opens the door for numerical approximation of eigenfunctions. A rigorous analysis is performed to show the equivalence of the new formulation to the traditional one. Subsequently, we present consistent monotone schemes to approximate infinity ground states and higher eigenfunctions on grids. We prove that our method converges (up to a subsequence) to a viscosity solution of the eigenvalue problem, and perform numerical experiments which investigate theoretical conjectures and compute eigenfunctions on a variety of different domains.

math.NA↗

arXiv2023

The convergence rate of $p$-harmonic to infinity-harmonic functions

The purpose of this paper is to prove a uniform convergence rate of the solutions of the $p$-Laplace equation $Δ_p u = 0$ with Dirichlet boundary conditions to the solution of the infinity-Laplace equation $Δ_\infty u = 0$ as $p\to\infty$. The rate scales like $p^{-1/4}$ for general solutions of the Dirichlet problem and like $p^{-1/2}$ for solutions with positive gradient. An explicit example shows that it cannot be better than $p^{-1}$. The proof of this result solely relies on the comparison principle with the fundamental solutions of the $p$-Laplace and the infinity-Laplace equation, respectively. Our argument does not use viscosity solutions, is purely metric, and is therefore generalizable to more general settings where a comparison principle with Hölder cones and Hölder regularity is available.

math.AP↗

arXiv2023

It begins with a boundary: A geometric view on probabilistically robust learning

Although deep neural networks have achieved super-human performance on many classification tasks, they often exhibit a worrying lack of robustness towards adversarially generated examples. Thus, considerable effort has been invested into reformulating standard Risk Minimization (RM) into an adversarially robust framework. Recently, attention has shifted towards approaches which interpolate between the robustness offered by adversarial training and the higher clean accuracy and faster training times of RM. In this paper, we take a fresh and geometric view on one such method -- Probabilistically Robust Learning (PRL). We propose a mathematical framework for understanding PRL, which allows us to identify geometric pathologies in its original formulation and to introduce a family of probabilistic nonlocal perimeter functionals to rectify them. We prove existence of solutions to the original and modified problems using novel relaxation methods and also study properties, as well as local limits, of the introduced perimeters. We also clarify, through a suitable $\Gamma$-convergence analysis, the way in which the original and modified PRL models interpolate between risk minimization and adversarial training.

cs.LG↗

arXiv2022

The inhomogeneous $p$-Laplacian equation with Neumann boundary conditions in the limit $p\to\infty$

We investigate the limiting behavior of solutions to the inhomogeneous $p$-Laplacian equation $-Δ_p u = μ_p$ subject to Neumann boundary conditions. For right hand sides which are arbitrary signed measures we show that solutions converge to a Kantorovich potential associated with the geodesic Wasserstein-$1$ distance. In the regular case with continuous right hand sides we characterize the limit as viscosity solution to an infinity Laplacian / eikonal type equation.

math.AP↗

arXiv2022

Gamma-convergence of a nonlocal perimeter arising in adversarial machine learning

In this paper we prove Gamma-convergence of a nonlocal perimeter of Minkowski type to a local anisotropic perimeter. The nonlocal model describes the regularizing effect of adversarial training in binary classifications. The energy essentially depends on the interaction between two distributions modelling likelihoods for the associated classes. We overcome typical strict regularity assumptions for the distributions by only assuming that they have bounded $BV$ densities. In the natural topology coming from compactness, we prove Gamma-convergence to a weighted perimeter with weight determined by an anisotropic function of the two densities. Despite being local, this sharp interface limit reflects classification stability with respect to adversarial perturbations. We further apply our results to deduce Gamma-convergence of the associated total variations, to study the asymptotics of adversarial training, and to prove Gamma-convergence of graph discretizations for the nonlocal perimeter.

math.AP↗

arXiv2022

Polarized consensus-based dynamics for optimization and sampling

In this paper we propose polarized consensus-based dynamics in order to make consensus-based optimization (CBO) and sampling (CBS) applicable for objective functions with several global minima or distributions with many modes, respectively. For this, we ``polarize'' the dynamics with a localizing kernel and the resulting model can be viewed as a bounded confidence model for opinion formation in the presence of common objective. Instead of being attracted to a common weighted mean as in the original consensus-based methods, which prevents the detection of more than one minimum or mode, in our method every particle is attracted to a weighted mean which gives more weight to nearby particles. We prove that in the mean-field regime the polarized CBS dynamics are unbiased for Gaussian targets. We also prove that in the zero temperature limit and for sufficiently well-behaved strongly convex objectives the solution of the Fokker--Planck equation converges in the Wasserstein-2 distance to a Dirac measure at the minimizer. Finally, we propose a computationally more efficient generalization which works with a predefined number of clusters and improves upon our polarized baseline method for high-dimensional optimization.

math.OC↗

arXiv2022

CLIP: Cheap Lipschitz Training of Neural Networks

Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of each layer which gives a poor bound for the actual Lipschitz constant. In this paper we investigate a variational regularization method named CLIP for controlling the Lipschitz constant of a neural network, which can easily be integrated into the training procedure. We mathematically analyze the proposed model, in particular discussing the impact of the chosen regularization parameter on the output of the network. Finally, we numerically evaluate our method on both a nonlinear regression problem and the MNIST and Fashion-MNIST classification databases, and compare our results with a weight regularization approach.

cs.LG↗