Searcharxiv⌕ Search

arXiv subjects

Henning Petzka

Publications and source records attributed to Henning Petzka.

15 recordsLinked to original sources
arXiv2026

Geometric and Information Compression of Representations in Deep Learning

Deep neural networks transform input data into latent representations that support a wide range of downstream tasks. These representations can be characterized along information-theoretic and geometric dimensions, but their relationship remains poorly understood. A central open question is whether low mutual information (MI) between inputs and representations necessarily implies geometrically compressed latent spaces and vice versa. We investigate this question using class-wise clustering as a measure of geometric compression and theoretically sound MI estimation in conditional entropy bottleneck (CEB) networks and continuous dropout networks. We evaluate the interplay between MI, geometric compression, and generalization on classification tasks under controlled noise injection schemes. Our findings show that low MI does not reliably correspond to geometric compression, and that the connection between the two is more nuanced than often assumed. Indeed, our experiments reveal a negative and nonlinear relationship that can reverse when varying training setup. Our results put forward a hypothesis that generalization acts as a potential confounder in this connection rather than being their direct consequence.

cs.LG↗
arXiv2025

PRADA: Probability-Ratio-Based Attribution and Detection of Autoregressive-Generated Images

Autoregressive (AR) image generation has recently emerged as a powerful paradigm for image synthesis. Leveraging the generation principle of large language models, they allow for efficiently generating deceptively real-looking images, further increasing the need for reliable detection methods. However, to date there is a lack of work specifically targeting the detection of images generated by AR image generators. In this work, we present PRADA (Probability-Ratio-Based Attribution and Detection of Autoregressive-Generated Images), a simple and interpretable approach that can reliably detect AR-generated images and attribute them to their respective source model. The key idea is to inspect the ratio of a model's conditional and unconditional probability for the autoregressive token sequence representing a given image. Whenever an image is generated by a particular model, its probability ratio shows unique characteristics which are not present for images generated by other models or real images. We exploit these characteristics for threshold-based attribution and detection by calibrating a simple, model-specific score function. Our experimental evaluation shows that PRADA is highly effective against eight class-to-image and four text-to-image models. We release our code and data at github.com/jonasricker/prada.

cs.CV↗
arXiv2025

Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokking

Neural collapse, i.e., the emergence of highly symmetric, class-wise clustered representations, is frequently observed in deep networks and is often assumed to reflect or enable generalization. In parallel, flatness of the loss landscape has been theoretically and empirically linked to generalization. Yet, the causal role of either phenomenon remains unclear: Are they prerequisites for generalization, or merely by-products of training dynamics? We disentangle these questions using grokking, a training regime in which memorization precedes generalization, allowing us to temporally separate generalization from training dynamics and we find that while both neural collapse and relative flatness emerge near the onset of generalization, only flatness consistently predicts it. Models encouraged to collapse or prevented from collapsing generalize equally well, whereas models regularized away from flat solutions exhibit delayed generalization, resembling grokking, even in architectures and datasets where it does not typically occur. Furthermore, we show theoretically that neural collapse leads to relative flatness under classical assumptions, explaining their empirical co-occurrence. Our results support the view that relative flatness is a potentially necessary and more fundamental property for generalization, and demonstrate how grokking can serve as a powerful probe for isolating its geometric underpinnings.

cs.LG↗
arXiv2023

FAM: Relative Flatness Aware Minimization

Flatness of the loss curve around a model at hand has been shown to empirically correlate with its generalization ability. Optimizing for flatness has been proposed as early as 1994 by Hochreiter and Schmidthuber, and was followed by more recent successful sharpness-aware optimization techniques. Their widespread adoption in practice, though, is dubious because of the lack of theoretically grounded connection between flatness and generalization, in particular in light of the reparameterization curse - certain reparameterizations of a neural network change most flatness measures but do not change generalization. Recent theoretical work suggests that a particular relative flatness measure can be connected to generalization and solves the reparameterization curse. In this paper, we derive a regularizer based on this relative flatness that is easy to compute, fast, efficient, and works with arbitrary loss functions. It requires computing the Hessian only of a single layer of the network, which makes it applicable to large neural networks, and with it avoids an expensive mapping of the loss surface in the vicinity of the model. In an extensive empirical evaluation we show that this relative flatness aware minimization (FAM) improves generalization in a multitude of applications and models, both in finetuning and standard training. We make the code available at github.

cs.LG↗
arXiv2022

Discriminating Against Unrealistic Interpolations in Generative Adversarial Networks

Interpolations in the latent space of deep generative models is one of the standard tools to synthesize semantically meaningful mixtures of generated samples. As the generator function is non-linear, commonly used linear interpolations in the latent space do not yield the shortest paths in the sample space, resulting in non-smooth interpolations. Recent work has therefore equipped the latent space with a suitable metric to enforce shortest paths on the manifold of generated samples. These are often, however, susceptible of veering away from the manifold of real samples, resulting in smooth but unrealistic generation that requires an additional method to assess the sample quality along paths. Generative Adversarial Networks (GANs), by construction, measure the sample quality using its discriminator network. In this paper, we establish that the discriminator can be used effectively to avoid regions of low sample quality along shortest paths. By reusing the discriminator network to modify the metric on the latent space, we propose a lightweight solution for improved interpolations in pre-trained GANs.

cs.LG↗
arXiv2021

Relative Flatness and Generalization

Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating it to the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and to identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flatness measure that correlates strongly with generalization, simplifies to ridge regression for ordinary least squares, and solves the reparameterization issue.

cs.LG↗
arXiv2020

Non-attracting Regions of Local Minima in Deep and Wide Neural Networks

Understanding the loss surface of neural networks is essential for the design of models with predictable performance and their success in applications. Experimental results suggest that sufficiently deep and wide neural networks are not negatively impacted by suboptimal local minima. Despite recent progress, the reason for this outcome is not fully understood. Could deep networks have very few, if at all, suboptimal local optima? or could all of them be equally good? We provide a construction to show that suboptimal local minima (i.e., non-global ones), even though degenerate, exist for fully connected neural networks with sigmoid activation functions. The local minima obtained by our construction belong to a connected set of local solutions that can be escaped from via a non-increasing path on the loss curve. For extremely wide neural networks of decreasing width after the wide layer, we prove that every suboptimal local minimum belongs to such a connected set. This provides a partial explanation for the successful application of deep neural networks. In addition, we also characterize under what conditions the same construction leads to saddle points instead of local minima for deep neural networks.

cs.LG↗
arXiv2019

A Reparameterization-Invariant Flatness Measure for Deep Neural Networks

The performance of deep neural networks is often attributed to their automated, task-related feature construction. It remains an open question, though, why this leads to solutions with good generalization, even in cases where the number of parameters is larger than the number of samples. Back in the 90s, Hochreiter and Schmidhuber observed that flatness of the loss surface around a local minimum correlates with low generalization error. For several flatness measures, this correlation has been empirically validated. However, it has recently been shown that existing measures of flatness cannot theoretically be related to generalization due to a lack of invariance with respect to reparameterizations. We propose a natural modification of existing flatness measures that results in invariance to reparameterization.

cs.LG↗
arXiv2017

On the regularization of Wasserstein GANs

Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. It was proposed that training can be improved by instead augmenting the loss by a regularization term that penalizes the deviation of the gradient of the critic (as a function of the network's input) from one. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on toy data sets.

stat.ML↗
arXiv2016

Perforation conditions and almost algebraic order in Cuntz semigroups

For a C$^*$-algebra $A$, it is an important problem to determine the Cuntz semigroup $\mathrm{Cu}(A\otimes\mathcal{Z})$ in terms of $\mathrm{Cu}(A)$. We approach this problem from the point of view of semigroup tensor products in the category of abstract Cuntz semigroups, by analysing the passage of significant properties from $\mathrm{Cu}(A)$ to $\mathrm{Cu}(A)\otimes_\mathrm{Cu}\mathrm{Cu}(\mathcal{Z})$. We describe the effect of the natural map $\mathrm{Cu}(A)\to\mathrm{Cu}(A)\otimes_\mathrm{Cu}\mathrm{Cu}(\mathcal{Z})$ in the order of $\mathrm{Cu}(A)$, and show that, if $A$ has real rank zero and no elementary subquotients, $\mathrm{Cu}(A)\otimes_\mathrm{Cu}\mathrm{Cu}(\mathcal{Z})$ enjoys the corresponding property of having a dense set of (equivalence classes of) projections. In the simple, nonelementary, real rank zero and stable rank one situation, our investigations lead us to identify almost unperforation for projections with the fact that tensoring with $\mathcal{Z}$ is inert at the level of the Cuntz semigroup.

math.OA↗
arXiv2016

Comparison Properties of the Cuntz semigroup and applications to C*-algebras

We study comparison properties in the category Cu aiming to lift results to the C*-algebraic setting. We introduce a new comparison property and relate it to both the CFP and $ω$-comparison. We show differences of all properties by providing examples, which suggest that the corona factorization property for C*-algebras might allow for both finite and infinite projections. In addition, we show that Rørdam's simple, nuclear C*-algebra with a finite and an infinite projection does not have the CFP.

math.OA↗
arXiv2013

Geometric Structure of Dimension Functions of Certain Continuous Fields

In this paper we study structural properties of the Cuntz semigroup and its functionals for continuous fields of C*-algebras over finite dimensional spaces. In a variety of cases, this leads to an answer to a conjecture posed by Blackadar and Handelman. Enroute to our results, we determine when the stable rank of continuous fields of C*-algebras over one dimensional spaces is one.

math.OA↗
arXiv2012

On certain multiplier projections

Let $\MCZK$, denote the multiplier algebra over $\CZK$, the algebra of continuous functions into the compact operators with spectrum the infinite product of two-spheres. We consider multiplier projections in $\MCZK$ of a certain diagonal form. We show that, while for each multiplier projection $Q$ of the special form, we have that $Q(x)\in\BH\setminus \KK$ for all $x\in \prod_{j=1}^\infty S^2$, the ideal generated by $Q$ in $\MCZK$ might be proper. We further show that the ideal generated by a multiplier projection of the special form is proper if and only if the projection is stably finite.

math.OA↗
arXiv2012

The Blackadar-Handelman theorem for non-unital C*-algebras

A well-known theorem of Blackadar and Handelman states that every unital stably finite C*-algebra has a bounded quasitrace. Rather strong generalizations of stable finiteness to the non-unital case can be obtained by either requiring the multiplier algebra to be stably finite, or alternatively requiring it to be at least stably not properly infinite. This paper deals with the question whether the Blackadar-Handelman result can be extended to the non-unital case with respect to these generalizations of stable finiteness. For suitably well-behaved C*-algebras there is a positive result, but none of the non-unital versions holds in full generality. Two examples of C*-algebras are constructed. The first one is a non-unital, stably commutative C*-algebra $A$ that contradicts the weakest possible generalization of the Blackadar-Handelman theorem: The multiplier algebras of all matrix algebras over $A$ are finite, while $A$ has no bounded quasitrace. The second example is a non-unital, simple C*-algebra $B$ that is stably non-stable, i.e. no matrix algebra over $B$ is a stable C*-algebra. In fact, the multiplier algebras over all matrix algebras of this C*-algebra are not properly infinite. Moreover, the C*-algebra $B$ has no bounded quasitrace and therefore gives a simple counterexample to a possible generalization of the Blackadar-Handelman theorem.

math.OA↗