arXiv subjects

Claudia Merger

Publications and source records attributed to Claudia Merger.

8 recordsLinked to original sources

arXiv2026

Local Coverage Governs Memorization in Diffusion Models

Memorization in diffusion models is often treated as a global property of the model or dataset. In practice, however, a single diffusion model can simultaneously generate both memorized and novel samples. Which training samples are most likely to be memorized? In this work, we show that memorization is governed by \emph{local data coverage}. Leveraging the connection between diffusion models and kernel density estimation (KDE), we derive a theoretical criterion that predicts whether a point is memorized based on the density of training data in its neighborhood and the size of the training dataset. In the high-dimensional limit, this leads to a sharp, local transition: regions of low coverage are dominated by isolated training samples, which are memorized, while dense regions support interpolation and generalization. We validate these predictions empirically, showing that memorization increases with local sparsity and that diffusion models exhibit a coexistence of memorized and novel samples within the same model. Extending this framework to multi-class settings, we further show that classes with higher intra-class sparsity (and thus lower local coverage) are more strongly memorized. Our results provide a local view of memorization in diffusion models, explaining when and where memorization occurs in terms of data geometry.

cond-mat.dis-nn↗

arXiv2026

A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights

Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within $n \ll N^3$ steps, but needs at least $n \gg N^3 \log^2{N}$ steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.

stat.ML↗

arXiv2026

A theory of learning data statistics in diffusion models, from easy to hard

While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on natural images exhibit a distributional simplicity bias, learning simple, pair-wise input statistics before specializing to higher-order correlations. We reproduce this behaviour in simple denoisers trained on a minimal data model, the mixed cumulant model, where we precisely control both pair-wise and higher-order correlations of the inputs. We identify a scalar invariant of the model that governs the sample complexity of learning pair-wise and higher-order correlations that we call the diffusion information exponent, in analogy to related invariants in different learning paradigms. Using this invariant, we prove that the denoiser learns simple, pair-wise statistics of the inputs at linear sample complexity, while more complex higher-order statistics, such as the fourth cumulant, require at least cubic sample complexity. We also prove that the sample complexity of learning the fourth cumulant is linear if pair-wise and higher-order statistics share a correlated latent structure. Our work describes a key mechanism for how diffusion models can learn distributions of increasing complexity.

stat.ML↗

arXiv2025

Characterizing Neural Manifolds' Properties and Curvatures using Normalizing Flows

Neuronal activity is found to lie on low-dimensional manifolds embedded within the high-dimensional neuron space. Variants of principal component analysis are frequently employed to assess these manifolds. These methods are, however, limited by assuming a Gaussian data distribution and a flat manifold. In this study, we introduce a method designed to satisfy three core objectives: (1) extract coordinated activity across neurons, described either statistically as correlations or geometrically as manifolds; (2) identify a small number of latent variables capturing these structures; and (3) offer an analytical and interpretable framework characterizing statistical properties by a characteristic function and describing manifold geometry through a collection of charts. To this end, we employ Normalizing Flows (NFs), which learn an underlying probability distribution of data by an invertible mapping between data and latent space. Their simplicity and ability to compute exact likelihoods distinguish them from other generative networks. We adjust the NF's training objective to distinguish between relevant (in manifold) and noise dimensions (out of manifold). Additionally, we find that different behavioral states align with the components of the latent Gaussian mixture model, enabling their treatment as distinct curved manifolds. Subsequently, we approximate the network for each mixture component with a quadratic mapping, allowing us to characterize both neural manifold curvature and non-Gaussian correlations among recording channels. Applying the method to recordings in macaque visual cortex, we demonstrate that state-dependent manifolds are curved and exhibit complex statistical dependencies. Our approach thus enables an expressive description of neural population activity, uncovering non-linear interactions among groups of neurons.

q-bio.NC↗

arXiv2025

Generalization Dynamics of Linear Diffusion Models

Diffusion models are powerful generative models that produce high-quality samples from complex data. While their infinite-data behavior is well understood, their generalization with finite data remains less clear. Classical learning theory predicts that generalization occurs at a sample complexity that is exponential in the dimension, far exceeding practical needs. We address this gap by analyzing diffusion models through the lens of data covariance spectra, which often follow power-law decays, reflecting the hierarchical structure of real data. To understand whether such a hierarchical structure can benefit learning in diffusion models, we develop a theoretical framework based on linear neural networks, congruent with a Gaussian hypothesis on the data. We quantify how the hierarchical organization of variance in the data and regularization impacts generalization. We find two regimes: When $N d$, we find that the sampling distributions of linear diffusion models approach their optimum (measured by the Kullback-Leibler divergence) linearly with $d/N$, independent of the specifics of the data distribution. Our work clarifies how sample complexity governs generalization in a simple model of diffusion-based generative models.

stat.ML↗

arXiv2023

Spurious self-feedback of mean-field predictions inflates infection curves

The susceptible-infected-recovered (SIR) model and its variants form the foundation of our understanding of the spread of diseases. Here, each agent can be in one of three states (susceptible, infected, or recovered), and transitions between these states follow a stochastic process. The probability of an agent becoming infected depends on the number of its infected neighbors, hence all agents are correlated. A common mean-field theory of the same stochastic process however, assumes that the agents are statistically independent. This leads to a self-feedback effect in the approximation: when an agent infects its neighbors, this infection may subsequently travel back to the original agent at a later time, leading to a self-infection of the agent which is not present in the underlying stochastic process. We here compute the first order correction to the mean-field assumption, which takes fluctuations up to second order in the interaction strength into account. We find that it cancels the self-feedback effect, leading to smaller infection rates. In the SIR model and in the SIRS model, the correction significantly improves predictions. In particular, it captures how sparsity dampens the spread of the disease: this indicates that reducing the number of contacts is more effective than predicted by mean-field models.

cond-mat.dis-nn↗

arXiv2023

Learning Interacting Theories from Data

One challenge of physics is to explain how collective properties arise from microscopic interactions. Indeed, interactions form the building blocks of almost all physical theories and are described by polynomial terms in the action. The traditional approach is to derive these terms from elementary processes and then use the resulting model to make predictions for the entire system. But what if the underlying processes are unknown? Can we reverse the approach and learn the microscopic action by observing the entire system? We use invertible neural networks (INNs) to first learn the observed data distribution. By the choice of a suitable nonlinearity for the neuronal activation function, we are then able to compute the action from the weights of the trained model; a diagrammatic language expresses the change of the action from layer to layer. This process uncovers how the network hierarchically constructs interactions via nonlinear transformations of pairwise relations. We test this approach on simulated data sets of interacting theories. The network consistently reproduces a broad class of unimodal distributions; outside this class, it finds effective theories that approximate the data statistics up to the third cumulant. We explicitly show how network depth and data quantity jointly improve the agreement between the learned and the true model. This work shows how to leverage the power of machine learning to transparently extract microscopic models from data.

cond-mat.dis-nn↗

arXiv2021

Global hierarchy vs. local structure: spurious self-feedback in scale-free networks

Networks with fat-tailed degree distributions are omnipresent across many scientific disciplines. Such systems are characterized by so-called hubs, specific nodes with high numbers of connections to other nodes. By this property, they are expected to be key to the collective network behavior, e.g., in Ising models on such complex topologies. This applies in particular to the transition into a globally ordered network state, which thereby proceeds in a hierarchical fashion, and with a non-trivial local structure. Standard mean-field theory of Ising models on scale-free networks underrates the presence of the hubs, while nevertheless providing remarkably reliable estimates for the onset of global order. Here, we expose that a spurious self-feedback effect, inherent to mean-field theory, underlies this apparent paradox. More specifically, we demonstrate that higher order interaction effects precisely cancel the self-feedback on the hubs, and we expose the importance of hubs for the distinct onset of local versus global order in the network. Due to the generic nature of our arguments, we expect the mechanism that we uncover for the archetypal case of Ising networks of the Barab\'asi-Albert type to be also relevant for other systems with a strongly hierarchical underlying network structure.

cond-mat.dis-nn↗