arXiv subjects

Alexander van Meegen

Publications and source records attributed to Alexander van Meegen.

9 recordsLinked to original sources

arXiv2026

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.

cs.LG↗

arXiv2024

Connectivity structure and dynamics of nonlinear recurrent neural networks

Studies of the dynamics of nonlinear recurrent neural networks often assume independent and identically distributed couplings, but large-scale connectomics data indicate that biological neural circuits exhibit markedly different connectivity properties. These include rapidly decaying singular-value spectra and structured singular-vector overlaps. Here, we develop a theory to analyze how these forms of structure shape high-dimensional collective activity in nonlinear recurrent neural networks. We first introduce the random-mode model, a random-matrix ensemble related to the singular-value decomposition that enables control over the spectrum and right-left mode overlaps. Then, using a novel path-integral calculation, we derive analytical expressions that reveal how connectivity structure affects features of collective dynamics: the dimension of activity, which quantifies the number of high-variance collective-activity fluctuations, and the temporal correlations that characterize the timescales of these fluctuations. We show that connectivity structure can be invisible in single-neuron activities while dramatically shaping collective activity. Furthermore, despite the nonlinear, high-dimensional nature of these networks, the dimension of activity depends on just two connectivity parameters -- the variance of the couplings and the effective rank of the coupling matrix, which quantifies the number of dominant rank-one connectivity components. We contrast the effects of single-neuron heterogeneity and low dimensional connectivity, making predictions about how z-scoring data affects the dimension of activity. Finally, we demonstrate the presence of structured overlaps between left and right modes in the Drosophila connectome, incorporate them into the theory, and show how they further shape collective dynamics.

q-bio.NC↗

arXiv2024

Coding schemes in neural networks learning classification tasks

Neural networks posses the crucial ability to generate meaningful representations of task-dependent features. Indeed, with appropriate scaling, supervised learning in neural networks can result in strong, task-dependent feature learning. However, the nature of the emergent representations, which we call the `coding scheme', is still unclear. To understand the emergent coding scheme, we investigate fully-connected, wide neural networks learning classification tasks using the Bayesian framework where learning shapes the posterior distribution of the network weights. Consistent with previous findings, our analysis of the feature learning regime (also known as `non-lazy', `rich', or `mean-field' regime) shows that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity. In linear networks, an analog coding scheme of the task emerges. Despite the strong representations, the mean predictor is identical to the lazy case. In nonlinear networks, spontaneous symmetry breaking leads to either redundant or sparse coding schemes. Our findings highlight how network properties such as scaling of weights and neuronal nonlinearity can profoundly influence the emergent representations.

cs.LG↗

arXiv2024

Linking Network and Neuron-level Correlations by Renormalized Field Theory

It is frequently hypothesized that cortical networks operate close to a critical point. Advantages of criticality include rich dynamics well-suited for computation and critical slowing down, which may offer a mechanism for dynamic memory. However, mean-field approximations, while versatile and popular, inherently neglect the fluctuations responsible for such critical dynamics. Thus, a renormalized theory is necessary. We consider the Sompolinsky-Crisanti-Sommers model which displays a well studied chaotic as well as a magnetic transition. Based on the analogue of a quantum effective action, we derive self-consistency equations for the first two renormalized Greens functions. Their self-consistent solution reveals a coupling between the population level activity and single neuron heterogeneity. The quantitative theory explains the population autocorrelation function, the single-unit autocorrelation function with its multiple temporal scales, and cross correlations.

cond-mat.dis-nn↗

arXiv2023

The Distribution of Unstable Fixed Points in Chaotic Neural Networks

We analytically determine the number and distribution of fixed points in a canonical model of a chaotic neural network. This distribution reveals that fixed points and dynamics are confined to separate shells in phase space. Furthermore, the distribution enables us to determine the eigenvalue spectra of the Jacobian at the fixed points. Despite the radial separation of fixed points and dynamics, we find that nearby fixed points act as partially attracting landmarks for the dynamics.

cond-mat.dis-nn↗

arXiv2022

Unified field theoretical approach to deep and recurrent neuronal networks

Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, $n\to\infty$. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in $1/n$ and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths $n$.

cond-mat.dis-nn↗

arXiv2021

A Microscopic Theory of Intrinsic Timescales in Spiking Neural Networks

A complex interplay of single-neuron properties and the recurrent network structure shapes the activity of cortical neurons. The single-neuron activity statistics differ in general from the respective population statistics, including spectra and, correspondingly, autocorrelation times. We develop a theory for self-consistent second-order single-neuron statistics in block-structured sparse random networks of spiking neurons. In particular, the theory predicts the neuron-level autocorrelation times, also known as intrinsic timescales, of the neuronal activity. The theory is based on an extension of dynamic mean-field theory from rate networks to spiking networks, which is validated via simulations. It accounts for both static variability, e.g. due to a distributed number of incoming synapses per neuron, and temporal fluctuations of the input. We apply the theory to balanced random networks of generalized linear model neurons, balanced random networks of leaky integrate-and-fire neurons, and a biologically constrained network of leaky integrate-and-fire neurons. For the generalized linear model network with an error function nonlinearity, a novel analytical solution of the colored noise problem allows us to obtain self-consistent firing rate distributions, single-neuron power spectra, and intrinsic timescales. For the leaky integrate-and-fire networks, we derive an approximate analytical solution of the colored noise problem, based on the Stratonovich approximation of the Wiener-Rice series and a novel analytical solution for the free upcrossing statistics. Again closing the system self-consistently, in the fluctuation-driven regime this approximation yields reliable estimates of the mean firing rate and its variance across neurons, the inter-spike interval distribution, the single-neuron power spectra, and intrinsic timescales.

q-bio.NC↗

arXiv2021

Large Deviations Approach to Random Recurrent Neuronal Networks: Parameter Inference and Fluctuation-Induced Transitions

We here unify the field theoretical approach to neuronal networks with large deviations theory. For a prototypical random recurrent network model with continuous-valued units, we show that the effective action is identical to the rate function and derive the latter using field theory. This rate function takes the form of a Kullback-Leibler divergence which enables data-driven inference of model parameters and calculation of fluctuations beyond mean-field theory. Lastly, we expose a regime with fluctuation-induced transitions between mean-field solutions.

cond-mat.dis-nn↗

arXiv2020

Usage and Scaling of an Open-Source Spiking Multi-Area Model of Monkey Cortex

We are entering an age of `big' computational neuroscience, in which neural network models are increasing in size and in numbers of underlying data sets. Consolidating the zoo of models into large-scale models simultaneously consistent with a wide range of data is only possible through the effort of large teams, which can be spread across multiple research institutions. To ensure that computational neuroscientists can build on each other's work, it is important to make models publicly available as well-documented code. This chapter describes such an open-source model, which relates the connectivity structure of all vision-related cortical areas of the macaque monkey with their resting-state dynamics. We give a brief overview of how to use the executable model specification, which employs NEST as simulation engine, and show its runtime scaling. The solutions found serve as an example for organizing the workflow of future models from the raw experimental data to the visualization of the results, expose the challenges, and give guidance for the construction of ICT infrastructure for neuroscience.

q-bio.NC↗