arXiv subjects

Raoul-Martin Memmesheimer

Publications and source records attributed to Raoul-Martin Memmesheimer.

17 recordsLinked to original sources

arXiv2026

Quadratic integrate-and-fire neurons exhibit less fragmented loss landscapes and outperform leaky integrate-and-fire neurons in spike-based gradient descent

The ability to train spiking neural networks is essential for modeling biological neural networks as well as for neuromorphic computing. However, for the extensively used leaky integrate-and-fire (LIF) neurons, arbitrarily small parameter changes can induce spike (dis)appearances that disrupt subsequent activity, leading to unstable neural representations and permanently silent neurons during exact spike-based gradient descent. Recent work shows that a class of neuron models, which includes the quadratic integrate-and-fire (QIF) neuron, avoids these discontinuities and enables continuous and even smooth spike-based gradient descent. However, it remains unclear whether these advantages translate into practice. Here, we demonstrate that they do so via a controlled comparison between networks of LIF and QIF neurons on the popular Spiking Heidelberg Digits dataset. Specifically, in a first step, we perform a thorough hyperparameter search to optimize both models, revealing a clear performance advantage of QIF neurons. In a second step, we visualize the loss and gradient landscapes. Consistent with their inferior performance, we find that the loss landscapes of LIF neurons, which are discontinuous, appear more fragmented and the related gradients more erratic. An analysis of the landscapes of single samples indicates that these features arise from changes in the temporal order of spikes, which often cause disruptive spike (dis)appearances. Overall, our results advocate replacing LIF neurons with neuron models exhibiting continuous spiking dynamics, such as QIF neurons, for gradient descent training.

cs.NE↗

arXiv2025

A generalized neural tangent kernel for surrogate gradient learning

State-of-the-art neural network training methods depend on the gradient of the network function. Therefore, they cannot be applied to networks whose activation functions do not have useful derivatives, such as binary and discrete-time spiking neural networks. To overcome this problem, the activation function's derivative is commonly substituted with a surrogate derivative, giving rise to surrogate gradient learning (SGL). This method works well in practice but lacks theoretical foundation. The neural tangent kernel (NTK) has proven successful in the analysis of gradient descent. Here, we provide a generalization of the NTK, which we call the surrogate gradient NTK, that enables the analysis of SGL. First, we study a naive extension of the NTK to activation functions with jumps, demonstrating that gradient descent for such activation functions is also ill-posed in the infinite-width limit. To address this problem, we generalize the NTK to gradient descent with surrogate derivatives, i.e., SGL. We carefully define this generalization and expand the existing key theorems on the NTK with mathematical rigor. Further, we illustrate our findings with numerical experiments. Finally, we numerically compare SGL in networks with sign activation function and finite width to kernel regression with the surrogate gradient NTK; the results confirm that the surrogate gradient NTK provides a good characterization of SGL.

stat.ML↗

arXiv2025

Smooth Exact Gradient Descent Learning in Spiking Neural Networks

Gradient descent prevails in artificial neural network training, but seems inept for spiking neural networks as small parameter changes can cause sudden, disruptive (dis-)appearances of spikes. Here, we demonstrate exact gradient descent based on continuously changing spiking dynamics. These are generated by neuron models whose spikes vanish and appear at the end of a trial, where it cannot influence subsequent dynamics. This also enables gradient-based spike addition and removal. We illustrate our scheme with various tasks and setups, including recurrent and deep, initially silent networks.

q-bio.NC↗

arXiv2020

Dynamical learning of dynamics

The ability of humans and animals to quickly adapt to novel tasks is difficult to reconcile with the standard paradigm of learning by slow synaptic weight modification. Here we show that fixed-weight neural networks can learn to generate required dynamics by imitation. After appropriate weight pretraining, the networks quickly and dynamically adapt to learn new tasks and thereafter continue to achieve them without further teacher feedback. We explain this ability and illustrate it with a variety of target dynamics, ranging from oscillatory trajectories to driven and chaotic dynamical systems.

q-bio.NC↗

arXiv2019

Dynamics and computation in mixed networks containing neurons that accelerate towards spiking

Networks in the brain consist of different types of neurons. Here we investigate the influence of neuron diversity on the dynamics, phase space structure and computational capabilities of spiking neural networks. We find that already a single neuron of a different type can qualitatively change the network dynamics and that mixed networks may combine the computational capabilities of ones with a single neuron type. We study inhibitory networks of concave leaky (LIF) and convex "anti-leaky" (XIF) integrate-and-fire neurons that generalize irregularly spiking non-chaotic LIF neuron networks. Endowed with simple conductance-based synapses for XIF neurons, our networks can generate a balanced state of irregular asynchronous spiking as well. We determine the voltage probability distributions and self-consistent firing rates assuming Poisson input with finite size spike impacts. Further, we compute the full spectrum of Lyapunov exponents (LEs) and the covariant Lyapunov vectors (CLVs) specifying the corresponding perturbation directions. We find that there is approximately one positive LE for each XIF neuron. This indicates in particular that a single XIF neuron renders the network dynamics chaotic. A simple mean-field approach, which can be justified by properties of the CLVs, explains the finding. As an application, we propose a spike-based computing scheme where our networks serve as computational reservoirs and their different stability properties yield different computational capabilities.

q-bio.NC↗

arXiv2019

From Single Neurons to Behavior in the Jellyfish Aurelia aurita

Jellyfish nerve nets provide insight into the origins of nervous systems, as both their taxonomic position and their evolutionary age imply that jellyfish resemble some of the earliest neuron-bearing, actively-swimming animals. Here we develop the first neuronal network model for the nerve nets of jellyfish. Specifically, we focus on the moon jelly Aurelia aurita and the control of its energy-efficient swimming motion. The proposed single neuron model disentangles the contributions of different currents to a spike. The network model identifies factors ensuring non-pathological activity and suggests an optimization for the transmission of signals. After modeling the jellyfish's muscle system and its bell in a hydrodynamic environment, we explore the swimming elicited by neural activity. We find that different delays between nerve net activations lead to well-controlled, differently directed movements. Our model bridges the scales from single neurons to behavior, allowing for a comprehensive understanding of jellyfish neural control.

q-bio.NC↗

arXiv2018

Growing Critical: Self-Organized Criticality in a Developing Neural System

Experiments in various neural systems found avalanches: bursts of activity with characteristics typical for critical dynamics. A possible explanation for their occurrence is an underlying network that self-organizes into a critical state. We propose a simple spiking model for developing neural networks, showing how these may "grow into" criticality. Avalanches generated by our model correspond to clusters of widely applied Hawkes processes. We analytically derive the cluster size and duration distributions and find that they agree with those of experimentally observed neuronal avalanches.

q-bio.NC↗

arXiv2016

Learning universal computations with spikes

Providing the neurobiological basis of information processing in higher animals, spiking neural networks must be able to learn a variety of complicated computations, including the generation of appropriate, possibly delayed reactions to inputs and the self-sustained generation of complex activity patterns, e.g.~for locomotion. Many such computations require previous building of intrinsic world models. Here we show how spiking neural networks may solve these different tasks. Firstly, we derive constraints under which classes of spiking neural networks lend themselves to substrates of powerful general purpose computing. The networks contain dendritic or synaptic nonlinearities and have a constrained connectivity. We then combine such networks with learning rules for outputs or recurrent connections. We show that this allows to learn even difficult benchmark tasks such as the self-sustained generation of desired low-dimensional chaotic dynamics or memory-dependent computations. Furthermore, we show how spiking networks can build models of external world systems and use the acquired knowledge to control them.

q-bio.NC↗

arXiv2015

Statistical physics of neural systems with non-additive dendritic coupling

How neurons process their inputs crucially determines the dynamics of biological and artificial neural networks. In such neural and neural-like systems, synaptic input is typically considered to be merely transmitted linearly or sublinearly by the dendritic compartments. Yet, single-neuron experiments report pronounced supralinear dendritic summation of sufficiently synchronous and spatially close-by inputs. Here, we provide a statistical physics approach to study the impact of such non-additive dendritic processing on single neuron responses and the performance of associative memory tasks in artificial neural networks. First, we compute the effect of random input to a neuron incorporating nonlinear dendrites. This approach is independent of the details of the neuronal dynamics. Second, we use those results to study the impact of dendritic nonlinearities on the network dynamics in a paradigmatic model for associative memory, both numerically and analytically. We find that dendritic nonlinearities maintain network convergence and increase the robustness of memory performance against noise. Interestingly, an intermediate number of dendritic branches is optimal for memory functionality.

physics.bio-ph↗

arXiv2014

Hub-activated signal transmission in complex networks

A wide range of networked systems exhibit highly connected nodes (hubs) as prominent structural elements. The functional roles of hubs in the collective nonlinear dynamics of many such networks, however, are not well understood. Here we propose that hubs in neural circuits may activate local signal transmission along sequences of specific subnetworks. Intriguingly, in contrast to previous suggestions of the functional roles of hubs, here not the hubs themselves, but non-hub subnetworks transfer the signals. The core mechanism relies on hubs and non-hubs providing activating feedback to each other. It may thus induce the propagation of specific pulse and rate signals in neuronal and other communication networks.

physics.bio-ph↗

arXiv2013

Guiding synchrony through random networks

Sparse random networks contain structures that can be considered as diluted feed-forward networks. Modeling of cortical circuits has shown that feed-forward structures, if strongly pronounced compared to the embedding random network, enable reliable signal transmission by propagating localized (sub-network) synchrony. This assumed prominence, however, is not experimentally observed in local cortical circuits. Here we show that nonlinear dendritic interactions as discovered in recent single neuron experiments, naturally enable guided synchrony propagation already in random recurrent neural networks exhibiting mildly enhanced, biologically plausible sub-structures.

q-bio.NC↗

arXiv2013

How Chaotic is the Balanced State?

Large sparse circuits of spiking neurons exhibit a balanced state of highly irregular activity under a wide range of conditions. It occurs likewise in sparsely connected random networks that receive excitatory external inputs and recurrent inhibition as well as in networks with mixed recurrent inhibition and excitation. Here we analytically investigate this irregular dynamics in finite networks keeping track of all individual spike times and the identities of individual neurons. For delayed, purely inhibitory interactions we show that the irregular dynamics is not chaotic but in fact stable. Moreover, we demonstrate that after long transients the dynamics converges towards periodic orbits and that every generic periodic orbit of these dynamical systems is stable. We investigate the collective irregular dynamics upon increasing the time scale of synaptic responses and upon iteratively replacing inhibitory by excitatory interactions. Whereas for small and moderate time scales as well as for few excitatory interactions, the dynamics stays stable, there is a smooth transition to chaos if the synaptic response becomes sufficiently slow (even in purely inhibitory networks) or the number of excitatory interactions becomes too large. These results indicate that chaotic and stable dynamics are equally capable of generating the irregular neuronal activity. More generally, chaos apparently is not essential for generating high irregularity of balanced activity, and we suggest that a mechanism different from chaos and stochasticity significantly contributes to irregular activity in cortical circuits.

q-bio.NC↗

arXiv2007

Stable Irregular Dynamics in Complex Neural Networks

For infinitely large sparse networks of spiking neurons mean field theory shows that a balanced state of highly irregular activity arises under various conditions. Here we analytically investigate the microscopic irregular dynamics in finite networks of arbitrary connectivity, keeping track of all individual spike times. For delayed, purely inhibitory interactions we demonstrate that the irregular dynamics is not chaotic but rather stable and convergent towards periodic orbits. Moreover, every generic periodic orbit of these dynamical systems is stable. These results highlight that chaotic and stable dynamics are equally capable of generating irregular activity.

cond-mat.dis-nn↗

arXiv2006

Designing Complex Networks

We suggest a new perspective of research towards understanding the relations between structure and dynamics of a complex network: Can we design a network, e.g. by modifying the features of units or interactions, such that it exhibits a desired dynamics? Here we present a case study where we positively answer this question analytically for networks of spiking neural oscillators. First, we present a method of finding the set of all networks (defined by all mutual coupling strengths) that exhibit an arbitrary given periodic pattern of spikes as an invariant solution. In such a pattern all the spike times of all the neurons are exactly predefined. The method is very general as it covers networks of different types of neurons, excitatory and inhibitory couplings, interaction delays that may be heterogeneously distributed, and arbitrary network connectivities. Second, we show how to design networks if further restrictions are imposed, for instance by predefining the detailed network connectivity. We illustrate the applicability of the method by examples of Erdös-Rényi and power-law random networks. Third, the method can be used to design networks that optimize network properties. To illustrate this idea, we design networks that exhibit a predefined pattern dynamics while at the same time minimizing the networks' wiring costs.

q-bio.NC↗

arXiv2006

Designing the Dynamics of Spiking Neural Networks

Precise timing of spikes and temporal locking are key elements of neural computation. Here we demonstrate how even strongly heterogeneous, deterministic neural networks with delayed interactions and complex topology can exhibit periodic patterns of spikes that are precisely timed. We develop an analytical method to find the set of all networks exhibiting a predefined pattern dynamics. Such patterns may be arbitrarily long and of complicated temporal structure. We point out that the same pattern can exist in very different networks and have different stability properties.

q-bio.NC↗

arXiv2004

Third post-Newtonian constrained canonical dynamics for binary point masses in harmonic coordinates

The conservative dynamics of two point masses given in harmonic coordinates up to the third post-Newtonian (3pN) order is treated within the framework of constrained canonical dynamics. A representation of the approximate Poincaré algebra is constructed with the aid of Dirac brackets. Uniqueness of the generators of the Poincaré group resp. the integrals of motion is achieved by imposing their action on the point mass coordinates to be identical with that of the usual infinitesimal Poincaré transformations. The second post-Coulombian approximation to the dynamics of two point charges as predicted by Feynman-Wheeler electrodynamics in Lorentz gauge is treated similarly.

gr-qc↗

arXiv2004

Third post-Newtonian accurate generalized quasi-Keplerian parametrization for compact binaries in eccentric orbits

We present Keplerian-type parametrization for the solution of third post-Newtonian (3PN) accurate equations of motion for two non-spinning compact objects moving in an eccentric orbit. The orbital elements of the parametrization are explicitly given in terms of the 3PN accurate conserved orbital energy and angular momentum in both Arnowitt, Deser, and Misner-type and harmonic coordinates. Our representation will be required to construct post-Newtonian accurate `ready to use' search templates for the detection of gravitational waves from compact binaries in inspiralling eccentric orbits. Due to the presence of certain 3PN accurate gauge invariant orbital elements, the parametrization should be useful to analyze the compatibility of general relativistic numerical simulations involving compact binaries with the corresponding post-Newtonian descriptions. If required, the present parametrization will also be needed to compute post-Newtonian corrections to the currently employed `timing formula' for the radio observations of relativistic binary pulsars.

gr-qc↗