arXiv subjects

Anna Levina

Publications and source records attributed to Anna Levina.

At least 19 recordsLinked to original sources

arXiv2026

Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons

Cortical neurons are complex, multi-timescale processors wired into recurrent circuits, shaped by long evolutionary pressure under stringent biological constraints. Mainstream machine learning, by contrast, predominantly builds models from extremely simple units, a default inherited from early neural-network theory. We treat this as a normative architectural question. How should one split a fixed parameter budget $P$ between the number of units $N$, per-unit effective complexity $k_e$, and per-unit connectivity $k_c$? What controls the optimal allocation? This calls for a model in which per-unit complexity can be tuned independently of width and connectivity. Accordingly, we introduce the ELM Network, whose recurrent layer is built from Expressive Leaky Memory (ELM) neurons, chosen to mirror functional components of cortical neurons. The architecture allows for individually adjusting $N$, $k_e$, and $k_c$ and trains stably across orders of magnitude in scale. We evaluate the model on two qualitatively different sequence benchmarks: the neuromorphic SHD-Adding task and Enwik8 character-level language modeling. Performance improves monotonically along each of the three axes individually. Under a fixed budget, a clear non-trivial optimum emerges in their tradeoff, and larger budgets favor both more and more complex neurons. A closed-form information-theoretic model captures these tradeoffs and attributes the diminishing returns at two ends to: per-neuron signal-to-noise saturation and across-neuron redundancy. A hyperparameter sweep spanning three orders of magnitude in trainable parameters traces a near-Pareto-frontier scaling law consistent with the framework. This suggests that the simple-unit default in ML is not obviously optimal once this tradeoff surface is probed, and offers a normative lens on cortex's reliance on complex spatio-temporal integrators.

cs.LG↗

arXiv2026

Diffusion of Neuromodulators for Temporal Credit Assignment

Biological learning achieves temporal credit assignment despite sparse and imprecise feedback, often relying on neuromodulatory signals acting over space and time. Here, we introduce a learning mechanism in which error information diffuses locally through the network, similar to volume transmission of neuromodulators. This distributed modulation allows neurons to learn even in the absence of direct feedback, using the local concentration of the diffusing credit signal. Applied to recurrent spiking neural networks with sparse feedback connectivity, diffusive credit signaling improves learning across three benchmark tasks. Using eligibility propagation as a baseline learning mechanism, we show how diffusion-based modulation can provide a plausible mechanism for credit assignment in sparsely connected neural circuits.

q-bio.NC↗

arXiv2026

Finite integration time can shift optimal sensitivity away from criticality

Sensitivity to small changes in the environment is crucial for many real-world tasks, enabling living and artificial systems to make correct behavioral decisions. It has been shown that such sensitivity is maximized when a system operates near the critical point of a phase transition. However, proximity to criticality introduces large fluctuations and diverging timescales. Hence, to leverage the maximal sensitivity, it would require impractically long integration periods. Here, we analytically and computationally demonstrate how the optimal tuning of a recurrent neural network is determined given a finite integration time. Rather than maximizing the theoretically available sensitivity, we find networks attain different sensitivities depending on the available time. Consequently, the optimal dynamic regime can shift away from criticality when integration times are finite, highlighting the necessity of incorporating finite-time considerations into studies of information processing.

cond-mat.dis-nn↗

arXiv2024

Neural timescales from a computational perspective

Neural activity fluctuates over a wide range of timescales within and across brain areas. Experimental observations suggest that diverse neural timescales reflect information in dynamic environments. However, how timescales are defined and measured from brain recordings vary across the literature. Moreover, these observations do not specify the mechanisms underlying timescale variations, nor whether specific timescales are necessary for neural computation and brain function. Here, we synthesize three directions where computational approaches can distill the broad set of empirical observations into quantitative and testable theories: We review (i) how different data analysis methods quantify timescales across distinct behavioral states and recording modalities, (ii) how biophysical models provide mechanistic explanations for the emergence of diverse timescales, and (iii) how task-performing networks and machine learning models uncover the functional relevance of neural timescales. This integrative computational perspective thus complements experimental investigations, providing a holistic view on how neural timescales reflect the relationship between brain structure, dynamics, and behavior.

q-bio.NC↗

arXiv2024

Modular Growth of Hierarchical Networks: Efficient, General, and Robust Curriculum Learning

Structural modularity is a pervasive feature of biological neural networks, which have been linked to several functional and computational advantages. Yet, the use of modular architectures in artificial neural networks has been relatively limited despite early successes. Here, we explore the performance and functional dynamics of a modular network trained on a memory task via an iterative growth curriculum. We find that for a given classical, non-modular recurrent neural network (RNN), an equivalent modular network will perform better across multiple metrics, including training time, generalizability, and robustness to some perturbations. We further examine how different aspects of a modular network's connectivity contribute to its computational capability. We then demonstrate that the inductive bias introduced by the modular topology is strong enough for the network to perform well even when the connectivity within modules is fixed and only the connections between modules are trained. Our findings suggest that gradual modular growth of RNNs could provide advantages for learning increasingly complex tasks on evolutionary timescales, and help build more scalable and compressible artificial networks.

cs.NE↗

arXiv2024

Learning with 3D rotations, a hitchhiker's guide to SO(3)

Many settings in machine learning require the selection of a rotation representation. However, choosing a suitable representation from the many available options is challenging. This paper acts as a survey and guide through rotation representations. We walk through their properties that harm or benefit deep learning with gradient-based optimization. By consolidating insights from rotation-based learning, we provide a comprehensive overview of learning functions with rotation representations. We provide guidance on selecting representations based on whether rotations are in the model's input or output and whether the data primarily comprises small angles.

cs.LG↗

arXiv2024

Network bottlenecks and task structure control the evolution of interpretable learning rules in a foraging agent

Developing reliable mechanisms for continuous local learning is a central challenge faced by biological and artificial systems. Yet, how the environmental factors and structural constraints on the learning network influence the optimal plasticity mechanisms remains obscure even for simple settings. To elucidate these dependencies, we study meta-learning via evolutionary optimization of simple reward-modulated plasticity rules in embodied agents solving a foraging task. We show that unconstrained meta-learning leads to the emergence of diverse plasticity rules. However, regularization and bottlenecks to the model help reduce this variability, resulting in interpretable rules. Our findings indicate that the meta-learning of plasticity rules is very sensitive to various parameters, with this sensitivity possibly reflected in the learning rules found in biological networks. When included in models, these dependencies can be used to discover potential objective functions and details of biological learning via comparisons with experimental observations.

q-bio.NC↗

arXiv2024

Revising clustering and small-worldness in brain networks

As more connectome data become available, the question of how to best analyse the structure of biological neural networks becomes increasingly pertinent. In brain networks, knowing that two areas are connected is often not sufficient, as the directionality and weight of the connection affect the dynamics in crucial ways. Still, the methods commonly used to estimate network properties, such as clustering and small-worldness, usually disregard features encoded in the directionality and strength of network connections. To address this issue, we propose using fully-weighted and directed clustering measures that provide higher sensitivity to non-random structural features. Using artificial networks, we demonstrate the problems with methods routinely used in the field and how fully-weighted and directed methods can alleviate them. Specifically, we highlight their robustness to noise and their ability to address thresholding issues, particularly in inferred networks. We further apply our method to the connectomes of different species and uncover regularities and correlations between neuronal structures and functions that cannot be detected with traditional clustering metrics. Finally, we extend the notion of small-worldness in brain networks to account for weights and directionality and show that some connectomes can no longer be considered ``small-world''. Overall, our study makes a case for a combined use of fully-weighted and directed measures to deal with the variability of brain networks and suggests the presence of complex patterns in neural connectivity that can only be revealed using such methods.

q-bio.NC↗

arXiv2023

When to be critical? Performance and evolvability in different regimes of neural Ising agents

It has long been hypothesized that operating close to the critical state is beneficial for natural, artificial and their evolutionary systems. We put this hypothesis to test in a system of evolving foraging agents controlled by neural networks that can adapt agents' dynamical regime throughout evolution. Surprisingly, we find that all populations that discover solutions, evolve to be subcritical. By a resilience analysis, we find that there are still benefits of starting the evolution in the critical regime. Namely, initially critical agents maintain their fitness level under environmental changes (for example, in the lifespan) and degrade gracefully when their genome is perturbed. At the same time, initially subcritical agents, even when evolved to the same fitness, are often inadequate to withstand the changes in the lifespan and degrade catastrophically with genetic perturbations. Furthermore, we find the optimal distance to criticality depends on the task complexity. To test it we introduce a hard and simple task: for the hard task, agents evolve closer to criticality whereas more subcritical solutions are found for the simple task. We verify that our results are independent of the selected evolutionary mechanisms by testing them on two principally different approaches: a genetic algorithm and an evolutionary strategy. In summary, our study suggests that although optimal behaviour in the simple task is obtained in a subcritical regime, initializing near criticality is important to be efficient at finding optimal solutions for new tasks of unknown complexity.

cs.NE↗

arXiv2023

Emergent mechanisms for long timescales depend on training curriculum and affect performance in memory tasks

Recurrent neural networks (RNNs) in the brain and in silico excel at solving tasks with intricate temporal dependencies. Long timescales required for solving such tasks can arise from properties of individual neurons (single-neuron timescale, $\tau$, e.g., membrane time constant in biological neurons) or recurrent interactions among them (network-mediated timescale). However, the contribution of each mechanism for optimally solving memory-dependent tasks remains poorly understood. Here, we train RNNs to solve $N$-parity and $N$-delayed match-to-sample tasks with increasing memory requirements controlled by $N$ by simultaneously optimizing recurrent weights and $\tau$s. We find that for both tasks RNNs develop longer timescales with increasing $N$, but depending on the learning objective, they use different mechanisms. Two distinct curricula define learning objectives: sequential learning of a single-$N$ (single-head) or simultaneous learning of multiple $N$s (multi-head). Single-head networks increase their $\tau$ with $N$ and are able to solve tasks for large $N$, but they suffer from catastrophic forgetting. However, multi-head networks, which are explicitly required to hold multiple concurrent memories, keep $\tau$ constant and develop longer timescales through recurrent connectivity. Moreover, we show that the multi-head curriculum increases training speed and network stability to ablations and perturbations, and allows RNNs to generalize better to tasks beyond their training regime. This curriculum also significantly improves training GRUs and LSTMs for large-$N$ tasks. Our results suggest that adapting timescales to task requirements via recurrent interactions allows learning more complex objectives and improves the RNN's performance.

cs.NE↗

arXiv2023

Available observation time regulates optimal balance between sensitivity and confidence

Tasks that require information about the world imply a trade-off between the time spent on observation and the variance of the response. In particular, fast decisions need to rely on uncertain information. However, standard estimates of information processing capabilities, such as the dynamic range, are defined based on mean values that assume infinite observation times. Here, we show that limiting the observation time results in distributions of responses whose variance increases with the temporal correlations in a system and, importantly, affects a system's confidence in distinguishing inputs and thereby making decisions. To quantify the ability to distinguish features of an input, we propose several measures and demonstrate them on the prime example of a recurrent neural network that represents an input rate by a response firing averaged over a finite observation time. We show analytically and in simulations that the optimal tuning of the network depends on the available observation time, implying that tasks require a ``useful'' rather than maximal sensitivity. Interestingly, this shifts the optimal dynamic regime from critical to subcritical for finite observation times and highlights the importance of incorporating the finite observation times concept in future studies of information processing capabilities in a principled manner.

q-bio.NC↗

arXiv2023

The Expressive Leaky Memory Neuron: an Efficient and Expressive Phenomenological Neuron Model Can Solve Long-Horizon Tasks

Biological cortical neurons are remarkably sophisticated computational devices, temporally integrating their vast synaptic input over an intricate dendritic tree, subject to complex, nonlinearly interacting internal biological processes. A recent study proposed to characterize this complexity by fitting accurate surrogate models to replicate the input-output relationship of a detailed biophysical cortical pyramidal neuron model and discovered it needed temporal convolutional networks (TCN) with millions of parameters. Requiring these many parameters, however, could stem from a misalignment between the inductive biases of the TCN and cortical neuron's computations. In light of this, and to explore the computational implications of leaky memory units and nonlinear dendritic processing, we introduce the Expressive Leaky Memory (ELM) neuron model, a biologically inspired phenomenological model of a cortical neuron. Remarkably, by exploiting such slowly decaying memory-like hidden states and two-layered nonlinear integration of synaptic input, our ELM neuron can accurately match the aforementioned input-output relationship with under ten thousand trainable parameters. To further assess the computational ramifications of our neuron design, we evaluate it on various tasks with demanding temporal structures, including the Long Range Arena (LRA) datasets, as well as a novel neuromorphic dataset based on the Spiking Heidelberg Digits dataset (SHD-Adding). Leveraging a larger number of memory units with sufficiently long timescales, and correspondingly sophisticated synaptic integration, the ELM neuron displays substantial long-range processing capabilities, reliably outperforming the classic Transformer or Chrono-LSTM architectures on LRA, and even solving the Pathfinder-X task with over 70% accuracy (16k context length).

cs.NE↗

arXiv2023

Locally adaptive cellular automata for goal-oriented self-organization

The essential ingredient for studying the phenomena of emergence is the ability to generate and manipulate emergent systems that span large scales. Cellular automata are the model class particularly known for their effective scalability but are also typically constrained by fixed local rules. In this paper, we propose a new model class of adaptive cellular automata that allows for the generation of scalable and expressive models. We show how to implement computation-effective adaptation by coupling the update rule of the cellular automaton with itself and the system state in a localized way. To demonstrate the applications of this approach, we implement two different emergent models: a self-organizing Ising model and two types of plastic neural networks, a rate and spiking model. With the Ising model, we show how coupling local/global temperatures to local/global measurements can tune the model to stay in the vicinity of the critical temperature. With the neural models, we reproduce a classical balanced state in large recurrent neuronal networks with excitatory and inhibitory neurons and various plasticity mechanisms. Our study opens multiple directions for studying collective behavior and emergence.

cs.NE↗

arXiv2023

Environmental variability and network structure determine the optimal plasticity mechanisms in embodied agents

The evolutionary balance between innate and learned behaviors is highly intricate, and different organisms have found different solutions to this problem. We hypothesize that the emergence and exact form of learning behaviors is naturally connected with the statistics of environmental fluctuations and tasks an organism needs to solve. Here, we study how different aspects of simulated environments shape an evolved synaptic plasticity rule in static and moving artificial agents. We demonstrate that environmental fluctuation and uncertainty control the reliance of artificial organisms on plasticity. Interestingly, the form of the emerging plasticity rule is additionally determined by the details of the task the artificial organisms are aiming to solve. Moreover, we show that co-evolution between static connectivity and interacting plasticity mechanisms in distinct sub-networks changes the function and form of the emerging plasticity rules in embodied agents performing a foraging task.

q-bio.NC↗

arXiv2022

Topology-dependent coalescence controls scaling exponents in finite networks

Multiple studies of neural avalanches across different data modalities led to the prominent hypothesis that the brain operates near a critical point. The observed exponents often indicate the mean-field directed-percolation universality class, leading to the fully-connected or random network models to study the avalanche dynamics. However, the cortical networks have distinct non-random features and spatial organization that is known to affect the critical exponents. Here we show that distinct empirical exponents arise in networks with different topology and depend on the network size. In particular, we find apparent scale-free behavior with mean-field exponents appearing as quasi-critical dynamics in structured networks. This quasi-critical dynamics cannot be easily discriminated from an actual critical point in small networks. We find that the local coalescence in activity dynamics can explain the distinct exponents. Therefore, both topology and system size should be considered when assessing criticality from empirical observables.

q-bio.NC↗

arXiv2022

Tackling the subsampling problem to infer collective properties from limited data

Complex systems are fascinating because their rich macroscopic properties emerge from the interaction of many simple parts. Understanding the building principles of these emergent phenomena in nature requires assessing natural complex systems experimentally. However, despite the development of large-scale data-acquisition techniques, experimental observations are often limited to a tiny fraction of the system. This spatial subsampling is particularly severe in neuroscience, where only a tiny fraction of millions or even billions of neurons can be individually recorded. Spatial subsampling may lead to significant systematic biases when inferring the collective properties of the entire system naively from a subsampled part. To overcome such biases, powerful mathematical tools have been developed in the past. In this perspective, we overview some issues arising from subsampling and review recently developed approaches to tackle the subsampling problem. These approaches enable one to assess, e.g., graph structures, collective dynamics of animals, neural network activity, or the spread of disease correctly from observing only a tiny fraction of the system. However, our current approaches are still far from having solved the subsampling problem in general, and hence we conclude by outlining what we believe are the main open challenges. Solving these challenges alongside the development of large-scale recording techniques will enable further fundamental insights into the working of complex and living systems.

q-bio.NC↗

arXiv2022

Spatial and temporal correlations in neural networks with structured connectivity

Correlated fluctuations in the activity of neural populations reflect the network's dynamics and connectivity. The temporal and spatial dimensions of neural correlations are interdependent. However, prior theoretical work mainly analyzed correlations in either spatial or temporal domains, oblivious to their interplay. We show that the network dynamics and connectivity jointly define the spatiotemporal profile of neural correlations. We derive analytical expressions for pairwise correlations in networks of binary units with spatially arranged connectivity in one and two dimensions. We find that spatial interactions among units generate multiple timescales in auto- and cross-correlations. Each timescale is associated with fluctuations at a particular spatial frequency, making a hierarchical contribution to the correlations. External inputs can modulate the correlation timescales when spatial interactions are nonlinear, and the modulation effect depends on the operating regime of network dynamics. These theoretical results open new ways to relate connectivity and dynamics in cortical networks via measurements of spatiotemporal neural correlations.

q-bio.NC↗

arXiv2022

Estimations by stable motions and applications

We propose a nonparametric parameter estimation of confidence intervals when the underlying has large or infinite variance. We explain the method by a simple numerical example and provide an application to estimate the coupling strength in neuronal networks.

math.ST↗