Searcharxiv⌕ Search

arXiv subjects

Pau Vilimelis Aceituno

Publications and source records attributed to Pau Vilimelis Aceituno.

16 recordsLinked to original sources
arXiv2026

Temporal horizons in forecasting: a performance-learnability trade-off

When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.

cs.LG↗
arXiv2026

Forward and reverse delay-driven hippocampal replay without symmetric plasticity

Hippocampal replay is a phenomenon observed in mammals and songbirds where neural activation sequences experienced during wakeful periods are repeated during rest or sleep. This mechanism is believed to play a crucial role in episodic memory consolidation, retrieval, and planning. Interestingly, replay can occur in both forward and reverse temporal orders and across a wide range of increased speeds. The learning of activation sequences has traditionally been modeled by temporally asymmetric Hebbian plasticity rules, which explain replay as a chain of activity. The inability of such rules to strengthen backward connections has led to the widespread belief that they cannot account for bidirectional replay. Existing theoretical work therefore explains reverse replay through symmetric connections, which would be formed through symmetric plasticity rules. We propose a delay-coupled neural field model of CA3, where activity propagates as a wave in a ring, and replays occur both forwards and backwards at different speeds. Our model can learn bidirectional replay after a single noisy exposure to a stimulus with either symmetric or asymmetric plasticity. We derive a low-dimensional description of the dynamics and analyze the stability of possible replay speeds.

q-bio.NC↗
arXiv2026

A combination of noise and bilateral filters achieve supralinear and scalable adversarial robustness in CNNs

The vulnerability of deep neural networks to adversarial examples poses a significant challenge for real-world deployment. Existing techniques to enhance deep network robustness rely on adversarial training, an approach that is powerful but computationally intensive and typically tailored to specific attack types. To address these limitations, existing works have explored techniques such as adding gaussian noise or filtering images, both of which can boost the network robustness to various adversarial attacks, albeit modestly. Here, we theoretically demonstrate that these two approaches enhance robustness against adversarial attacks through complementary mechanisms, resulting in supralinear robustness when combined. Building on this insight, we experimentally show that a simple preprocessor combining Gaussian noise and bilateral filtering yields supralinear improvements in adversarial robustness with minimal computational cost. Next, we combine our preprocessor with adversarial training and test on RobustBench to assess its supralinear improvement over state-of-the-art defenses. First, this combination ranks second on AutoAttack and third overall, while using only $\sim$35% of the training FLOPs, using a model with $\sim$50% less parametets, trained with $\sim$33% of the epochs and $\sim$15% the data compared to state-of-the-art defenses. Second, our method scales efficiently, matching the accuracy of competing models with roughly 2-8x less total compute across 3 orders of magnitude. Overall, our approach provides a principled and easily integrable framework for enhancing adversarial robustness, offering negligible computational overhead and a simple yet theoretically grounded design.

cs.LG↗
arXiv2026

Continual Learning through Control Minimization

Catastrophic forgetting remains a fundamental challenge for neural networks when tasks are trained sequentially. In this work, we reformulate continual learning as a control problem where learning and preservation signals compete within neural activity dynamics. We convert regularization penalties into preservation signals that protect prior-task representations. Learning then proceeds by minimizing the control effort required to integrate new tasks while competing with the preservation of prior tasks. At equilibrium, the neural activities produce weight updates that implicitly encode the full prior-task curvature, a property we term the continual-natural gradient, requiring no explicit curvature storage. Experiments confirm that our learning framework recovers true prior-task curvature and enables task discrimination, outperforming existing methods on standard benchmarks without replay.

cs.LG↗
arXiv2025

The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training

Self-attention is essential to Transformer architectures, yet how information is embedded in the self-attention matrices and how different objective functions impact this process remains unclear. We present a mathematical framework to analyze self-attention matrices by deriving the structures governing their weight updates. Using this framework, we demonstrate that bidirectional training induces symmetry in the weight matrices, while autoregressive training results in directionality and column dominance. Our theoretical findings are validated across multiple Transformer models - including ModernBERT, GPT, LLaMA3, and Mistral - and input modalities like text, vision, and audio. Finally, we apply these insights by showing that symmetric initialization improves the performance of encoder-only models on language tasks. This mathematical analysis offers a novel theoretical perspective on how information is embedded through self-attention, thereby improving the interpretability of Transformer models.

cs.LG↗
arXiv2025

Two types of pyramidal cells and their role in temporal processing

Recent work has provided new insights into the temporal specialization of Intratelencephalic (IT) and Pyramidal tract neurons (PT). However, functional and anatomical differences of IT and PT have not been connected yet. This perspective article contributes by highlighting empirical studies about the connectivity of IT and PT as well as their specialization in sensory and motor processing in diverse brain regions. We further review conceptual models that would align with the connectivity motif. We conclude that further research is needed to understand the role of the unidirectional connectivity from IT to PT in temporal processing and suggest concrete experiments.

q-bio.NC↗
arXiv2024

The Role of Temporal Hierarchy in Spiking Neural Networks

Spiking Neural Networks (SNNs) have the potential for rich spatio-temporal signal processing thanks to exploiting both spatial and temporal parameters. The temporal dynamics such as time constants of the synapses and neurons and delays have been recently shown to have computational benefits that help reduce the overall number of parameters required in the network and increase the accuracy of the SNNs in solving temporal tasks. Optimizing such temporal parameters, for example, through gradient descent, gives rise to a temporal architecture for different problems. As has been shown in machine learning, to reduce the cost of optimization, architectural biases can be applied, in this case in the temporal domain. Such inductive biases in temporal parameters have been found in neuroscience studies, highlighting a hierarchy of temporal structure and input representation in different layers of the cortex. Motivated by this, we propose to impose a hierarchy of temporal representation in the hidden layers of SNNs, highlighting that such an inductive bias improves their performance. We demonstrate the positive effects of temporal hierarchy in the time constants of feed-forward SNNs applied to temporal tasks (Multi-Time-Scale XOR and Keyword Spotting, with a benefit of up to 4.1% in classification accuracy). Moreover, we show that such architectural biases, i.e. hierarchy of time constants, naturally emerge when optimizing the time constants through gradient descent, initialized as homogeneous values. We further pursue this proposal in temporal convolutional SNNs, by introducing the hierarchical bias in the size and dilation of temporal kernels, giving rise to competitive results in popular temporal spike-based datasets.

cs.NE↗
arXiv2022

Bio-Inspired, Task-Free Continual Learning through Activity Regularization

The ability to sequentially learn multiple tasks without forgetting is a key skill of biological brains, whereas it represents a major challenge to the field of deep learning. To avoid catastrophic forgetting, various continual learning (CL) approaches have been devised. However, these usually require discrete task boundaries. This requirement seems biologically implausible and often limits the application of CL methods in the real world where tasks are not always well defined. Here, we take inspiration from neuroscience, where sparse, non-overlapping neuronal representations have been suggested to prevent catastrophic forgetting. As in the brain, we argue that these sparse representations should be chosen on the basis of feed forward (stimulus-specific) as well as top-down (context-specific) information. To implement such selective sparsity, we use a bio-plausible form of hierarchical credit assignment known as Deep Feedback Control (DFC) and combine it with a winner-take-all sparsity mechanism. In addition to sparsity, we introduce lateral recurrent connections within each layer to further protect previously learned representations. We evaluate the new sparse-recurrent version of DFC on the split-MNIST computer vision benchmark and show that only the combination of sparsity and intra-layer recurrent connections improves CL performance with respect to standard backpropagation. Our method achieves similar performance to well-known CL methods, such as Elastic Weight Consolidation and Synaptic Intelligence, without requiring information about task boundaries. Overall, we showcase the idea of adopting computational principles from the brain to derive new, task-free learning algorithms for CL.

cs.NE↗
arXiv2022

Disentangling the Predictive Variance of Deep Ensembles through the Neural Tangent Kernel

Identifying unfamiliar inputs, also known as out-of-distribution (OOD) detection, is a crucial property of any decision making process. A simple and empirically validated technique is based on deep ensembles where the variance of predictions over different neural networks acts as a substitute for input uncertainty. Nevertheless, a theoretical understanding of the inductive biases leading to the performance of deep ensemble's uncertainty estimation is missing. To improve our description of their behavior, we study deep ensembles with large layer widths operating in simplified linear training regimes, in which the functions trained with gradient descent can be described by the neural tangent kernel. We identify two sources of noise, each inducing a distinct inductive bias in the predictive variance at initialization. We further show theoretically and empirically that both noise sources affect the predictive variance of non-linear deep ensembles in toy models and realistic settings after training. Finally, we propose practical ways to eliminate part of these noise sources leading to significant changes and improved OOD detection in trained deep ensembles.

cs.LG↗
arXiv2022

Credit Assignment in Neural Networks through Deep Feedback Control

The success of deep learning sparked interest in whether the brain learns by using similar techniques for assigning credit to each synaptic weight for its contribution to the network output. However, the majority of current attempts at biologically-plausible learning methods are either non-local in time, require highly specific connectivity motives, or have no clear link to any known mathematical optimization method. Here, we introduce Deep Feedback Control (DFC), a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss-Newton optimization for a wide range of feedback connectivity patterns. To further underline its biological plausibility, we relate DFC to a multi-compartment model of cortical pyramidal neurons with a local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing. By combining dynamical system theory with mathematical optimization theory, we provide a strong theoretical foundation for DFC that we corroborate with detailed results on toy experiments and standard computer-vision benchmarks.

cs.LG↗
arXiv2020

Resonances induced by Spiking Time Dependent Plasticity

Neural populations exposed to a certain stimulus learn to represent it better. However, the process that leads local, self-organized rules to do so is unclear. We address the question of how can a neural periodic input be learned and use the Differential Hebbian Learning framework, coupled with a homeostatic mechanism to derive two self-consistency equations that lead to increased responses to the same stimulus. Although all our simulations are done with simple Leaky-Integrate and Fire neurons and standard Spiking Time Dependent Plasticity learning rules, our results can be easily interpreted in terms of rates and population codes.

q-bio.NC↗
arXiv2020

Tailoring Artificial Neural Networks for Optimal Learning

As one of the most important paradigms of recurrent neural networks, the echo state network (ESN) has been applied to a wide range of fields, from robotics to medicine, finance, and language processing. A key feature of the ESN paradigm is its reservoir --- a directed and weighted network of neurons that projects the input time series into a high dimensional space where linear regression or classification can be applied. Despite extensive studies, the impact of the reservoir network on the ESN performance remains unclear. Combining tools from physics, dynamical systems and network science, we attempt to open the black box of ESN and offer insights to understand the behavior of general artificial neural networks. Through spectral analysis of the reservoir network we reveal a key factor that largely determines the ESN memory capacity and hence affects its performance. Moreover, we find that adding short loops to the reservoir network can tailor ESN for specific tasks and optimize learning. We validate our findings by applying ESN to forecast both synthetic and real benchmark time series. Our results provide a new way to design task-specific ESN. More importantly, it demonstrates the power of combining tools from physics, dynamical systems and network science to offer new insights in understanding the mechanisms of general artificial neural networks.

cs.LG↗
arXiv2020

Eigenvalues of random graphs with cycles

Networks are often studied using the eigenvalues of their adjacency matrix, a powerful mathematical tool with a wide range of applications. Since in real systems the exact graph structure is not known, researchers resort to random graphs to obtain eigenvalue properties from known structural features. However, this theory is far from intuitive and often requires training of free probability, cavity methods or a strong familiarity with probability theory. In this note we offer a different perspective on this field by focusing on the cycles in a graph. We use the so-called method of moments to obtain relation between eigenvalues and cycle weights and then we obtain spectral properties of random graphs with cyclic motifs. We use it to explore properties of the eigenvalues of adjacency matrices of graphs with short cycles and of circulant directed graphs. Although our result is not as powerful as the some of the existing methods, they are nevertheless useful and far easier to understand.

math.SP↗
arXiv2019

Synaptic Time-Dependent Plasticity Leads to Efficient Coding of Predictions

Latency reduction of postsynaptic spikes is a well-known effect of Synaptic Time-Dependent Plasticity. We expand this notion for long postsynaptic spike trains, showing that, for a fixed input spike train, STDP reduces the number of postsynaptic spikes and concentrates the remaining ones. Then we study the consequences of this phenomena in terms of coding, finding that this mechanism improves the neural code by increasing the signal-to-noise ratio and lowering the metabolic costs of frequent stimuli. Finally, we illustrate that the reduction of postsynaptic latencies can lead to the emergence of predictions.

q-bio.NC↗
arXiv2019

Universal hypotrochoidic law for random matrices with cyclic correlations

The celebrated elliptic law describes the distribution of eigenvalues of random matrices with correlations between off-diagonal pairs of elements, having applications to a wide range of physical and biological systems. Here, we investigate the generalization of this law to random matrices exhibiting higher-order cyclic correlations between $k$-tuples of matrix entries. We show that the eigenvalue spectrum in this ensemble is bounded by a hypotrochoid curve with $k$-fold rotational symmetry. This hypotrochoid law applies to full matrices as well as sparse ones, and thereby holds with remarkable universality. We further extend our analysis to matrices and graphs with competing cycle motifs, which are described more generally by polytrochoid spectral boundaries.

math-ph↗