arXiv · 2611.17019
Temporal Regularized Learning: Self-Supervised Learning Local in Space and Time
Abstract
Self-supervised objectives such as VICReg learn representations by making related inputs similar, while variance and covariance penalties prevent collapse. These penalties are statistics over a batch of samples, so layer-local variants, too, are typically local in space but not in time. We introduce Temporal Regularized Learning (TeReL), which optimizes a VICReg-style objective one sample at a time with purely local updates: each update uses only the current activity of its own and the preceding layer and a fixed-size running state. Temporally adjacent inputs serve as positive pairs, which makes the objective a soft form of Slow Feature Analysis (SFA). Running estimates of each neuron's mean and variance, together with one lateral matrix per layer, replace the batch statistics; held fixed during differentiation, they turn the gradient into a Hebbian feedforward update and an anti-Hebbian lateral update. We show that a two-layer network trained with batch size one on an MNIST stream of class-homogeneous chunks reaches 96.48% linear-probe accuracy, above random features (95.09%) and classical SFA, and that the temporal order and each loss term contribute to this result. With both locality constraints relaxed, the same objective matches supervised backpropagation. TeReL thus shows that the collapse-preventing statistics of modern self-supervised learning can be held as running state at each neuron, which brings these objectives closer to what brains and neuromorphic hardware can implement.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Davide Wiest. 2026-11-16. Temporal Regularized Learning: Self-Supervised Learning Local in Space and Time. https://searcharxiv.com/abs/2611.17019
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.