arXiv subjects

Eduardo Santos-Escriche

Publications and source records attributed to Eduardo Santos-Escriche.

2 recordsLinked to original sources

arXiv2026

How Do Transformers Learn to Represent Symmetries?

Training Transformer-based architectures with finite data augmentation has become an increasingly popular approach in geometric machine learning. Despite its empirical success, the interplay between the Transformer architecture, invariance to different symmetries, and augmentation budgets remains underexplored. In this paper, we study the ability of a vanilla Transformer to learn various symmetries through finite data augmentation for point cloud datasets. We identify an ordering of increasing learnability across the following symmetry groups: (i) non-angle-preserving symmetries, (ii) angle-preserving symmetries, and (iii) base angle-preserving subgroups, such as translation, rotation, and scale. For the base angle-preserving groups, we further investigate the Transformer's extrapolation behavior and conduct a structural analysis of the trained models, allowing us to identify interpretable mechanisms that induce invariance. Finally, we extend our analysis to equivariant functions and show that the detected mechanisms for approximate invariance can also provide a key building block for learned equivariance. Our project page is available at https://transformers-learn-symmetries.github.io/

cs.LG↗

arXiv2026

LieAugmenter: Equivariant Learning by Discovering Symmetries with Learnable Augmentations

Data augmentation is a powerful mechanism in equivariant machine learning, encouraging symmetry by training networks to produce consistent outputs under transformed inputs. Yet, effective augmentation typically requires the underlying symmetry to be specified a priori, which can limit generalization when symmetries are unknown or only approximately valid. To address this, we introduce LieAugmenter, an end-to-end framework that discovers task-relevant continuous symmetries through learnable augmentations. Specifically, the augmentation generator is parameterized using the theory of Lie groups and trained jointly with the prediction network using the augmented views. The learned augmentations are task-adaptive, enabling effective and interpretable symmetry discovery. We provide a theoretical analysis of identifiability and show that our method yields symmetry-respecting models for the identified groups. Empirically, LieAugmenter outperforms baselines on image classification, as well as on the prediction of N-body dynamics and molecular properties. In addition, it can also provide an interpretable signature for detecting the absence of symmetries.

cs.LG↗