arXiv subjects

Anna Shalova

Publications and source records attributed to Anna Shalova.

7 recordsLinked to original sources

arXiv2026

Random Quadratic Form with random forcing: Metastable synchronization by noise

We study the Random Quadratic Form (RQF) on a sphere in the presence of random Brownian forcing. We show that the forcing does not effectively change the law of the process but affects the synchronization properties of the system. While the RQF without forcing exhibits partial synchronization due to the intrinsic symmetries, the introduction of an arbitrarily small forcing results in long-term symmetry breaking and leads to full synchronization. In this work we focus on the small forcing regime and recover the multiscale behavior of the two-point process. We show that in the first stage the model converges to an anti-polar configuration due to the symmetries of the RQF and in the second stage the two clusters meet due to the symmetry breaking phenomenon. The model is motivated by continuous-time machine learning models such as Neural ODEs and continuous-time formulations of transformers. In particular, the results of this work explain the role of the bias and the scale of its initialization.

math.PR↗

arXiv2026

Random Quadratic Form on a Sphere: Synchronization by Common Noise

We introduce the Random Quadratic Form (RQF): a stochastic differential equation which formally corresponds to the gradient flow of a random quadratic functional on a sphere. While the one-point dynamics of the system is a Brownian motion and thus has no preferred direction, the two-point motion exhibits nontrivial synchronizing behaviour. In this work we study synchronization of the RQF, namely we give both distributional and path-wise characterizations of the solutions by studying invariant measures and random attractors of the system. The RQF model is motivated by the study of the role of linear layers in transformers and illustrates the synchronization by common noise phenomena arising in the simplified models of transformers. In particular, we provide an alternative (independent of self-attention) explanation of the clustering behaviour in deep transformers and show that tokens cluster even in the absence of the self-attention mechanism.

math.PR↗

arXiv2025

Nonlinear diffusion limit of non-local interactions on a sphere

We study an aggregation PDE with competing attractive and repulsive forces on a sphere of arbitrary dimension. In particular, we consider the limit of strongly localized repulsion with a constant attraction term. We prove convergence of solutions of such a system to solutions of the aggregation-diffusion equation with a porous-medium-type diffusion term. The proof combines variational techniques with elements of harmonic analysis on a sphere. In particular, we characterize the square root of the convolution operator in terms of the spherical harmonics, which allows us to overcome difficulties arising due to the convolution on a sphere being non-commutative. The study is motivated by the toy model of transformers introduced by Geshkovski et al. (2025); and we discuss the applicability of the results to this model.

math.AP↗

arXiv2025

Solutions of stationary McKean-Vlasov equation on a high-dimensional sphere and other Riemannian manifolds

We study stationary solutions of McKean-Vlasov equation on a high-dimensional sphere and other compact Riemannian manifolds. We extend the equivalence of the energetic problem formulation to the manifold setting and characterize critical points of the corresponding free energy functional. On a sphere, we employ the properties of spherical convolution to study the bifurcation branches around the uniform state. We also give a sufficient condition for an existence of a discontinuous transition point in terms of the interaction kernel and compare it to the Euclidean setting. We illustrate our results on a range of system, including the particle system arising from the transformer models and the Onsager model of liquid crystals.

math.AP↗

arXiv2024

Singular-limit analysis of gradient descent with noise injection

We study the limiting dynamics of a large class of noisy gradient descent systems in the overparameterized regime. In this regime the set of global minimizers of the loss is large, and when initialized in a neighbourhood of this zero-loss set a noisy gradient descent algorithm slowly evolves along this set. In some cases this slow evolution has been related to better generalisation properties. We characterize this evolution for the broad class of noisy gradient descent systems in the limit of small step size. Our results show that the structure of the noise affects not just the form of the limiting process, but also the time scale at which the evolution takes place. We apply the theory to Dropout, label noise and classical SGD (minibatching) noise, and show that these evolve on different two time scales. Classical SGD even yields a trivial evolution on both time scales, implying that additional noise is required for regularization. The results are inspired by the training of neural networks, but the theorems apply to noisy gradient descent of any loss that has a non-trivial zero-loss set.

cs.LG↗

arXiv2020

Tensorized Transformer for Dynamical Systems Modeling

The identification of nonlinear dynamics from observations is essential for the alignment of the theoretical ideas and experimental data. The last, in turn, is often corrupted by the side effects and noise of different natures, so probabilistic approaches could give a more general picture of the process. At the same time, high-dimensional probabilities modeling is a challenging and data-intensive task. In this paper, we establish a parallel between the dynamical systems modeling and language modeling tasks. We propose a transformer-based model that incorporates geometrical properties of the data and provide an iterative training algorithm allowing the fine-grid approximation of the conditional probabilities of high-dimensional dynamical systems.

cs.LG↗

arXiv2020

Deep Representation Learning for Dynamical Systems Modeling

Proper states' representations are the key to the successful dynamics modeling of chaotic systems. Inspired by recent advances of deep representations in various areas such as natural language processing and computer vision, we propose the adaptation of the state-of-art Transformer model in application to the dynamical systems modeling. The model demonstrates promising results in trajectories generation as well as in the general attractors' characteristics approximation, including states' distribution and Lyapunov exponent.

math.DS↗