arXiv subjects

Igor Ignashin

Publications and source records attributed to Igor Ignashin.

4 recordsLinked to original sources

arXiv2026

Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

Stochastic Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion. However, this approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate. In this work, we propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling. Starting directly from the discrete update, we derive a master equation for the parameter distribution and obtain a discrete Fokker--Planck equation that differs from the standard Langevin form at order eta^2. Using this framework, we analyze SGD dynamics near critical points of the loss. We show that the behavior decomposes along the eigenbasis of the mean Hessian into qualitatively distinct regimes. In particular, nearly-flat directions do not admit a stationary distribution: the variance grows over time, corresponding to effective diffusion along valleys with a coefficient proportional to the learning rate. We provide empirical evidence supporting these predictions on neural network models in computer vision and natural language processing, observing a clear qualitative separation between confined and diffusive modes.

cs.LG↗

arXiv2025

Stochastic Origin Frank-Wolfe for traffic assignment

In this paper, we present the Stochastic Origin Frank-Wolfe (SOFW) method, which is a special case of the block-coordinate Frank-Wolfe algorithm, applied to the problem of finding equilibrium flow distributions. By significantly reducing the computational complexity of the minimization oracle, the method improves overall efficiency at the cost of increased memory consumption. Its key advantage lies in minimizing the number of shortest path computations. We refer to existing theoretical convergence guarantees for generalized coordinate Frank-Wolfe methods and, in addition, extend the analysis by providing a convergence proof for a batched version of the Block-Coordinate Frank-Wolfe algorithm, which was not covered in the original work. We also demonstrate the practical effectiveness of our approach through experimental results. In particular, our findings show that the proposed method significantly outperforms the classical Frank-Wolfe algorithm and its variants on large-scale datasets. On smaller datasets, SOFW also remains effective, though the performance gap relative to classical methods becomes less pronounced. In such cases, there is a trade-off between solution quality, iteration time complexity, and memory usage.

math.OC↗

arXiv2025

Modeling skiers flows via Wardrope equilibrium in closed capacitated networks

We propose an equilibrium model of ski resorts where users are assigned to cycles in a closed network. As queues form on lifts with limited capacity, we derive an efficient way to find waiting times via convex optimization. The equilibrium problem is formulated as a variational inequality, and numerical experiments show that it can be solved using standard algorithms.

eess.SY↗

arXiv2025

Aligning Distributionally Robust Optimization with Practical Deep Learning Needs

While traditional Deep Learning (DL) optimization methods treat all training samples equally, Distributionally Robust Optimization (DRO) adaptively assigns importance weights to different samples. However, a significant gap exists between DRO and current DL practices. Modern DL optimizers require adaptivity and the ability to handle stochastic gradients, as these methods demonstrate superior performance. Additionally, for practical applications, a method should allow weight assignment not only to individual samples, but also to groups of objects (for example, all samples of the same class). This paper aims to bridge this gap by introducing ALSO $\unicode{x2013}$ Adaptive Loss Scaling Optimizer $\unicode{x2013}$ an adaptive algorithm for a modified DRO objective that can handle weight assignment to sample groups. We prove the convergence of our proposed algorithm for non-convex objectives, which is the typical case for DL models. Empirical evaluation across diverse Deep Learning tasks, from Tabular DL to Split Learning tasks, demonstrates that ALSO outperforms both traditional optimizers and existing DRO methods.

cs.LG↗