Searcharxiv⌕ Search

arXiv subjects

Alexander Gasnikov

Publications and source records attributed to Alexander Gasnikov.

At least 19 recordsLinked to original sources
arXiv2026

Continuized Quasar-Convex Acceleration via an Unbiased Finite-Mixing Network Oracle

We study decentralized minimization of a smooth network average that is globally quasar-convex with respect to a fixed minimizer; the node functions may be nonconvex and need not share a minimizer. A unit-rate Poisson clock drives an exactly simulable continuized process: between events the state follows a closed-form accelerated flow, and each event uses one oracle vector. One master theorem covers oracles with a predictable bias and a state-dependent variance and keeps the inverse-quadratic term $\mathcal{O}(LR_0^2/(γ^2T^2))$; one selected-row lemma supplies the oracle moments of three network implementations. CQA-Net keeps a common state, mixes local rows for finitely many rounds, and disseminates one uniformly selected row; its oracle is unbiased at every mixing depth, so incomplete averaging affects only the variance. A walking token removes dissemination, the shared clock, and the shared root, and an importance weight keeps it unbiased on fixed graphs. D-CQA keeps local states and mixes them; under bounded heterogeneity it needs $\widetilde{\mathcal{O}}(\sqrtχ\sqrt{L}R_0/(γθ\sqrt{\varepsilon}))$ communication rounds and no dissemination. Changing graphs enter through certified operator norms. Experiments on a certified nonconvex regression on static and changing graphs compare the three implementations.

math.OC↗
arXiv2026

Federated Affine Variational Inequalities: Exact Averaging and Complexity Limits

We study the classical Local Stochastic Extragradient method (Local SEG) for monotone variational inequalities under intermittent communication, where devices perform several local oracle calls between synchronization rounds. For affine monotone mean operators, including nonsymmetric bilinear saddle systems, we prove that equal device averaging commutes pathwise with both stages of the stochastic extragradient update. Local trajectories may therefore drift apart while their mean follows a centralized stochastic-extragradient recursion driven by averaged noise. The identity remains valid when individual oracle realizations are nonlinear and query dependent. Under conditionally centered oracle errors that are conditionally independent across devices within each stage, the reduction yields a restricted weak-gap rate that uses the local sequential depth for deterministic progress and the aggregate oracle budget across devices for variance reduction. When every device makes at least two local oracle calls per communication round, the bound contains no additional mixed client-drift term and is also attained by a terminal-only aggregation schedule. An affine Gaussian lower bound matches the depth and sample dependences before statistical saturation for distributed zero-respecting algorithms; its statistical component applies to arbitrary randomized algorithms. A complementary lower bound for feasible distributed zero-respecting methods shows that bounded geometry and query-dependent noise can create an additional communication-locality obstruction. These results delineate the exact-averaging regime of unconstrained affine Local SEG and a complexity boundary that arises under active feasibility constraints.

math.OC↗
arXiv2026

Forgetting matters: when subspace optimization accelerates without the strong convexity parameter

Sequential subspace optimization (SESOP) minimizes the objective exactly over a few directions built from past iterates. It needs no problem constants, is accelerated on smooth convex functions and behaves like an accelerated method in experiments. This led to the conjecture that it is accelerated in the strongly convex case without knowing the strong convexity parameter. We show that the conjecture fails in the worst case. For every subspace method whose memory forgets slowly, namely a window of recent gradients and steps together with cumulative sums with subexponential weights, we construct a counterexample: acceleration requires a window of length of order $\sqrtκ$, where $κ$ is the condition number. SESOP and the plane form of conjugate gradients are therefore not accelerated. Memories that forget fast avoid this obstruction. Pairs of geometric filters give accelerated methods with logarithmic memory or only two stored vectors. They need neither the strong convexity parameter nor the smoothness constant, never restart the iterate and restart their memory at most $O(\logκ)$ times. Finally, the method with fixed activation times remains asymptotically accelerated on functions with Lipschitz Hessian even when each subproblem is solved by a single Newton step.

math.OC↗
arXiv2026

The Minimax Complexity of Federated Optimization under Interpolation: The Role of Oracle Models

We study the minimax complexity of smooth convex stochastic optimization with intermittent communication: $m$ workers run for $N$ rounds, and between consecutive communication rounds each worker makes $K$ sequential stochastic gradient queries. We focus on the interpolation regime, in which all sample losses share a common minimizer, so that the gradient noise vanishes at the solution. Let $L$ be the smoothness of the objective, $H_s\ge L$ the smoothness of the sample losses, and $R$ the distance from the initial point to a minimizer. We consider three oracle models that provide increasing information: an unbiased stochastic gradient oracle $g(x,ξ)$ satisfying $\mathbb{E}_ξ\|g(x,ξ)\|^2 \leq 2H_s\bigl(F(x)-F(x^*)\bigr)$, gradients of the sample losses at fresh samples, and repeated gradient queries to stored samples. Up to logarithmic factors, all our results are described by two rates: \[ R^2\min\left\{L,\;\frac{H_s}{KN},\; \frac{L}{N^2}+\frac{H_s}{mKN}\right\} \qquad\text{and}\qquad R^2\min\left\{L,\; \frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}, \] Under the weak growth condition, the first rate is the minimax rate, attained by the best of three simple baselines, as in the classical bounded-variance setting. With repeated access to samples, the second rate becomes the minimax rate for general convex objectives. For general convex objectives with fresh sample gradients, the minimax rate lies between the two rates. Our lower bounds apply to arbitrary randomized intermittent-communication algorithms.

math.OC↗
arXiv2026

When Momentum Helps Polyak--Ruppert Averaging

Polyak--Ruppert averaging gives SGD and momentum methods the same asymptotic covariance. We characterize how fast finite runs reach it and how far they can be parallelized. For strongly convex quadratics with additive noise, we prove an online--offline bridge: the empirical risk minimizer on the samples of a tail average is one Newton step from the tail-averaged query point of any online procedure. For SGD, heavy ball, and Nesterov's method with constant momentum, this step is a boundary term at the two ends of the tail and splits the gap to the offline estimator into center, covariance, and shape errors. Necessary and sufficient conditions show that, on the least curved direction, the center error sets the pace: SGD needs $m\ggκ^2$ rounds but momentum only $m\ggκ$, where $κ$ is the condition number, so at sample size $N$ the critical batch size grows from $Θ(\sqrt{N}/κ)$ to $Θ(\sqrt{N/κ})$. The covariance error separates the momentum methods: heavy ball has relative variance excess $Θ(κ^{3/2}/m)$ along the stiffest direction, whereas Nesterov has relative error $O(\sqrtκ/m)$ in every direction. The shape error is method-independent by a Berry--Esseen bound uniform in $κ$ and batch size, so confidence intervals are valid when the other two errors vanish. We further examine how the conclusions change under heavy-tailed or state-dependent noise and clipping, and establish a first-order comparison with ERM for nonlinear Nesterov averaging under explicit conditions.

cs.LG↗
arXiv2026

Weak-$L^α$ Noise, Strong Guarantees: Gradient Clipping under Affine Heavy Tails

Gradient clipping is a key tool for obtaining high-probability convergence guarantees for stochastic first-order methods under heavy-tailed noise. Existing analyses typically assume a bounded central moment of order $α\in(1,2]$, which excludes distributions with tails of order $u^{-α}$ at the critical exponent. In this paper, we consider smooth convex optimization under affine heavy-tailed noise, whose scale may grow with the gradient norm. We derive high-probability convergence guarantees for clipped SGD (Clip-SGD) and its accelerated variant Clip-SSTM under a weak-$L^α$ tail condition, which is strictly weaker than the corresponding moment bound. For $1<α<2$, the clipping bounds have the same powers of the clipping level as in the moment case, and both methods attain the additive stochastic rate $K^{-(α-1)/α}$. The gradient-dependent noise contributes a faster-decaying term of order $K^{-2(α-1)/α}$, up to logarithmic factors. At $α=2$, we characterize the additional logarithmic factor on the Lorentz scale and establish its necessity for the additive error at fixed confidence. We also analyze Clip-SGD with prescribed clipping levels and derive finite-horizon guarantees for reaching a neighborhood determined by the clipping bias. For additive noise, matching lower bounds show that this neighborhood is unavoidable, up to constants and the natural error cap. Numerical experiments with Pareto noise illustrate the theoretical results.

math.OC↗
arXiv2026

Adaptive First-Order Methods with Composite Gradient Error

We study first-order optimization with inexact gradients whose errors combine a component proportional to the true gradient norm and a uniformly bounded absolute component. We develop an adaptive gradient descent scheme that selects estimates of the smoothness constant and both error parameters through a computable backtracking condition using exact function values. Its analysis yields convergence for smooth objectives and functions satisfying Polyak-Lojasiewicz condition. For strongly convex objectives, we propose a composite-error intermediate gradient method, that adapts to unknown smoothness while using known error bounds and a strong convexity parameter $μ$. With fixed smoothness $L$, acceleration is retained when $α=\mathcal{O}\left(\left(μ/ L \right)^{1/4}\right)$, larger admissible errors yield intermediate rates. Throughout the range $0 \leqslant α\leqslant 1/3$, the absolute error bound $δ$ contributes at most $\mathcal{O} \left(δ^2/μ\right)$ to the objective residual, without additional amplification by the condition number.

math.OC↗
arXiv2026

Stochastic Optimization and Data Science

This paper aims to motivate stochastic optimization problems from a statistical perspective and a statistical learning perspective, where the goal is to maximize the log-likelihood or minimize the population risk. We briefly describe the two main approaches: offline (Monte Carlo / Sample Average Approximation) and online (Stochastic Approximation) approaches -- to solve the expectation minimization problems.

math.OC↗
arXiv2026

Minimax Oracle Call Complexity, Rounds and Admissible Noise in Zeroth-Order Optimization of Highly Smooth Functions

We study zeroth-order optimization of convex and strongly convex functions with Hölder smoothness of order $β>2$ from values corrupted by random noise with bias independent of the method's random directions, a Lipschitz systematic error and a bounded adversarial error. We measure the number of function evaluations (calls), of sequential rounds of parallel calls, and the largest tolerable systematic and adversarial errors. For strongly convex functions with a bounded Hessian, we prove a lower bound of order $d^2\varepsilon^{-β/(β-1)}$ on the number of calls, where $d$ is the dimension and $\varepsilon$ the accuracy, and find the sharp power of $d$ under the stronger Hilbert--Schmidt smoothness for all $β>2$. An accelerated finite-difference method attains them in $O(\sqrtκ\log(1/\varepsilon))$ rounds for condition number $κ$, like Nesterov's method with exact gradients, and tolerates the optimal systematic and, up to logarithms, adversarial errors; rate-optimal methods need $Ω(\log d)$ rounds. The same number of calls is attained when the Hessian grows at most linearly with the gradient and, for centered noise, without an upper bound on the Hessian, also in logarithmically many rounds. For convex functions, our lower bound improves the known one polynomially in $d$ and is tight at the curvature of the hard instances; the systematic level is again optimal. These bounds extend to general norms and constraints; on the $\ell_1$ ball, mirror descent needs calls nearly linear in $d$. Experiments confirm the theory.

math.OC↗
arXiv2026

Matching the Lower Bounds: Stochastic Contracting Cubic Newton and Its Optimal Acceleration

We study second-order methods for convex stochastic optimization, where gradients and Hessians are available only through stochastic estimates with variances $σ_1^2$ and $σ_2^2$, respectively. First, we propose the Stochastic Contracting Cubic Newton method. At each iteration, it minimizes a cubic model with additional quadratic regularization and then contracts the step toward the current point. After $T$ iterations, the method achieves the expected convergence rate $\mathcal{O}(σ_1/\sqrt{T}+σ_2/T+1/T^2)$. Building on this construction, we develop an accelerated variant achieving $\mathcal{O}(σ_1/\sqrt{T}+σ_2/T^2+1/T^{7/2})$, matching the known lower bounds of Agafonov et al. (2024) in all three terms.

math.OC↗
arXiv2026

Beyond Aggregate Drift: Optimal Client-Specific Windows in Federated Learning

We study federated learning from continuously arriving data: each client samples from its own distribution, these distributions drift over time, and the server must track the minimizer of the clients' weighted average loss. Clients are not alike as they collect data at different rates, with different noise, and change at different speeds. The main question is how much history should each one keep? A single shared window must serve every client at once, and so makes a fast-changing client carry the sampling noise of a slow, nearly stationary one. We give each client a memory of its own, sized to that client's noise, arrival rate, and drift, and refresh it by restarted accelerated method that starts from a common point. The tracking error is then bounded by \[ R\left(\sum_m w_m^{3/2}\,σ_m\sqrt{γ_m/λ_m}\right)^{2/3}, \] for clients with weight $w_m$, noise $σ_m$, arrival rate $λ_m$, and drift rate $γ_m$ on a domain of diameter $R$. This is never worse than what any common window can achieve, and strictly better whenever noise and drift are spread unevenly across clients. A lower bound that puts different groups of clients on different drift clocks shows this quantity is the right one. The two bounds agree up to a constant whenever the clients occupy boundedly many refresh scales, and in general differ only by the sixth root of the number of scales. Experiments show the gain comes from the memory rather than from the solver: it persists across accelerated, stochastic-gradient, and one-step methods, and against a separately tuned common-memory baseline.

math.OC↗
arXiv2026

Adaptive two-points zero-order accelerated methods for stochastic convex optimization problems

We study stochastic convex optimization, where each sampled function can be evaluated at two arbitrary points. Our accelerated method chooses step sizes, batch sizes and smoothing radii from function values. A forward difference between smoothing scales controls the radius; we account for changes in the smoothed objective. Under the stated regularity and second-moment assumptions, exact function evaluations give \begin{multline*} \mathbb{E}\bigl[f(x_N)-f(x^\star)\bigr] \le \widetilde{O}\biggl( \frac{\sqrt{d}\,M_2 D}{\sqrt{T}} +\min\biggl( \frac{L D^2}{N^2},\; \frac{d^{1/4} M D}{N}+\frac{\sqrt{d}\,M D^2}{γ_{\max} N^2} \biggr)\biggr). \end{multline*} Here $N,T,d,D$ denote the numbers of optimization steps and paired evaluations, the dimension and a feasible-set diameter bound, respectively. Evaluations are available within Euclidean distance $γ_{\max}$ of the feasible set. For the population objective $f(x)=\mathbb{E}_ξ F(x,ξ)$, $M$ satisfies $|f(x)-f(y)|\le M\|x-y\|_2$, and $M_2$ satisfies $\mathbb{E}_ξ\|\nabla F(x,ξ)\|_2^2\le M_2^2$ for sample weak gradients in the interior of the evaluation domain. The method requires no $L,M,M_2$, where $L$ is a Lipschitz constant of $\nabla f$; set $L=+\infty$ without smoothness. The method uses exactly $2T$ values in $3N$ sequential sampling rounds with predetermined batches, or $5N$ with batches chosen from observations. An extension selects a search radius without a supplied distance bound. Fixed radius parameters can leave nonvanishing error; suitable choices depending on the budgets yield convergence.

math.OC↗
arXiv2026

Sliding Methods for Hölder-Smooth Convex--Concave Minimax Optimization with Bilinear Coupling

We study convex-concave minimax optimization problems with bilinear coupling of the form $\min_{x\in \mathcal X}\max_{y\in \mathcal Y} \; f(x)+\langle y,\mathbf{B}x\rangle-g(y),$ where the functions $f$ and $g$ have Hölder continuous (sub)gradients. This setting covers a broad range of regimes, from nonsmooth problems with bounded subgradient variation to smooth problems with Lipschitz continuous gradients; for a smooth component used in the coupling-induced regularizer, its Lipschitz-gradient constant is assumed to hold in the ambient space. We propose a sliding method that exploits the composite structure of the problem by querying the oracles associated with $f$, $g$, and the bilinear coupling operator at prescribed frequencies determined by their individual properties. The method is based on a recursive sliding scheme for monotone variational inequalities. We establish convergence guarantees under Hölder continuity and show how the resulting complexity bounds depend explicitly on the Hölder exponents, Hölder constants, strong convexity parameters, and spectral properties of the coupling matrix. Our analysis covers nonstrongly convex and partially strongly convex regimes. For stochastic function oracles, we prove componentwise Hölder guarantees with separate variance costs: a uniform expected-gap bound on bounded domains and, when both functions are strongly convex on their feasible sets, a restarted Lyapunov guarantee without an anchor noise floor. Numerical experiments reproduce the predicted Hölder exponents and confirm that the number of gradient evaluations required for each function separates according to its own smoothness level rather than the worse of the two. A tomographic benchmark shows runtime gains when gradient evaluations are more expensive than the additional matrix-vector products.

math.OC↗
arXiv2026

Optimal Rates for Federated Variational Inequalities under different relaxations of bounded variance assumption

We study how interpolation affects local steps and parallel sampling in monotone variational inequalities (VIs), with $m$ workers, $K$ oracle calls per worker per round, and $N$ rounds. Under the strong growth condition (SGC), we determine the minimax rate for affine VIs, attained by local extragradient with worker averaging. For nonlinear monotone VIs and convex-concave saddle problems with Lipschitz operators satisfying SGC, we improve the stochastic terms of bounded-variance expected-gap guarantees from $(KN)^{-1/2}$ to $(KN)^{-1}$ with local steps and from $(mKN)^{-1/2}$ to $(mKN)^{-1}$ with minibatches. Under strong monotonicity, we improve the noise-dependent condition-number scaling of existing extragradient bounds from quadratic to linear. The resulting monotone-gap and strongly monotone accuracy bounds match lower bounds up to logarithmic factors in $m$. Under generalized smoothness and SGC, adaptive steps give polynomial dependence on the initial radius. For affine VIs with globally bounded oracle variance, we also attain the minimax rate: the deterministic rate over $KN$ sequential calls plus the statistical rate over all $mKN$ samples. For composite VIs with smooth convex sample interpolation and an exactly evaluated monotone Lipschitz component, accelerated mirror-prox sliding achieves an $N^{-2}$ gradient term, a $(KN)^{-1}$ monotone term, and a $(mKN)^{-1}$ sampling term, without assuming SGC.

math.OC↗
arXiv2026

Complexity of Multiblock Convex-Concave Saddle-Point Problems with General Smooth Coupling

We study multiblock convex--concave saddle-point problems of the form $\min_x\max_y\{f(x)+H(x,y)-g(y)\}$, with separately charged partial-gradient oracles for all three objective components. We impose joint smoothness and strong convexity bounds on the functions $f$ and $g$, together with blockwise Lipschitz conditions on the partial gradients of $H$, so that the resulting oracle complexities reflect the smoothness and dependency parameters specific to each oracle. Under strong convexity, we develop a randomized lifted extragradient method that achieves linear convergence while providing simultaneous complexity guarantees for all oracles. The partial-gradient complexities for $f$ and $g$ exhibit the expected square-root dependence on the corresponding block condition numbers, whereas the coupling-gradient complexities depend on the associated curvature-normalized smoothness parameters. Finite-dimensional bilinear subclasses provide lower-complexity benchmarks within a randomized blockwise linear-span model. When strong convexity of $f$ and $g$ is not assumed, a regularization argument yields expected restricted-gap guarantees with accuracy dependence $\varepsilon^{-1/2}$ for the objective-gradient oracles and $\varepsilon^{-1}$ for the coupling-gradient oracles, up to logarithmic factors. Experiments show observed oracle savings and runtime improvements in regimes of poor objective conditioning.

math.OC↗
arXiv2026

Asking the Crowd the Right Question: Bias-Cancelling Weights for Federated Learning

A federated objective is a weighted sum of client risks, and the weights are almost always fixed in advance. We treat them instead as the only instrument of a wisdom-of-crowds mechanism: clients are noisy views of one truth, each seeing it through an independent distortion that is unbiased across the crowd. That the optimal weights are inversely proportional to the clients' error energies is classical; we begin at the question that answer presupposes, which energies belong there and whether a crowd can recover them from itself. Excess risk on the truth is of the exact order of the aggregate bias energy, so no optimizer can repair a bad weight vector; the truth itself is identifiable only up to a linear tilt, so subtracting estimated client biases provably reproduces uniform weighting. The expected per-client second moments, however, are exactly identified from the law of the crowd's disagreement by a well-conditioned linear inversion, a step random-effects meta-analysis cannot take because a source reports once; their realized counterparts are estimable up to an incoherence floor the algorithm can measure. This yields CROWD, which reads the disagreement off the optimization trajectory at no extra cost and matches a Bayesian minimax lower bound in the same constant: per instance as the horizon grows, and unconditionally as the prior becomes diffuse. For arbitrary distortions it stays competitive with the optimal weights, at a ratio governed by a geometric incoherence the algorithm can measure. On real scans split into sites with their own miscalibrated detectors it attains the oracle excess risk; on a companion federation that pulls bias and noise apart, weighting by noise variance is worse than not weighting at all, and CROWD is not.

math.OC↗
arXiv2026

Gradient-Free Methods for Stochastic Convex Optimization with Stochastic Functional Constraints

We develop accelerated gradient-free methods for stochastic convex optimization with constraints defined by expectations. Our batched primal-dual sliding method uses two-point evaluations sharing a random sample and guarantees expected objective error and expected maximum constraint violation at most $\varepsilon$. It achieves $O(\varepsilon^{-1/2})$ sequential oracle rounds for smooth data and $O(d^{1/4}/\varepsilon)$ for nonsmooth data in dimension $d$, recovering the accuracy and dimension dependence of the corresponding unconstrained accelerated methods. The smooth rate is optimal in accuracy. Each round collects all samples needed for its inner primal-dual updates. Total evaluations retain quadratic dependence on inverse accuracy, with only polylogarithmic dependence on the number of constraints under vector feedback. Complementary lower bounds distinguish the dimension cost of gradient estimation from the unavoidable logarithmic cost of estimating noisy constraint levels. For smooth strongly convex objectives, restarts give logarithmic round complexity and objective evaluation cost linear in inverse accuracy, while constraint-level estimation necessarily remains quadratic. Known affine constraints require no constraint-oracle calls.

math.OC↗
arXiv2026

Greedy sign-based box splitting: exact solvable classes, sharp thresholds and order-oracle complexity

Greedy box splitting reads gradient signs at the centre and retains one smaller box. Its summable travel budget makes an incorrect exclusion irreversible. We determine when this geometry is safe. Halving is minimax optimal in the sign-vector oracle model on a coordinatewise sign-consistent class that need not be convex. A dimensionless sign defect yields a sharp scalar envelope, attained by one smooth convex potential; a three-dimensional trajectory ends farther from the target than its initial radius. For quadratics, universal convergence with aspect $w$ and contraction beta holds exactly when the weighted absolute row defect is at most $(2 β- 1)$. Perron weights minimise this defect. Adaptive half-sizes avoid initial enlargement and, under persistent error, converge to an exact capped fixed-point limit. For binary comparisons, an independent probe fraction $q$ permits universal localisation on common-target monotone sections exactly when $(q \leq 2 β- 1)$; coupled probes give the sharp threshold $2/3$. Matching information bounds distinguish comparisons from sign-vector queries. Error bands give finite-time guarantees and sharp residual floors. Extensions and reproducible experiments identify the assumptions and costs of freezing, smoothing, voting, multistart and random-matrix certificates.

math.OC↗