arXiv subjects

Aleksandr Lobanov

Publications and source records attributed to Aleksandr Lobanov.

22 records · Page 2Linked to original sources

arXiv2023

Non-Smooth Setting of Stochastic Decentralized Convex Optimization Problem Over Time-Varying Graphs

Distributed optimization has a rich history. It has demonstrated its effectiveness in many machine learning applications, etc. In this paper we study a subclass of distributed optimization, namely decentralized optimization in a non-smooth setting. Decentralized means that $m$ agents (machines) working in parallel on one problem communicate only with the neighbors agents (machines), i.e. there is no (central) server through which agents communicate. And by non-smooth setting we mean that each agent has a convex stochastic non-smooth function, that is, agents can hold and communicate information only about the value of the objective function, which corresponds to a gradient-free oracle. In this paper, to minimize the global objective function, which consists of the sum of the functions of each agent, we create a gradient-free algorithm by applying a smoothing scheme via $l_2$ randomization. We also verify in experiments the obtained theoretical convergence results of the gradient-free algorithm proposed in this paper.

math.OC↗

arXiv2023

Zero-Order Stochastic Conditional Gradient Sliding Method for Non-smooth Convex Optimization

The conditional gradient idea proposed by Marguerite Frank and Philip Wolfe in 1956 was so well received by the community that new algorithms (also called Frank--Wolfe type algorithms) are still being actively created. In this paper, we study a non-smooth stochastic convex optimization problem with constraints. Using a smoothing technique and based on an accelerated batched first-order Stochastic Conditional Gradient Sliding method, we propose a novel gradient-free Frank--Wolfe type algorithm called Zero-Order Stochastic Conditional Gradient Sliding (ZO-SCGS). This algorithm is robust not only for the class of non-smooth problems, but surprisingly also for the class of smooth black box problems, outperforming the SOTA algorithms in the smooth case in term oracle calls. In practical experiments we confirm our theoretical results.

math.OC↗

arXiv2023

Gradient-Free Federated Learning Methods with $l_1$ and $l_2$-Randomization for Non-Smooth Convex Stochastic Optimization Problems

This paper studies non-smooth problems of convex stochastic optimization. Using the smoothing technique based on the replacement of the function value at the considered point by the averaged function value over a ball (in $l_1$-norm or $l_2$-norm) of small radius with the center in this point, the original problem is reduced to a smooth problem (whose Lipschitz constant of the gradient is inversely proportional to the radius of the ball). An important property of the smoothing used is the possibility to calculate an unbiased estimation of the gradient of a smoothed function based only on realizations of the original function. The obtained smooth stochastic optimization problem is proposed to be solved in a distributed federated learning architecture (the problem is solved in parallel: nodes make local steps, e.g. stochastic gradient descent, then they communicate - all with all, then all this is repeated). The goal of this paper is to build on the current advances in gradient-free non-smooth optimization and in feild of federated learning, gradient-free methods for solving non-smooth stochastic optimization problems in federated learning architecture.

math.OC↗

arXiv2023

Stochastic Adversarial Noise in the "Black Box" Optimization Problem

This paper is devoted to the study of the solution of a stochastic convex black box optimization problem. Where the black box problem means that the gradient-free oracle only returns the value of objective function, not its gradient. We consider non-smooth and smooth setting of the solution to the black box problem under adversarial stochastic noise. For two techniques creating gradient-free methods: smoothing schemes via $L_1$ and $L_2$ randomizations, we find the maximum allowable level of adversarial stochastic noise that guarantees convergence. Finally, we analyze the convergence behavior of the algorithms under the condition of a large value of noise level.

math.OC↗