arXiv subjects

Alexander Rogozin

Publications and source records attributed to Alexander Rogozin.

At least 19 recordsLinked to original sources

arXiv2026

Decentralized Inexact Cubic Newton Method with Consensus Procedure

Distributed optimization is widely used in large-scale and privacy-preserving machine learning, where each agent stores a local objective and communicates only with its neighbors in a connected network. We study decentralized second-order optimization and focus on consensus procedures that approximately average local iterates, gradients, and Hessians through neighbor-to-neighbor communications. We propose a general Decentralized Cubic Newton method for convex optimization under $L_1$-smoothness of gradients and $L_2$-Lipschitz continuity of Hessians, and develop a theory that accurately tracks the inaccuracies caused by consensus and by disagreement between local iterates. Under these assumptions, the method matches the iteration complexity of the exact Cubic Newton method and requires only additional polylogarithmic communication-round overhead to reach the necessary consensus accuracy. We further propose an Accelerated Decentralized Cubic Newton method for strongly convex objectives and show that it matches the iteration complexity of the exact Accelerated Cubic Newton method, again with only additional polylogarithmic communication-round overhead. Finally, although the general method requires exchanging full $d \times d$ Hessian matrices, we show how it can be implemented for generalized linear models by transmitting only vectors, making the approach substantially more practical in high dimensions.

math.OC↗

arXiv2026

A Parameter-Free Zeroth-Order Algorithm for Decentralized Stochastic Convex Optimization

We consider decentralized stochastic convex optimization on connected network, in which gradients of agents are unavailable and each agent can query only noisy function values of its own local objective. The goal is to minimize the average objective over a compact convex domain using only local two point zeroth-order oracles and peer-to-peer communication. We propose a decentralized POEM method (D-POEM) that combines symmetric two point smoothing with adaptive radius and stepsize rules, thereby avoiding prior knowledge of the Lipschitz constant and diameter. For convex Lipschitz continuous objectives, we prove an convergence rate that separates a centralized optimization term from a network disagreement term. We further conduct the numerical experiments to demonstrate POEM outperforms existing distributed zeroth-order method.

math.OC↗

arXiv2026

Exploring New Frontiers in Vertical Federated Learning: the Role of Saddle Point Reformulation

The objective of Vertical Federated Learning (VFL) is to collectively train a model using features available on different devices while sharing the same users. This paper focuses on the saddle point reformulation of the VFL problem via the classical Lagrangian function. We first demonstrate how this formulation can be solved using deterministic methods. More importantly, we explore various stochastic modifications to adapt to practical scenarios, such as employing compression techniques for efficient information transmission, enabling partial participation for asynchronous communication, and utilizing coordinate selection for faster local computation. We show that the saddle point reformulation plays a key role and opens up possibilities to use mentioned extension that seem to be impossible in the standard minimization formulation. Convergence estimates are provided for each algorithm, demonstrating their effectiveness in addressing the VFL problem. Additionally, alternative reformulations are investigated, and numerical experiments are conducted to validate performance and effectiveness of the proposed approach.

math.OC↗

arXiv2026

Decentralized Optimization with Mixed Affine Constraints

This paper considers decentralized optimization of convex functions with mixed affine equality constraints involving both local and global variables. Constraints on global variables may vary across different nodes in the network, while local variables are subject to coupled and node-specific constraints. Such problem formulations arise in machine learning applications, including federated learning and multi-task learning, as well as in resource allocation and distributed control. We analyze this problem under smooth and non-smooth assumptions, considering both strongly convex and general convex objective functions. Our main contribution is an optimal algorithm for the smooth, strongly convex regime, whose convergence rate matches established lower complexity bounds. We further provide near-optimal methods for the remaining cases.

math.OC↗

arXiv2025

Dual Smoothing for Decentralized Optimization

Decentralized optimization is widely used in different fields of study such as distributed learning, signal processing, and various distributed control problems. In these types of problems, nodes of the network are connected to each other and seek to optimize some objective function. In this article, we present a method for smoothing the non-smooth and non-strongly convex problems. This is done using the dual smoothing technique. We study two types of problems: consensus optimization of linear models and coupled constraints optimization. It is shown that these two problem classes are dual to each other.

math.OC↗

arXiv2025

Robustifying networks for flow problems against edge failure

We consider the robust version of a multi-commodity network flow problem. The robustness is defined with respect to the deletion, or failure, of edges. While the flow problem itself is a polynomially-sized linear program, its robust version is a saddle-point problem with discrete variables. We present two approaches for the solution of the robust network flow problem. One way is to formulate the problem as a bigger linear program. The other is to solve a multi-level optimization problem, where the linear programs appearing at the lower level can be solved by the dual simplex method with a warm start. We then consider the problem of robustifying the network. This is accomplished by optimally using a fixed budget for strengthening certain edges, i.e., increasing their capacity. This problem is solved by a sequence of linear programs at the upper level, while at the lower levels the mentioned dual simplex algorithm is employed.

math.OC↗

arXiv2024

Average-case optimization analysis for distributed consensus algorithms on regular graphs

The consensus problem in distributed computing involves a network of agents aiming to compute the average of their initial vectors through local communication, represented by an undirected graph. This paper focuses on the studying of this problem using an average-case analysis approach, particularly over regular graphs. Traditional algorithms for solving the consensus problem often rely on worst-case performance evaluation scenarios, which may not reflect typical performance in real-world applications. Instead, we apply average-case analysis, focusing on the expected spectral distribution of eigenvalues to obtain a more realistic view of performance. Key contributions include deriving the optimal method for consensus on regular graphs, showing its relation to the Heavy Ball method, analyzing its asymptotic convergence rate, and comparing it to various first-order methods through numerical experiments.

math.OC↗

arXiv2024

Decentralized Optimization with Coupled Constraints

We consider the decentralized minimization of a separable objective $\sum_{i=1}^{n} f_i(x_i)$, where the variables are coupled through an affine constraint $\sum_{i=1}^n\left(\mathbf{A}_i x_i - b_i\right) = 0$. We assume that the functions $f_i$, matrices $\mathbf{A}_i$, and vectors $b_i$ are stored locally by the nodes of a computational network, and that the functions $f_i$ are smooth and strongly convex. This problem has significant applications in resource allocation and systems control and can also arise in distributed machine learning. We propose lower complexity bounds for decentralized optimization problems with coupled constraints and a first-order algorithm achieving the lower bounds. To the best of our knowledge, our method is also the first linearly convergent first-order decentralized algorithm for problems with general affine coupled constraints.

math.OC↗

arXiv2024

Decentralized Finite-Sum Optimization over Time-Varying Networks

We consider decentralized time-varying stochastic optimization problems where each of the functions held by the nodes has a finite sum structure. Such problems can be efficiently solved using variance reduction techniques. Our aim is to explore the lower complexity bounds (for communication and number of stochastic oracle calls) and find optimal algorithms. The paper studies strongly convex and nonconvex scenarios. To the best of our knowledge, variance reduced schemes and lower bounds for time-varying graphs have not been studied in the literature. For nonconvex objectives, we obtain lower bounds and develop an optimal method GT-PAGE. For strongly convex objectives, we propose the first decentralized time-varying variance-reduction method ADOM+VR and establish lower bound in this scenario, highlighting the open question of matching the algorithms complexity and lower bounds even in static network case.

math.OC↗

arXiv2023

Min-max optimization over slowly time-varying graphs

Distributed optimization is an important direction of research in modern optimization theory. Its applications include large scale machine learning, distributed signal processing and many others. The paper studies decentralized min-max optimization for saddle point problems. Saddle point problems arise in training adversarial networks and in robust machine learning. The focus of the work is optimization over (slowly) time-varying networks. The topology of the network changes from time to time, and the velocity of changes is limited. We show that, analogically to decentralized optimization, it is sufficient to change only two edges per iteration in order to slow down convergence to the arbitrary time-varying case. At the same time, we investigate several classes of time-varying graphs for which the communication complexity can be reduced.

math.OC↗

arXiv2023

Decentralized Optimization Over Slowly Time-Varying Graphs: Algorithms and Lower Bounds

We consider a decentralized convex unconstrained optimization problem, where the cost function can be decomposed into a sum of strongly convex and smooth functions, associated with individual agents, interacting over a static or time-varying network. Our main concern is the convergence rate of first-order optimization algorithms as a function of the network's graph, more specifically, of the condition numbers of gossip matrices. We are interested in the case when the network is time-varying but the rate of changes is restricted. We study two cases: randomly changing network satisfying Markov property and a network changing in a deterministic manner. For the random case, we propose a decentralized optimization algorithm with accelerated consensus. For the deterministic scenario, we show that if the graph is changing in a worst-case way, accelerated consensus is not possible even if only two edges are changed at each iteration. The fact that such a low rate of network changes is sufficient to make accelerated consensus impossible is novel and improves the previous results in the literature.

math.OC↗

arXiv2023

Decentralized conditional gradient method over time-varying graphs

In this paper we study a generalization of distributed conditional gradient method to time-varying network architectures. We theoretically analyze convergence properties of the algorithm and provide numerical experiments. The time-varying network is modeled as a deterministic of a stochastic sequence of graphs.

math.OC↗

arXiv2023

Decentralized optimization with affine constraints over time-varying networks

The decentralized optimization paradigm assumes that each term of a finite-sum objective is privately stored by the corresponding agent. Agents are only allowed to communicate with their neighbors in the communication graph. We consider the case when the agents additionally have local affine constraints and the communication graph can change over time. We provide the first linearly convergent decentralized algorithm for time-varying networks by generalizing the optimal decentralized algorithm ADOM to the case of affine constraints. We show that its rate of convergence is optimal for first-order methods by providing the lower bounds for the number of communications and oracle calls.

math.OC↗

arXiv2023

Decentralized Proximal Optimization Method with Consensus Procedure

Decentralized optimization is well studied for smooth unconstrained problems. However, constrained problems or problems with composite terms are an open direction for research. We study structured (or composite) optimization problems, where the functional is a sum of a convex smooth function and a proper convex proximal-friendly term. Our method builds upon an accelerated proximal gradient descent and makes several consensus iterations between computations. Our result illustrates that a consensus procedure approach works for composite optimization and yields a method with a relatively simple structure and analysis.

math.OC↗

arXiv2023

On Decentralized Nonsmooth Optimization

In decentralized optimization, several nodes connected by a network collaboratively minimize some objective function. For minimization of Lipschitz functions lower bounds are known along with optimal algorithms. We study a specific class of problems: linear models with nonsmooth loss functions. Our algorithm combines regularization and dual reformulation to get an effective optimization method with complexity better than the lower bounds.

math.OC↗

arXiv2023

A General Framework for Distributed Partitioned Optimization

Decentralized optimization is widely used in large scale and privacy preserving machine learning and various distributed control and sensing systems. It is assumed that every agent in the network possesses a local objective function, and the nodes interact via a communication network. In the standard scenario, which is mostly studied in the literature, the local functions are dependent on a common set of variables, and, therefore, have to send the whole variable set at each communication round. In this work, we study a different problem statement, where each of the local functions held by the nodes depends only on some subset of the variables. Given a network, we build a general algorithm-independent framework for decentralized partitioned optimization that allows to construct algorithms with reduced communication load using a generalization of Laplacian matrix. Moreover, our framework allows to obtain algorithms with non-asymptotic convergence rates with explicit dependence on the parameters of the network, including accelerated and optimal first-order methods. We illustrate the efficacy of our approach on a synthetic example.

math.OC↗

arXiv2023

Is Consensus Acceleration Possible in Decentralized Optimization over Slowly Time-Varying Networks?

We consider decentralized optimization problems where one aims to minimize a sum of convex smooth objective functions distributed between nodes in the network. The links in the network can change from time to time. For the setting when the amount of changes is arbitrary, lower complexity bounds and corresponding optimal algorithms are known, and the consensus acceleration is not possible. However, in practice the magnitude of network changes may be limited. We derive lower communication complexity bounds for several regimes of velocity of networks changes. Moreover, we show how to obtain accelerated communication rates for a certain class of time-varying graphs using a specific consensus algorithm.

math.OC↗

arXiv2023

Decentralized Strongly-Convex Optimization with Affine Constraints: Primal and Dual Approaches

Decentralized optimization is a common paradigm used in distributed signal processing and sensing as well as privacy-preserving and large-scale machine learning. It is assumed that several computational entities locally hold objective functions and are connected by a network. The agents aim to commonly minimize the sum of the local objectives subject by making gradient updates and exchanging information with their immediate neighbors. Theory of decentralized optimization is pretty well-developed in the literature. In particular, it includes lower bounds and optimal algorithms. In this paper, we assume that along with an objective, each node also holds affine constraints. We discuss several primal and dual approaches to decentralized optimization problem with affine constraints.

math.OC↗