arXiv subjects

Soummya Kar

Publications and source records attributed to Soummya Kar.

At least 55 records · Page 3Linked to original sources

arXiv2022

Large deviations rates for stochastic gradient descent with strongly convex functions

Recent works have shown that high probability metrics with stochastic gradient descent (SGD) exhibit informativeness and in some cases advantage over the commonly adopted mean-square error-based ones. In this work we provide a formal framework for the study of general high probability bounds with SGD, based on the theory of large deviations. The framework allows for a generic (not-necessarily bounded) gradient noise satisfying mild technical assumptions, allowing for the dependence of the noise distribution on the current iterate. Under the preceding assumptions, we find an upper large deviations bound for SGD with strongly convex functions. The corresponding rate function captures analytical dependence on the noise distribution and other problem parameters. This is in contrast with conventional mean-square error analysis that captures only the noise dependence through the variance and does not capture the effect of higher order moments nor interplay between the noise geometry and the shape of the cost function. We also derive exact large deviation rates for the case when the objective function is quadratic and show that the obtained function matches the one from the general upper bound hence showing the tightness of the general upper bound. Numerical examples illustrate and corroborate theoretical findings.

cs.LG↗

arXiv2022

Secure Distributed Optimization Under Gradient Attacks

In this paper, we study secure distributed optimization against arbitrary gradient attack in multi-agent networks. In distributed optimization, there is no central server to coordinate local updates, and each agent can only communicate with its neighbors on a predefined network. We consider the scenario where out of $n$ networked agents, a fixed but unknown fraction $ρ$ of the agents are under arbitrary gradient attack in that their stochastic gradient oracles return arbitrary information to derail the optimization process, and the goal is to minimize the sum of local objective functions on unattacked agents. We propose a distributed stochastic gradient method that combines local variance reduction and clipping (CLIP-VRG). We show that, in a connected network, when unattacked local objective functions are convex and smooth, share a common minimizer, and their sum is strongly convex, CLIP-VRG leads to almost sure convergence of the iterates to the exact sum cost minimizer at all agents. We quantify a tight upper bound of the fraction $ρ$ of attacked agents in terms of problem parameters such as the condition number of the associated sum cost that guarantee exact convergence of CLIP-VRG, and characterize its asymptotic convergence rate. Finally, we empirically demonstrate the effectiveness of the proposed method under gradient attacks in both synthetic dataset and image classification datasets.

math.OC↗

arXiv2022

Numerical Comparisons of Linear Power Flow Approximations: Optimality, Feasibility, and Computation Time

Linear approximations of the AC power flow equations are of great significance for the computational efficiency of large-scale optimal power flow (OPF) problems. Put differently, the feasibility of the obtained solution is essential for practical use cases of OPF. However, most studies focus on approximation error and come short of comprehensively studying the AC feasibility of different linear approximations of power flow. This paper discusses the merits of widely-used linear approximations of active power in OPF problems. The advantages and disadvantages of the linearized models are discussed with respect to four criteria; accuracy of the linear approximation, optimality, feasibility, and computation time. Each method is tested on five different systems.

eess.SY↗

arXiv2022

Gradient Based Clustering

We propose a general approach for distance based clustering, using the gradient of the cost function that measures clustering quality with respect to cluster assignments and cluster center positions. The approach is an iterative two step procedure (alternating between cluster assignment and cluster center updates) and is applicable to a wide range of functions, satisfying some mild assumptions. The main advantage of the proposed approach is a simple and computationally cheap update rule. Unlike previous methods that specialize to a specific formulation of the clustering problem, our approach is applicable to a wide range of costs, including non-Bregman clustering methods based on the Huber loss. We analyze the convergence of the proposed algorithm, and show that it converges to the set of appropriately defined fixed points, under arbitrary center initialization. In the special case of Bregman cost functions, the algorithm converges to the set of centroidal Voronoi partitions, which is consistent with prior works. Numerical experiments on real data demonstrate the effectiveness of the proposed method.

cs.LG↗

arXiv2022

Nonlinear gradient mappings and stochastic optimization: A general framework with applications to heavy-tail noise

We introduce a general framework for nonlinear stochastic gradient descent (SGD) for the scenarios when gradient noise exhibits heavy tails. The proposed framework subsumes several popular nonlinearity choices, like clipped, normalized, signed or quantized gradient, but we also consider novel nonlinearity choices. We establish for the considered class of methods strong convergence guarantees assuming a strongly convex cost function with Lipschitz continuous gradients under very general assumptions on the gradient noise. Most notably, we show that, for a nonlinearity with bounded outputs and for the gradient noise that may not have finite moments of order greater than one, the nonlinear SGD's mean squared error (MSE), or equivalently, the expected cost function's optimality gap, converges to zero at rate~$O(1/t^ζ)$, $ζ\in (0,1)$. In contrast, for the same noise setting, the linear SGD generates a sequence with unbounded variances. Furthermore, for the nonlinearities that can be decoupled component wise, like, e.g., sign gradient or component-wise clipping, we show that the nonlinear SGD asymptotically (locally) achieves a $O(1/t)$ rate in the weak convergence sense and explicitly quantify the corresponding asymptotic variance. Experiments show that, while our framework is more general than existing studies of SGD under heavy-tail noise, several easy-to-implement nonlinearities from our framework are competitive with state of the art alternatives on real data sets with heavy tail noises.

math.OC↗

arXiv2022

Distributed Stochastic Gradient Descent: Nonconvexity, Nonsmoothness, and Convergence to Local Minima

In centralized settings, it is well known that stochastic gradient descent (SGD) avoids saddle points and converges to local minima in nonconvex problems. However, similar guarantees are lacking for distributed first-order algorithms. The paper studies distributed stochastic gradient descent (D-SGD)--a simple network-based implementation of SGD. Conditions under which D-SGD avoids saddle points and converges to local minima are studied. First, we consider the problem of computing critical points. Assuming loss functions are nonconvex and possibly nonsmooth, it is shown that, for each fixed initialization, D-SGD converges to critical points of the loss with probability one. Next, we consider the problem of avoiding saddle points. In this case, we again assume that loss functions may be nonconvex and nonsmooth, but are smooth in a neighborhood of a saddle point. It is shown that, for any fixed initialization, D-SGD avoids such saddle points with probability one. Results are proved by studying the underlying (distributed) gradient flow, using the ordinary differential equation (ODE) method of stochastic approximation, and extending classical techniques from dynamical systems theory such as stable manifolds. Results are proved in the general context of subspace-constrained optimization, of which D-SGD is a special case.

math.OC↗

arXiv2022

Personalized Federated Learning via Convex Clustering

We propose a parametric family of algorithms for personalized federated learning with locally convex user costs. The proposed framework is based on a generalization of convex clustering in which the differences between different users' models are penalized via a sum-of-norms penalty, weighted by a penalty parameter $λ$. The proposed approach enables "automatic" model clustering, without prior knowledge of the hidden cluster structure, nor the number of clusters. Analytical bounds on the weight parameter, that lead to simultaneous personalization, generalization and automatic model clustering are provided. The solution to the formulated problem enables personalization, by providing different models across different clusters, and generalization, by providing models different than the per-user models computed in isolation. We then provide an efficient algorithm based on the Parallel Direction Method of Multipliers (PDMM) to solve the proposed formulation in a federated server-users setting. Numerical experiments corroborate our findings. As an interesting byproduct, our results provide several generalizations to convex clustering.

cs.LG↗

arXiv2022

A Fully Decentralized Tuning-free Inexact Projection Method for P2P Energy Trading

Agent-based solutions lend themselves well to address privacy concerns and the computational scalability needs of future distributed electric grids and end-use energy exchanges. Decentralized decision-making methods are the key to enabling peer-to-peer energy trading between electricity prosumers. However, the performance of existing decentralized decision-making algorithms highly depends on the algorithmic design and hyperparameter tunings, limiting applicability. This paper aims to address this gap by proposing a decentralized inexact projection method that does not rely on parameter tuning or central coordination to achieve the optimal solution for Peer-to-Peer (P2P) energy trading problems. The proposed algorithm does not require parameter readjustments, and once tuned, it converges for a wide range of P2P setups. Moreover, each prosumer only needs to share limited information (i.e., updated coupled variable) with neighboring prosumers. The IEEE 13 bus test system is used to showcase our proposed method's robustness and privacy advantages.

eess.SY↗

arXiv2022

Variance reduced stochastic optimization over directed graphs with row and column stochastic weights

This paper proposes AB-SAGA, a first-order distributed stochastic optimization method to minimize a finite-sum of smooth and strongly convex functions distributed over an arbitrary directed graph. AB-SAGA removes the uncertainty caused by the stochastic gradients using a node-level variance reduction and subsequently employs network-level gradient tracking to address the data dissimilarity across the nodes. Unlike existing methods that use the nonlinear push-sum correction to cancel the imbalance caused by the directed communication, the consensus updates in AB-SAGA are linear and uses both row and column stochastic weights. We show that for a constant step-size, AB-SAGA converges linearly to the global optimal. We quantify the directed nature of the underlying graph using an explicit directivity constant and characterize the regimes in which AB-SAGA achieves a linear speed-up over its centralized counterpart. Numerical experiments illustrate the convergence of AB-SAGA for strongly convex and nonconvex problems.

math.OC↗

arXiv2021

Distributed design of deterministic discrete-time privacy preserving average consensus for multi-agent systems through network augmentation

Average consensus protocols emerge with a central role in distributed systems and decision-making such as distributed information fusion, distributed optimization, distributed estimation, and control. A key advantage of these protocols is that agents exchange and reveal their state information only to their neighbors. Yet, it can raise privacy concerns in situations where the agents' states contain sensitive information. In this paper, we propose a novel (noiseless) privacy preserving distributed algorithms for multi-agent systems to reach an average consensus. The main idea of the algorithms is that each agent runs a (small) network with a crafted structure and dynamics to form a network of networks (i.e., the connection between the newly created networks and their interconnections respecting the initial network connections). Together with a re-weighting of the dynamic parameters dictating the inter-agent dynamics and the initial states, we show that it is possible to ensure that the value of each node converges to the consensus value of the original network. Furthermore, we show that, under mild assumptions, it is possible to craft the dynamics such that the design can be achieved in a distributed fashion. Finally, we illustrate the proposed algorithm with examples.

math.OC↗

arXiv2021

Dynamic Median Consensus Over Random Networks

This paper studies the problem of finding the median of N distinct numbers distributed across networked agents. Each agent updates its estimate for the median from noisy local observations of one of the N numbers and information from neighbors. We consider an undirected random network that is connected on average, and a noisy observation sequence that has finite variance and almost surely decaying bias. We present a consensus+innovations algorithm with clipped innovations. Under some regularity assumptions on the network and observation model, we show that each agent's local estimate converges to the set of median(s) almost surely at an asymptotic sublinear rate. Numerical experiments demonstrate the effectiveness of the presented algorithm.

math.OC↗

arXiv2021

A Stochastic Proximal Gradient Framework for Decentralized Non-Convex Composite Optimization: Topology-Independent Sample Complexity and Communication Efficiency

Decentralized optimization is a promising parallel computation paradigm for large-scale data analytics and machine learning problems defined over a network of nodes. This paper is concerned with decentralized non-convex composite problems with population or empirical risk. In particular, the networked nodes are tasked to find an approximate stationary point of the average of local, smooth, possibly non-convex risk functions plus a possibly non-differentiable extended valued convex regularizer. Under this general formulation, we propose the first provably efficient, stochastic proximal gradient framework, called ProxGT. Specifically, we construct and analyze several instances of ProxGT that are tailored respectively for different problem classes of interest. Remarkably, we show that the sample complexities of these instances are network topology-independent and achieve linear speedups compared to that of the corresponding centralized optimal methods implemented on a single node.

math.OC↗

arXiv2021

A fast randomized incremental gradient method for decentralized non-convex optimization

We study decentralized non-convex finite-sum minimization problems described over a network of nodes, where each node possesses a local batch of data samples. In this context, we analyze a single-timescale randomized incremental gradient method, called GT-SAGA. GT-SAGA is computationally efficient as it evaluates one component gradient per node per iteration and achieves provably fast and robust performance by leveraging node-level variance reduction and network-level gradient tracking. For general smooth non-convex problems, we show the almost sure and mean-squared convergence of GT-SAGA to a first-order stationary point and further describe regimes of practical significance where it outperforms the existing approaches and achieves a network topology-independent iteration complexity respectively. When the global function satisfies the Polyak-Lojaciewisz condition, we show that GT-SAGA exhibits linear convergence to an optimal solution in expectation and describe regimes of practical interest where the performance is network topology-independent and improves upon the existing methods. Numerical experiments are included to highlight the main convergence aspects of GT-SAGA in non-convex settings.

math.OC↗

arXiv2021

Fast decentralized non-convex finite-sum optimization with recursive variance reduction

This paper considers decentralized minimization of $N:=nm$ smooth non-convex cost functions equally divided over a directed network of $n$ nodes. Specifically, we describe a stochastic first-order gradient method, called GT-SARAH, that employs a SARAH-type variance reduction technique and gradient tracking (GT) to address the stochastic and decentralized nature of the problem. We show that GT-SARAH, with appropriate algorithmic parameters, finds an $ε$-accurate first-order stationary point with $O\big(\max\big\{N^{\frac{1}{2}},n(1-λ)^{-2},n^{\frac{2}{3}}m^{\frac{1}{3}}(1-λ)^{-1}\big\}Lε^{-2}\big)$ gradient complexity, where ${(1-λ)\in(0,1]}$ is the spectral gap of the network weight matrix and $L$ is the smoothness parameter of the cost functions. This gradient complexity outperforms that of the existing decentralized stochastic gradient methods. In particular, in a big-data regime such that ${n = O(N^{\frac{1}{2}}(1-λ)^{3})}$, this gradient complexity furthers reduces to ${O(N^{\frac{1}{2}}Lε^{-2})}$, independent of the network topology, and matches that of the centralized near-optimal variance-reduced methods. Moreover, in this regime GT-SARAH achieves a non-asymptotic linear speedup, in that, the total number of gradient computations at each node is reduced by a factor of $1/n$ compared to the centralized near-optimal algorithms that perform all gradient computations at a single node. To the best of our knowledge, GT-SARAH is the first algorithm that achieves this property. In addition, we show that appropriate choices of local minibatch size balance the trade-offs between the gradient and communication complexity of GT-SARAH. Over infinite time horizon, we establish that all nodes in GT-SARAH asymptotically achieve consensus and converge to a first-order stationary point in the almost sure and mean-squared sense.

math.OC↗

arXiv2021

A Hybrid Variance-Reduced Method for Decentralized Stochastic Non-Convex Optimization

This paper considers decentralized stochastic optimization over a network of $n$ nodes, where each node possesses a smooth non-convex local cost function and the goal of the networked nodes is to find an $ε$-accurate first-order stationary point of the sum of the local costs. We focus on an online setting, where each node accesses its local cost only by means of a stochastic first-order oracle that returns a noisy version of the exact gradient. In this context, we propose a novel single-loop decentralized hybrid variance-reduced stochastic gradient method, called GT-HSGD, that outperforms the existing approaches in terms of both the oracle complexity and practical implementation. The GT-HSGD algorithm implements specialized local hybrid stochastic gradient estimators that are fused over the network to track the global gradient. Remarkably, GT-HSGD achieves a network topology-independent oracle complexity of $O(n^{-1}ε^{-3})$ when the required error tolerance $ε$ is small enough, leading to a linear speedup with respect to the centralized optimal online variance-reduced approaches that operate on a single node. Numerical experiments are provided to illustrate our main technical results.

math.OC↗

arXiv2021

On the Accuracy of Deterministic Models for Viral Spread on Networks

We consider the emergent behavior of viral spread when agents in a large population interact with each other over a contact network. When the number of agents is large and the contact network is a complete graph, it is well known that the population behavior -- that is, the fraction of susceptible, infected and recovered agents -- converges to the solution of an ordinary differential equation (ODE) known as the classical SIR model as the population size approaches infinity. In contrast, we study interactions over contact networks with generic topologies and derive conditions under which the population behavior concentrates around either the classic SIR model or other deterministic models. Specifically, we show that when most vertex degrees in the contact network are sufficiently large, the population behavior concentrates around an ODE known as the network SIR model. We then study the short and intermediate-term evolution of the network SIR model and show that if the contact network has an expander-type property or the initial set of infections is well-mixed in the population, the network SIR model reduces to the classical SIR model. To complement these results, we illustrate through simulations that the two models can yield drastically different predictions, hence use of the classical SIR model can be misleading in certain cases.

physics.soc-ph↗

arXiv2021

Finite-Time In-Network Computation of Linear Transforms

This paper focuses on finite-time in-network computation of linear transforms of distributed graph data. Finite-time transform computation problems are of interest in graph-based computing and signal processing applications in which the objective is to compute, by means of distributed iterative methods, various (linear) transforms of the data distributed at the agents or nodes of the graph. While finite-time computation of consensus-type or more generally rank-one transforms have been studied, systematic approaches toward scalable computing of general linear transforms, specifically in the case of heterogeneous agent objectives in which each agent is interested in obtaining a different linear combination of the network data, are relatively less explored. In this paper, by employing ideas from algebraic geometry, we develop a systematic characterization of linear transforms that are amenable to distributed in-network computation in finite-time using linear iterations. Further, we consider the general case of directed inter-agent communication graphs. Specifically, it is shown that \emph{almost all} linear transformations of data distributed on the nodes of a digraph containing a Hamiltonian cycle may be computed using at most $N$ linear distributed iterations. Finally, by studying an associated matrix factorization based reformulation of the transform computation problem, we obtain, as a by-product, certain results and characterizations on sparsity-constrained matrix factorization that are of independent interest.

math.OC↗

arXiv2020

An improved convergence analysis for decentralized online stochastic non-convex optimization

In this paper, we study decentralized online stochastic non-convex optimization over a network of nodes. Integrating a technique called gradient tracking in decentralized stochastic gradient descent, we show that the resulting algorithm, GT-DSGD, enjoys certain desirable characteristics towards minimizing a sum of smooth non-convex functions. In particular, for general smooth non-convex functions, we establish non-asymptotic characterizations of GT-DSGD and derive the conditions under which it achieves network-independent performances that match the centralized minibatch SGD. In contrast, the existing results suggest that GT-DSGD is always network-dependent and is therefore strictly worse than the centralized minibatch SGD. When the global non-convex function additionally satisfies the Polyak-Lojasiewics (PL) condition, we establish the linear convergence of GT-DSGD up to a steady-state error with appropriate constant step-sizes. Moreover, under stochastic approximation step-sizes, we establish, for the first time, the optimal global sublinear convergence rate on almost every sample path, in addition to the asymptotically optimal sublinear rate in expectation. Since strongly convex functions are a special case of the functions satisfying the PL condition, our results are not only immediately applicable but also improve the currently known best convergence rates and their dependence on problem parameters.

math.OC↗