arXiv subjects

Thinh T. Doan

Publications and source records attributed to Thinh T. Doan.

46 records · Page 3Linked to original sources

arXiv2019

A Reinforcement Learning Framework for Sequencing Multi-Robot Behaviors

Given a list of behaviors and associated parameterized controllers for solving different individual tasks, we study the problem of selecting an optimal sequence of coordinated behaviors in multi-robot systems for completing a given mission, which could not be handled by any single behavior. In addition, we are interested in the case where partial information of the underlying mission is unknown, therefore, the robots must cooperatively learn this information through their course of actions. Such problem can be formulated as an optimal decision problem in multi-robot systems, however, it is in general intractable due to modeling imperfections and the curse of dimensionality of the decision variables. To circumvent these issues, we first consider an alternate formulation of the original problem through introducing a sequence of behaviors' switching times. Our main contribution is then to propose a novel reinforcement learning based method, that combines Q-learning and online gradient descent, for solving this reformulated problem. In particular, the optimal sequence of the robots' behaviors is found by using Q-learning while the optimal parameters of the associated controllers are obtained through an online gradient descent method. Finally, to illustrate the effectiveness of our proposed method we implement it on a team of differential-drive robots for solving two different missions, namely, convoy protection and object manipulation.

cs.RO↗

arXiv2019

Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning

We study the policy evaluation problem in multi-agent reinforcement learning. In this problem, a group of agents works cooperatively to evaluate the value function for the global discounted accumulative reward problem, which is composed of local rewards observed by the agents. Over a series of time steps, the agents act, get rewarded, update their local estimate of the value function, then communicate with their neighbors. The local update at each agent can be interpreted as a distributed consensus-based variant of the popular temporal difference learning algorithm TD(0). While distributed reinforcement learning algorithms have been presented in the literature, almost nothing is known about their convergence rate. Our main contribution is providing a finite-time analysis for the convergence of the distributed TD(0) algorithm. We do this when the communication network between the agents is time-varying in general. We obtain an explicit upper bound on the rate of convergence of this algorithm as a function of the network topology and the discount factor. Our results mirror what we would expect from using distributed stochastic gradient descent for solving convex optimization problems.

math.OC↗

arXiv2019

On the convergence rate of distributed gradient methods for finite-sum optimization under communication delays

Motivated by applications in machine learning and statistics, we study distributed optimization problems over a network of processors, where the goal is to optimize a global objective composed of a sum of local functions. In these problems, due to the large scale of the data sets, the data and computation must be distributed over processors resulting in the need for distributed algorithms. In this paper, we consider a popular distributed gradient-based consensus algorithm, which only requires local computation and communication. An important problem in this area is to analyze the convergence rate of such algorithms in the presence of communication delays that are inevitable in distributed systems. We prove the convergence of the gradient-based consensus algorithm in the presence of uniform, but possibly arbitrarily large, communication delays between the processors. Moreover, we obtain an upper bound on the rate of convergence of the algorithm as a function of the network size, topology, and the inter-processor communication delays.

math.OC↗

arXiv2019

Fast Convergence Rates of Distributed Subgradient Methods with Adaptive Quantization

We study distributed optimization problems over a network when the communication between the nodes is constrained, and so information that is exchanged between the nodes must be quantized. Recent advances using the distributed gradient algorithm with a quantization scheme at a fixed resolution have established convergence, but at rates significantly slower than when the communications are unquantized. In this paper, we introduce a novel quantization method, which we refer to as adaptive quantization, that allows us to match the convergence rates under perfect communications. Our approach adjusts the quantization scheme used by each node as the algorithm progresses: as we approach the solution, we become more certain about where the state variables are localized, and adapt the quantizer codebook accordingly. We bound the convergence rates of the proposed method as a function of the communication bandwidth, the underlying network topology, and structural properties of the constituent objective functions. In particular, we show that if the objective functions are convex or strongly convex, then using adaptive quantization does not affect the rate of convergence of the distributed subgradient methods when the communications are quantized, except for a constant that depends on the resolution of the quantizer. To the best of our knowledge, the rates achieved in this paper are better than any existing work in the literature for distributed gradient methods under finite communication bandwidths. We also provide numerical simulations that compare convergence properties of the distributed gradient methods with and without quantization for solving distributed regression problems for both quadratic and absolute loss functions.

math.OC↗

arXiv2018

Distributed Stochastic Approximation for Solving Network Optimization Problems Under Random Quantization

We study distributed optimization problems over a network when the communication between the nodes is constrained, and so information that is exchanged between the nodes must be quantized. This imperfect communication poses a fundamental challenge, and this imperfect communication, if not properly accounted for, prevents the convergence of these algorithms. Our first contribution in this paper is to propose a modified consensus-based gradient method for solving such problems using random (dithered) quantization. This algorithm can be interpreted as a distributed variant of a well-known two-time-scale stochastic algorithm. We then study the convergence and derive upper bounds on the rates of convergence of the proposed method as a function of the bandwidths available between the nodes and the underlying network topology, for both convex and strongly convex objective functions. Our results complement for existing literature where such convergence and explicit formulas of the convergence rates are missing. Finally, we provide numerical simulations to compare the convergence properties of the distributed gradient methods with and without quantization for solving the well-known regression problems over networks, for both quadratic and absolute loss functions.

math.OC↗

arXiv2018

Distributed Resource Allocation Over Dynamic Networks with Uncertainty

Motivated by broad applications in various fields of engineering, we study a network resource allocation problem where the goal is to optimally allocate a fixed quantity of resources over a network of nodes. We consider large scale networks with complex interconnection structures, thus any solution must be implemented in parallel and based only on local data resulting in a need for distributed algorithms. In this paper, we study a distributed Lagrangian method for such problems. By utilizing the so-called distributed subgradient methods to solve the dual problem, our approach eliminates the need for central coordination in updating the dual variables, which is often required in classic Lagrangian methods. Our focus is to understand the performance of this distributed algorithm when the number of resources is unknown and may be time-varying. In particular, we obtain an upper bound on the convergence rate of the algorithm to the optimal value, in expectation, as a function of the size and the topology of the underlying network. The effectiveness of the proposed method is demonstrated by its application to the economic dispatch problem in power systems, with simulations completed on the benchmark IEEE-14 and IEEE-118 bus test systems.

math.OC↗

arXiv2018

Convergence of the Iterates in Mirror Descent Methods

We consider centralized and distributed mirror descent algorithms over a finite-dimensional Hilbert space, and prove that the problem variables converge to an optimizer of a possibly nonsmooth function when the step sizes are square summable but not summable. Prior literature has focused on the convergence of the function value to its optimum. However, applications from distributed optimization and learning in games require the convergence of the variables to an optimizer, which is generally not guaranteed without assuming strong convexity of the objective function. We provide numerical simulations comparing entropic mirror descent and standard subgradient methods for the robust regression problem.

math.OC↗

arXiv2017

Distributed Lagrangian Methods for Network Resource Allocation

Motivated by a variety of applications in control engineering and information sciences, we study network resource allocation problems where the goal is to optimally allocate a fixed amount of resource over a network of nodes. In these problems, due to the large scale of the network and complicated inter-connections between nodes, any solution must be implemented in parallel and based only on local data resulting in a need for distributed algorithms. In this paper, we propose a novel distributed Lagrangian method, which requires only local computation and communication. Our focus is to understand the performance of this algorithm on the underlying network topology. Specifically, we obtain an upper bound on the rate of convergence of the algorithm as a function of the size and the topology of the underlying network. The effectiveness and applicability of the proposed method is demonstrated by its use in solving the important economic dispatch problem in power systems, specifically on the benchmark IEEE-14 and IEEE-118 bus systems.

math.OC↗

arXiv2016

On the geometric convergence rate of distributed economic dispatch/demand response in power networks

Motivated by potential applications in power systems, we study a problem of optimizing a sum of $n$ convex functions on dynamic networks of $n$ nodes when each function is known to only a single node. The nodes' variables, while satisfy their local constraints, are coupled through a linear constraint. Our main contribution is to design a fully distributed primal-dual method for this problem. Under some fairly standard assumptions on objective functions, strong convexity and smoothness, we provide an explicit analysis for the convergence rate of our method on different networks. In particular, the nodes variables achieve a geometric convergence to the optimal with the associated convergence time scales quartically in the number of nodes on any sequence of time-varying undirected graphs satisfying a long-term connectivity condition. Moreover, this convergence time is constant independent on the number of nodes when the network is a b-regular simple graph with $b\geq 3$. Finally, to show the effectiveness of our method we also simulate a number of studies on economic dispatch problems and demand response problems in power systems.

math.OC↗

arXiv2016

Distributed Resource Allocation on Dynamic Networks in Quadratic Time

We consider the problem of allocating a fixed amount of resource among nodes in a network when each node suffers a cost which is a convex function of the amount of resource allocated to it. We propose a new deterministic and distributed protocol for this problem. Our main result is that the associated convergence time for the global objective scales quadratically in the number of nodes on any sequence of time-varying undirected graphs satisfying a long-term connectivity condition.

math.OC↗