Searcharxiv⌕ Search

arXiv subjects

Siva Theja Maguluri

Publications and source records attributed to Siva Theja Maguluri.

At least 55 records · Page 3Linked to original sources
arXiv2021

Stationary Behavior of Constant Stepsize SGD Type Algorithms: An Asymptotic Characterization

Stochastic approximation (SA) and stochastic gradient descent (SGD) algorithms are work-horses for modern machine learning algorithms. Their constant stepsize variants are preferred in practice due to fast convergence behavior. However, constant step stochastic iterative algorithms do not converge asymptotically to the optimal solution, but instead have a stationary distribution, which in general cannot be analytically characterized. In this work, we study the asymptotic behavior of the appropriately scaled stationary distribution, in the limit when the constant stepsize goes to zero. Specifically, we consider the following three settings: (1) SGD algorithms with smooth and strongly convex objective, (2) linear SA algorithms involving a Hurwitz matrix, and (3) nonlinear SA algorithms involving a contractive operator. When the iterate is scaled by $1/\sqrtα$, where $α$ is the constant stepsize, we show that the limiting scaled stationary distribution is a solution of an integral equation. Under a uniqueness assumption (which can be removed in certain settings) on this equation, we further characterize the limiting distribution as a Gaussian distribution whose covariance matrix is the unique solution of a suitable Lyapunov equation. For SA algorithms beyond these cases, our numerical experiments suggest that unlike central limit theorem type results: (1) the scaling factor need not be $1/\sqrtα$, and (2) the limiting distribution need not be Gaussian. Based on the numerical study, we come up with a formula to determine the right scaling factor, and make insightful connection to the Euler-Maruyama discretization scheme for approximating stochastic differential equations.

cs.LG↗
arXiv2021

Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes

Stochastic Approximation (SA) is a popular approach for solving fixed-point equations where the information is corrupted by noise. In this paper, we consider an SA involving a contraction mapping with respect to an arbitrary norm, and show its finite-sample error bounds while using different stepsizes. The idea is to construct a smooth Lyapunov function using the generalized Moreau envelope, and show that the iterates of SA have negative drift with respect to that Lyapunov function. Our result is applicable in Reinforcement Learning (RL). In particular, we use it to establish the first-known convergence rate of the V-trace algorithm for off-policy TD-learning. Moreover, we also use it to study TD-learning in the on-policy setting, and recover the existing state-of-the-art results for $Q$-learning. Importantly, our construction results in only a logarithmic dependence of the convergence bound on the size of the state-space.

cs.LG↗
arXiv2021

Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators

In temporal difference (TD) learning, off-policy sampling is known to be more practical than on-policy sampling, and by decoupling learning from data collection, it enables data reuse. It is known that policy evaluation (including multi-step off-policy importance sampling) has the interpretation of solving a generalized Bellman equation. In this paper, we derive finite-sample bounds for any general off-policy TD-like stochastic approximation algorithm that solves for the fixed-point of this generalized Bellman operator. Our key step is to show that the generalized Bellman operator is simultaneously a contraction mapping with respect to a weighted $\ell_p$-norm for each $p$ in $[1,\infty)$, with a common contraction factor. Off-policy TD-learning is known to suffer from high variance due to the product of importance sampling ratios. A number of algorithms (e.g. $Q^π(λ)$, Tree-Backup$(λ)$, Retrace$(λ)$, and $Q$-trace) have been proposed in the literature to address this issue. Our results immediately imply finite-sample bounds of these algorithms. In particular, we provide first-known finite-sample guarantees for $Q^π(λ)$, Tree-Backup$(λ)$, and Retrace$(λ)$, and improve the best known bounds of $Q$-trace in [19]. Moreover, we show the bias-variance trade-offs in each of these algorithms.

cs.LG↗
arXiv2021

Throughput Optimal Routing in Blockchain Based Payment Systems

Cryptocurrency networks such as Bitcoin have emerged as a distributed alternative to traditional centralized financial transaction networks. However, there are major challenges in scaling up the throughput of such networks. Lightning network and Spider network are alternates that build bidirectional payment channels on top of cryptocurrency networks using smart contracts, to enable fast transactions that bypass the Blockchain. In this paper, we study the problem of routing transactions in such a payment processing network. We first propose a Stochastic model to study such a system, as opposed to a fluid model that is studied in the literature. Each link in such a model is a two-sided queue, and unlike classical queues, such queues are not stable unless there is an external control. We propose a notion of stability for the payment processing network consisting of such two-sided queues using the notion of on-chain rebalancing. We then characterize the capacity region and propose a throughput optimal algorithm that stabilizes the system under any load within the capacity region. The stochastic model enables us to study closed loop policies, which typically have better queuing/delay performance than the open loop policies (or static split rules) studied in the literature. We investigate this through simulations.

cs.DC↗
arXiv2021

Near Optimal Control in Ride Hailing Platforms with Strategic Servers

Motivated by applications in online marketplaces such as ride-hailing, we study how strategic servers impact the system performance. We consider a discrete-time process in which, heterogeneous types of customers and servers arrive. Each customer joins their type's queue, while servers might join a different type's queue depending on the prices posted by the system operator and an inconvenience cost. Then the system operator, constrained by a compatibility graph, decides the matching. The objective is to design an optimal control (pricing and matching scheme) to maximize the profit minus the expected waiting times. We develop a general framework that enables us to analyze a broad range of strategic behaviors. In particular, we encode servers' behavior in a properly defined \emph{cost function} that can be tailored to various settings. Using this general cost function, we introduce a novel probabilistic fluid problem. The probabilistic fluid model provides an upper bound on the achievable net profit. We then study the system under a large market regime in which the arrival rates are scaled by $η$ and present a probabilistic two-price policy and a max-weight matching policy which results in a net profit-loss of at most $O(η^{1/3})$. In addition, under a broad class of customer pricing policies, we show that any matching policy has net profit-loss of at least $Ω(η^{1/3})$. To show generality of our framework, we present multiple extensions to our model and analysis. We conclude the discussion by presenting numerical simulations comparing different cost models and analyzing performance of the proposed pricing and matching policies.

math.OC↗
arXiv2021

Finite-Sample Analysis of Off-Policy Natural Actor-Critic Algorithm

In this paper, we provide finite-sample convergence guarantees for an off-policy variant of the natural actor-critic (NAC) algorithm based on Importance Sampling. In particular, we show that the algorithm converges to a global optimal policy with a sample complexity of $\mathcal{O}(ε^{-3}\log^2(1/ε))$ under an appropriate choice of stepsizes. In order to overcome the issue of large variance due to Importance Sampling, we propose the $Q$-trace algorithm for the critic, which is inspired by the V-trace algorithm \cite{espeholt2018impala}. This enables us to explicitly control the bias and variance, and characterize the trade-off between them. As an advantage of off-policy sampling, a major feature of our result is that we do not need any additional assumptions, beyond the ergodicity of the Markov chain induced by the behavior policy.

cs.LG↗
arXiv2021

Low-Complexity Switch Scheduling Algorithms: Delay Optimality in Heavy Traffic

Motivated by applications in data center networks, in this paper, we study the problem of scheduling in an input queued switch. While throughput maximizing algorithms in a switch are well-understood, delay analysis was developed only recently. It was recently shown that the well-known MaxWeight algorithm achieves optimal scaling of mean queue lengths in steady state in the heavy-traffic regime, and is within a factor less than $2$ of a universal lower bound. However, MaxWeight is not used in practice because of its high time complexity. In this paper, we study several low complexity algorithms and show that their heavy-traffic performance is identical to that of MaxWeight. We first present a negative result that picking a random schedule does not have optimal heavy-traffic scaling of queue lengths even under uniform traffic. We then show that if one picks the best among two matchings or modifies a random matching even a little, using the so-called flip operation, it leads to MaxWeight like heavy-traffic performance under uniform traffic. We then focus on the case of non-uniform traffic and show that a large class of low time complexity algorithms have the same heavy-traffic performance as MaxWeight, as long as it is ensured that a MaxWeight matching is picked often enough. We also briefly discuss the performance of these algorithms in the large scale heavy-traffic regime when the size of the switch increases simultaneously with the load. Finally, we perform empirical study on a new algorithm to compare its performance with some existing algorithms.

math.PR↗
arXiv2021

Load balancing system under Join the Shortest Queue: Many-Server-Heavy-Traffic Asymptotics

We study the load balancing system operating under Join the Shortest Queue (JSQ) in the many-server heavy-traffic regime. If $N$ is the number of servers, we let the difference between the total service rate and the total arrival rate be $N^{1-α}$ with $α>0$. We show that for $α>4$ the average queue length behaves similarly to the classical heavy-traffic regime. Specifically, we prove that the distribution of the average queue length multiplied by $N^{1-α}$ converges to an exponential random variable. Moreover, we show a result analogous to state space collapse. We provide two proofs for our result: one using the one-sided Laplace transform, and one using Stein's method. We additionally obtain the rate of convergence in the Wasserstein's distance.

math.PR↗
arXiv2021

Optimal Pricing in Multi Server Systems

We study optimal service pricing in server farms where customers arrive according to a renewal process and have independent and identical ($i.i.d.$) exponential service times and $i.i.d.$ valuations of the service. The service provider charges a time varying service fee aiming at maximizing its revenue rate. The customers that find free servers and service fees lesser than their valuation join for the service else they leave without waiting. We consider both finite server and infinite server farms. We solve the optimal pricing problems using the framework of Markov decision problems. We show that the optimal prices depend on the number of free servers. We propose algorithms to compute the optimal prices. We also establish several properties of the optimal prices and the corresponding revenue rates in the case of Poisson customer arrivals. We illustrate all our findings via numerical results.

math.OC↗
arXiv2021

On the Linear convergence of Natural Policy Gradient Algorithm

Markov Decision Processes are classically solved using Value Iteration and Policy Iteration algorithms. Recent interest in Reinforcement Learning has motivated the study of methods inspired by optimization, such as gradient ascent. Among these, a popular algorithm is the Natural Policy Gradient, which is a mirror descent variant for MDPs. This algorithm forms the basis of several popular Reinforcement Learning algorithms such as Natural actor-critic, TRPO, PPO, etc, and so is being studied with growing interest. It has been shown that Natural Policy Gradient with constant step size converges with a sublinear rate of O(1/k) to the global optimal. In this paper, we present improved finite time convergence bounds, and show that this algorithm has geometric (also known as linear) asymptotic convergence rate. We further improve this convergence result by introducing a variant of Natural Policy Gradient with adaptive step sizes. Finally, we compare different variants of policy gradient methods experimentally.

cs.LG↗
arXiv2021

Heavy-Traffic Insensitive Bounds for Weighted Proportionally Fair Bandwidth Sharing Policies

We consider a connection-level model proposed by Massoulié and Roberts for bandwidth sharing among file transfer flows in a communication network. We study weighted proportionally fair sharing policies and establish explicit-form bounds on the weighted sum of the expected numbers of flows on different routes in heavy traffic. The bounds are linear in the number of critically loaded links in the network, and they hold for a class of phase-type file-size distributions; i.e., the bounds are heavy-traffic insensitive to the distributions in this class. Our approach is Lyapunov-drift based, which is different from the widely used diffusion approximation approach. A key technique we develop is to construct a novel inner product in the state space, which then allows us to obtain a multiplicative type of state-space collapse in steady state. Furthermore, this state-space collapse result implies the interchange of limits as a by-product for the diffusion approximation of the equal-weight case under phase-type file-size distributions, demonstrating the heavy-traffic insensitivity of the stationary distribution.

cs.PF↗
arXiv2020

Throughput and Delay Optimality of Power-of-d Choices in Inhomogeneous Load Balancing Systems

It is well-known that the power-of-d choices routing algorithm maximizes throughput and is heavy-traffic optimal in load balancing systems with homogeneous servers. However, if the servers are heterogeneous, throughput optimality does not hold in general. We find necessary and sufficient conditions for throughput optimality of power-of-d choices when the servers are heterogeneous, and we prove that almost the same conditions are sufficient to show heavy-traffic optimality. Additionally, we generalize the sufficient condition for throughput optimality to a larger class of routing policies.

math.PR↗
arXiv2020

QPS-r: A Cost-Effective Crossbar Scheduling Algorithm and Its Stability and Delay Analysis

In an input-queued switch, a crossbar schedule, or a matching between the input ports and the output ports needs to be computed in each switching cycle, or time slot. Designing switching algorithms with very low computational complexity, that lead to high throughput and small delay is a challenging problem. There appears to be a fundamental tradeoff between the computational complexity of the switching algorithm and the resultants throughput and delay. Parallel maximal matching algorithms (adapted for switching) appear to have stricken a sweet spot in this tradeoff, and prior work has shown the following performance guarantees. Using maximal matchings in every time slot results in at least 50% switch throughput and order-optimal (i.e., independent of the switch size N) average delay bounds for various traffic arrival processes. On the other hand, their computational complexity can be as low as $O(log^2N)$ per port/processor, which is much lower than those of the algorithms such as maximum weighted matching which ensures better throughput performance. In this work, we propose QPS-r, a parallel iterative switching algorithm that has the lowest possible computational complexity: O(1) per port. Using Lyapunov stability analysis, we show that the throughput and delay performance is identical to that of maximal matching algorithm. Although QPS-r builds upon an existing technique called Queue-Proportional Sampling (QPS), in this paper, we provide analytical guarantees on its throughput and delay under i.i.d. traffic as well as a Markovian traffic model which can model many realistic traffic patterns. We also demonstrate that QPS-3 (running 3 iterations) has comparable empirical throughput and delay performances as iSLIP (running $log_2 N$ iterations), a refined and optimized representative maximal matching algorithm adapted for switching.

cs.PF↗
arXiv2020

Transform Methods for Heavy-Traffic Analysis

The drift method was recently developed to study queueing systems in steady-state. It was successfully used to obtain bounds on the moments of the scaled queue lengths, that are asymptotically tight in heavy-traffic, in a wide variety of systems including generalized switches, input-queued switches, bandwidth sharing networks, etc. In this paper we develop the use of transform techniques for heavy-traffic analysis, with a special focus on the use of moment generating functions. This approach simplifies the proofs of the drift method, and provides a new perspective on the drift method. We present a general framework and then use the MGF method to obtain the stationary distribution of queue lengths in heavy-traffic in queueing systems that satisfy the Complete Resource Pooling condition. In particular, we study load balancing systems and generalized switches under general settings.

math.PR↗
arXiv2020

Logarithmic Heavy Traffic Error Bounds in Generalized Switch and Load Balancing Systems

Motivated by application in wireless networks, cloud computing, data centers etc, Stochastic Processing Networks have been studied in the literature under various asymptotic regimes. In the heavy-traffic regime, the steady state mean queue length is proved to be $O(\frac{1}ε)$ where $ε$ is the heavy-traffic parameter, that goes to zero in the limit. The focus of this paper is on obtaining queue length bounds on prelimit systems, thus establishing the rate of convergence to the heavy traffic. In particular, we study the generalized switch model operating under the MaxWeight algorithm, and we show that the mean queue length of the prelimit system is only $O\left(\log \left(\frac{1}ε\right)\right)$ away from its heavy-traffic limit. We do this even when the so called complete resource pooling (CRP) condition is not satisfied. When the CRP condition is satisfied, in addition, we show that the MaxWeight algorithm is within $O\left(\log \left(\frac{1}ε\right)\right)$ of the optimal. Finally, we obtain similar results in load balancing systems operating under the join the shortest queue routing algorithm.

math.PR↗
arXiv2020

Finite-Time Performance of Distributed Temporal Difference Learning with Linear Function Approximation

We study the policy evaluation problem in multi-agent reinforcement learning, modeled by a Markov decision process. In this problem, the agents operate in a common environment under a fixed control policy, working together to discover the value (global discounted accumulative reward) associated with each environmental state. Over a series of time steps, the agents act, get rewarded, update their local estimate of the value function, then communicate with their neighbors. The local update at each agent can be interpreted as a distributed variant of the popular temporal difference learning methods {\sf TD}$ (λ)$. Our main contribution is to provide a finite-analysis on the performance of this distributed {\sf TD}$(λ)$ algorithm for both constant and time-varying step sizes. The key idea in our analysis is to use the geometric mixing time $τ$ of the underlying Markov chain, that is, although the "noise" in our algorithm is Markovian, its dependence is very weak at samples spaced out at every $τ$. We provide an explicit upper bound on the convergence rate of the proposed method as a function of the network topology, the discount factor, the constant $λ$, and the mixing time $τ$. Our results also provide a mathematical explanation for observations that have appeared previously in the literature about the choice of $λ$. Our upper bound illustrates the trade-off between approximation accuracy and convergence speed implicit in the choice of $λ$. When $λ=1$, the solution will correspond to the best possible approximation of the value function, while choosing $λ= 0$ leads to faster convergence when the noise in the algorithm has large variance.

math.OC↗
arXiv2019

Finite-Time Analysis of Distributed TD(0) with Linear Function Approximation for Multi-Agent Reinforcement Learning

We study the policy evaluation problem in multi-agent reinforcement learning. In this problem, a group of agents works cooperatively to evaluate the value function for the global discounted accumulative reward problem, which is composed of local rewards observed by the agents. Over a series of time steps, the agents act, get rewarded, update their local estimate of the value function, then communicate with their neighbors. The local update at each agent can be interpreted as a distributed consensus-based variant of the popular temporal difference learning algorithm TD(0). While distributed reinforcement learning algorithms have been presented in the literature, almost nothing is known about their convergence rate. Our main contribution is providing a finite-time analysis for the convergence of the distributed TD(0) algorithm. We do this when the communication network between the agents is time-varying in general. We obtain an explicit upper bound on the rate of convergence of this algorithm as a function of the network topology and the discount factor. Our results mirror what we would expect from using distributed stochastic gradient descent for solving convex optimization problems.

math.OC↗