arXiv subjects

Siva Theja Maguluri

Publications and source records attributed to Siva Theja Maguluri.

At least 19 recordsLinked to original sources

arXiv2026

Finite-Time Behavior of Erlang-C Model: Mixing Time, Mean Queue Length and Tail Bounds

Resource allocation problems in service systems like data centers and ride-hailing are usually studied using queueing models. Such systems are primarily studied in the steady-state and in asymptotic regimes such as under heavy traffic due to their analytical tractability. However, almost all applications in real life do not operate in asymptotic regimes, and so, there is a clear discrepancy in translating theoretical queuing results to practical applications. In this work, we bridge this gap by presenting nonasymptotic and finite-time bounds for Erlang-C systems, providing a stepping stone towards understanding the transient behavior of more general queuing systems. We bound the Chi-square distance between the finite-time queue length distribution and the stationary distribution, show that it decays exponentially fast, and characterize the rate of decay. We observe that the Erlang-C system exhibits a phase transition, depending on a parameter that measures the load relative to the size of the system. We then use these results to obtain bounds on the mean queue length and tails of the queue lengths in finite time for the nonasymptotic system. We also establish that the rate we obtain is tight up to universal constants in appropriate heavy-traffic asymptotic regimes. We obtain these results using the Lyapunov-Poincaré approach, where we first carefully design a Lyapunov function to obtain a negative drift outside a finite set. Within the finite set, we develop different strategies depending on the properties of the finite set to get a handle on the mixing behavior via a local Poincaré inequality. A key aspect of our methodological contribution is obtaining tight guarantees in these two regions, which when combined, give us tight mixing time bounds. We believe that this approach is of independent interest for studying mixing in reversible countable-state Markov chains more generally.

math.PR↗

arXiv2026

Constant-Stepsize Stochastic Approximation: Finite-Time Convergence, Gaussian Approximation, and Tail Bounds

Constant-stepsize stochastic approximation (SA) is widely used in learning for computational efficiency, yet the distribution of the iterates is typically intractable. Classical asymptotics results give $X_k^{(α)} \approx X^{(α)} \approx x^\star+\sqrtαY$, where $X^{(α)}$ is the steady state and $Y$ is an appropriate Gaussian limit, by progressively taking the time $k\uparrow\infty$ and stepsize $α\downarrow0$. Such limit results, however, do not quantify finite-time, finite-stepsize errors. We develop an explicit pre-limit characterization for SA with i.i.d.\ and Markovian noise. We establish existence and uniqueness of the stationary law, a geometric Wasserstein convergence to stationarity, and almost-sure and $L^3$ convergence of the steady state to the root $x^\star$, identifying the scale $\sqrtα$ as first-order fluctuation. At this scale, we derive a higher-order quantitative Gaussian approximation with a Wasserstein error, using Stein's method and Poisson equation techniques. We further obtain non-uniform Berry--Esseen-type tail bounds, incorporating both steady-state approximation and finite-time convergence errors. We instantiate the theory for strongly convex smooth SGD, linear SA, and nonlinear contractive SA. Beyond strong convexity, for general convex SGD, we identify a Gibbs limiting law and prove a pre-limit Wasserstein approximation error under stability and Stein-equation hypothesis, which are validated numerically.

cs.LG↗

arXiv2026

Transform Method for Stochastic Processing and Matching Networks

Modern service systems, ranging from cloud data centers and ride-hailing platforms to healthcare facilities, operate at massive scales where it is important to handle congestion. Queueing theory is used to understand the delay and queue length behavior in these systems. Except in simple queues, it is not possible to obtain a closed form solution for the quantities of interest, and so, one studies the system in certain asymptotic regimes such as the heavy traffic. The transform method, presented in this tutorial, is a framework to understand the steady-state behavior of Stochastic Processing and Matching Networks (SPNs/SMNs). By exploiting the zero-drift property of exponential test functions, the method derives explicit functional equations (acting as a proxy for global balance equations) for the transforms (such as moment-generating functions) of queue-length distributions. These functional equations can be used to either characterize the exact behavior of the system in an asymptotic regime or to obtain non-asymptotic performance bounds on the mean, higher order moments, or tail bounds on the queue lengths. Since its introduction for load-balancing in data center networks, the transform method, as a framework, has been extended to handle various features that arise in different systems, including customer abandonment, state-dependent arrivals, Markov-modulated arrivals, large-system scale, and multi-dimensional networks with multiple bottlenecks. This tutorial presents an overview of the transform method starting with the simplest setting viz., a single server queue. The transform method is introduced as a three step procedure. We then illustrate how the method can be adapted within this three-step framework to handle the aforementioned features.

math.OC↗

arXiv2026

Convergence Rate of the Join-the-Shortest-Queue System

The Join-the-Shortest-Queue (JSQ) policy is among the most widely used load balancing algorithms and has been extensively studied. However, an exact characterization of the system behavior remains challenging. Most prior research has focused on analyzing its performance in the steady state in certain asymptotic regimes, such as the heavy-traffic regime. However, convergence to the steady state in these regimes is often slow, so steady-state and heavy-traffic characterizations may be less informative over practical time horizons. To address this limitation, we provide a finite-time convergence rate analysis of a JSQ system with two symmetric servers. In sharp contrast to the existing literature, we directly study the original system rather than an approximate limiting system such as a diffusion approximation. Our results demonstrate that for such a system, the convergence rate to its steady state, measured in the total variation distance, is $O \left(\frac{1}{(1-ρ)^3} \frac{1}{t} \right)$, where $ρ\in (0,1)$ is the traffic intensity.

math.PR↗

arXiv2026

Dynamic Pricing and Matching for Two-Sided Queues

Motivated by applications from gig economy and online marketplaces, we study a two-sided queueing system under joint pricing and matching controls. The queueing system is modeled by a bipartite graph, where the vertices represent customer or server types and the edges represent compatible customer-server pairs. Both customers and servers sequentially arrive to the system and join separate queues according to their types. The arrival rates of different types depend on the prices set by the system operator and the expected waiting time. At any point in time, the system operator can choose certain customers to match with compatible servers. The objective is to maximize the long-run average profit for the system. We first propose a fluid approximation based pricing and max-weight matching policy, which achieves an $O(\sqrtη)$ optimality rate when all the arrival rates are scaled by $η$. We further show that a two-price and max-weight matching policy achieves an improved $O(η^{1/3})$ optimality rate. Under a broad class of pricing policies, we prove that any matching policy has an optimality rate that is lower bounded by $Ω(η^{1/3})$. Thus, the latter policy achieves the optimal rate with respect to $η$. We also demonstrate the advantage of max-weight matching with respect to the number of server and customer types $n$. Under a complete resource pooling condition, we show that max-weight matching achieves $O(\sqrt{n})$ and $O(n^{1/3})$ optimality rates for static and two-price policies, respectively, and the latter matches the lower bound $Ω(n^{1/3})$. In comparison, the randomized matching policy may have an $Ω(n)$ optimality rate.

math.OC↗

arXiv2026

A Heavy Traffic Theory of Matching Queues

Motivated by emerging applications in online matching platforms and marketplaces, we study a matching queue. Customers and servers that arrive in a matching queue depart as soon as they are matched. While state-dependent control is an effective lever to regulate the throughput and delay, it often comes at a cost in practice for matching platforms. Optimizing this fundamental trade-off motivates the use of small amounts of control, so we study a matching queue in an asymptotic regime where the state-dependent control decreases to zero. Unlike the heavy traffic regime in classical queues, there are two different ways the control can be sent to zero, via a magnitude scaling parameter $ε$ that goes to zero and a time scaling parameter $τ$ that goes to infinity. Depending on the cost of control, we show that the rates of $ε$ and $τ$ that optimize the trade-off between delay and cost of control could correspond to three different regimes. As we traverse these regimes, we observe a phase transition in the limiting distribution of the matching queue. We show that a low cost of control corresponds to the regime $ετ\rightarrow 0$ and we call it the delay-driven regime. The limiting behavior in this regime is an asymmetrical Laplace distribution. On the other hand, $ετ\rightarrow \infty$ is the cost-driven regime corresponding to a high cost of control where the limiting behavior is either a uniform or a truncated exponential distribution. We christen the in-between regime of $ετ\rightarrow (0, \infty)$ the hybrid regime where the limiting behavior is a Gibbs distribution. These results are obtained by novel generalizations of the transform method, where each regime requires new ideas. The hybrid regime employs inverse Fourier transforms while the other two regimes engineer multiple complex exponential test functions.

math.PR↗

arXiv2026

How Accurately Can a Gaussian Approximate Stochastic Approximation Iterates?

Stochastic approximation (SA) is a method for finding the root of an operator perturbed by noise. The focus of this paper is studying the distribution of SA iterates in finite time. In general, it is not possible to characterize the exact distribution, and therefore our goal is to find an approximation which can yield useful tail bounds. Inspired by the rich literature on the asymptotic normality of rescaled SA iterates, we approximate the pre-limit distributions by a sequence of Gaussians whose covariance is recursively defined. In particular, we establish explicit bounds on the Wasserstein-1 distance between the rescaled iterate at time $k$ and the aforementioned Gaussian for various choices of step-sizes. Since these covariances converge to the classical asymptotic limit, our analysis also provides a convergence rate for asymptotic normality as a by-product. As an immediate consequence of our bounds, we obtain tail bounds on the error of SA iterates at any time. Finally, we establish the sharpness of our rates by providing matching lower bounds and validate our findings through simulations. We obtain the sharp rates by first studying the convergence rate of the discrete Ornstein-Uhlenbeck (O-U) process driven by general noise, whose stationary distribution is identical to the limiting Gaussian distribution of the rescaled SA iterates. We believe that this is of independent interest, given its connection to sampling literature. The analysis involves adapting Stein's method for Gaussian approximation to handle the matrix weighted sum of i.i.d. random variables. The desired finite-time bounds for SA are obtained by characterizing the error dynamics between the rescaled SA iterate and the discrete time O-U process and combining it with the convergence rate of the latter process.

stat.ML↗

arXiv2026

Non-Asymptotic Convergence of Stochastic Iterative Algorithms: A Lyapunov Framework

We survey Lyapunov-based techniques for the finite-time analysis of stochastic iterative algorithms, also known as stochastic approximation (SA) algorithms, for solving fixed-point equations $\bar{F}(x)=x$, where the operator $\bar{F}(\cdot)$ can only be accessed through a noisy oracle. We first focus on the standard setting in which $\bar{F}(\cdot)$ is contractive with respect to some norm and the noise is i.i.d., and explain how generalized Moreau envelopes serve as universal Lyapunov functions, regardless of the underlying norm. We then show how this framework yields mean-square convergence guarantees and applies to stochastic gradient descent, linear SA, and value-based reinforcement learning algorithms such as Q-learning and temporal-difference learning. Finally, we discuss extensions to Markovian noise, seminorm-contractive operators, dissipative operators, and high-probability bounds, and conclude with open problems. The goal is to present a unified and self-contained roadmap for the finite-time analysis of SA and its applications, especially in reinforcement learning.

cs.LG↗

arXiv2026

Concentration of General Stochastic Approximation Under Heavy-Tailed Markovian Noise

We establish maximal concentration bounds for the iterates generated by stochastic approximation algorithms with general step sizes, where the noise has a finite-state Markovian component plus a Martingale-difference component. When the Martingale-difference noise is bounded, we show that the tail of the error can be sub-Gaussian, sub-Weibull, or something lighter than any Pareto but heavier than any Weibull, depending on the step size sequence and on whether the random operator is almost surely contractive, almost surely non-expansive, or expansive with positive probability. Our analysis relies on a novel Lyapunov function involving the moment-generating function of the solution to a Poisson equation, together with an auxiliary projected algorithm. We complement the upper bounds with worst-case examples showing that qualitatively sharper bounds are impossible. We further study the case of unbounded Martingale-difference noise when the average operator is contractive, and the step sizes are of order $1/k$. In this setting, we show that if the random operator is almost surely non-expansive, then the error tail is at most three times heavier than the noise tail, whereas if the random operator is expansive with positive probability, then the error may have substantially heavier tails. These results are obtained through a novel black-box truncation argument that reduces the unbounded-noise setting to the bounded-noise case.

math.PR↗

arXiv2026

Tail Bounds for Queues with Abandonment: Constant, Moderate, Large Deviations, and Efficient Concentration

We study a heavily overloaded single-server queue with abandonment and derive bounds on stationary tail probabilities of the queue length. As the abandonment rate $γ\downarrow 0$, the centered-scaled queue length $\tilde{q}$ is known to converge in distribution to a Gaussian. However, such asymptotic limits do not quantify the pre-limit tail $\mathbb{P}(\tilde{q}>a)$ for fixed $γ>0$. Our goal is to obtain pre-limit bounds that are \emph{efficient} across different deviation regimes. For constant deviations, efficiency means Gaussian-type decay in $a$ together with a pre-limit error that vanishes as $γ\downarrow 0$, yielding the correct Gaussian tail in the limit. We establish such an efficient bound that is best-of-both-worlds. For larger deviations when $a$ is a function of $γ$, efficiency translates into exponentially tight, matching upper and lower bounds. For moderate deviation, we obtain sub-Gaussian tails, while in the large deviation regime the decay becomes sub-Poisson. Our bounds are obtained using a combination of Stein's method for Wasserstein-$p$ distance and the transform method. We then consider a load-balancing system of abandonment queues with heterogeneous servers, operating under the join-the-shortest-queue (JSQ) policy in the heavily overloaded regime. As in the case of single-server queue, we again obtain Wasserstein-$p$ bounds w.r.t.\ a Gaussian, and efficient concentration for constant and moderate deviations. For larger deviations, our JSQ upper bounds exhibit a transition from Gaussian-type decay to sub-Weibull decay. All these results are obtained using Stein's method. In addition, a key ingredient here is establishing a state space collapse (SSC) where all queues become equal. We establish a $p$-th moment bound on the orthogonal component of the queue length vector that is essential for our Wasserstein-$p$ bound.

math.PR↗

arXiv2026

Almost Sure Convergence of Stochastic Approximation: An Interplay of Noise and Step Size

We study the almost sure convergence of the Stochastic Approximation algorithm to the fixed point $x^\star$ of a nonlinear operator under a negative drift condition and a general noise sequence with finite $p$-th moment for some $p > 1$. Classical almost sure convergence results of Stochastic Approximation are mostly analyzed for the square-integrable noise setting, and it is shown that any non-summable but square-summable step size sequence is sufficient to obtain almost sure convergence. However, such a limitation prevents wider algorithmic application. In particular, many applications in Machine Learning and Operations Research admit heavy-tailed noise with infinite variance, rendering such guarantees inapplicable. On the other hand, when a stronger condition on the noise is available, such guarantees on the step size would be too conservative, as practitioners would like to pick a larger step size for a more preferable convergence behavior. To this end, we show that any non-summable but $p$-th power summable step size sequence is sufficient to guarantee almost sure convergence, covering the gap in the literature. Our guarantees are obtained using a universal Lyapunov drift argument. For the regime $p \in (1, 2)$, we show that using the Lyapunov function $\norm{x-x^\star}^p$ and applying a Taylor-like bound suffice. For $p > 2$, such an approach is no longer applicable, and therefore, we introduce a novel iterate projection technique to control the nonlinear terms produced by high-moment bounds and multiplicative noise. We believe our proof techniques and their implications could be of independent interest and pave the way for finite-time analysis of Stochastic Approximation under a general noise condition.

math.OC↗

arXiv2026

Higher-Order Approximations of Sojourn Times in M/G/1 Queues via Stein's Method

We study the stationary sojourn time distribution in an M/G/1 queue operating under heavy traffic. It is known that the sojourn time converges to an exponential distribution in the limit. Our focus is on obtaining pre-asymptotic, higher-order approximations that go beyond the classical exponential limit. Using Stein's method, we develop an approach based on higher-order expansions of the generator of the underlying Markov process. The key technical step is to represent higher-order derivatives in terms of lower-order ones and control the resulting error via derivative bounds of the Stein equation. Under suitable moment-matching conditions on the service distribution, we show that the approximation error decays as a high-order power of the slack parameter $\varepsilon=1-ρ$. Error bounds are established in the Zolotarev metric, which further imply bounds on the Wasserstein distance as well as the moments. Our results demonstrate that the accuracy of the exponential approximation can be systematically improved by matching progressively more moments of the service distribution.

math.PR↗

arXiv2025

Stochastic Approximation with Unbounded Markovian Noise: A General-Purpose Theorem

Motivated by engineering applications such as resource allocation in networks and inventory systems, we consider average-reward Reinforcement Learning with unbounded state space and reward function. Recent works studied this problem in the actor-critic framework and established finite sample bounds assuming access to a critic with certain error guarantees. We complement their work by studying Temporal Difference (TD) learning with linear function approximation and establishing finite-time bounds with the optimal $\mathcal{O}\left(1/ε^2\right)$ sample complexity. These results are obtained using the following general-purpose theorem for non-linear Stochastic Approximation (SA). Suppose that one constructs a Lyapunov function for a non-linear SA with certain drift condition. Then, our theorem establishes finite-time bounds when this SA is driven by unbounded Markovian noise under suitable conditions. It serves as a black box tool to generalize sample guarantees on SA from i.i.d. or martingale difference case to potentially unbounded Markovian noise. The generality and the mild assumption of the setup enables broad applicability of our theorem. We illustrate its power by studying two more systems: (i) We improve upon the finite-time bounds of $Q$-learning by tightening the error bounds and also allowing for a larger class of behavior policies. (ii) We establish the first ever finite-time bounds for distributed stochastic optimization of high-dimensional smooth strongly convex function using cyclic block coordinate descent.

cs.LG↗

arXiv2025

Electric Vehicle Fleet and Charging Infrastructure Planning

We study electric vehicle (EV) fleet and charging infrastructure planning in a spatial setting. With customer requests arriving continuously at rate $λ$ throughout the day, we determine the minimum number of vehicles and chargers for a target service level, along with matching and charging policies. While non-EV systems require extra $Θ(λ^{2/3})$ vehicles due to pickup times, EV systems differ. Charging increases nominal capacity, enabling pickup time reductions and allowing for an extra fleet requirement of only $Θ(λ^ν)$ for $ν\in (1/2, 2/3]$, depending on charging infrastructure and battery pack sizes. We propose the Power-of-$d$ dispatching policy, which achieves this performance by selecting the closest vehicle with the highest battery level from $d$ options. We extend our results to accommodate time-varying demand patterns and discuss conditions for transitioning between EV and non-EV capacity planning. Extensive simulations verify our scaling results, insights, and policy effectiveness while also showing the viability of low-range, low-cost fleets.

math.OC↗

arXiv2025

Stochastic Approximation for Nonlinear Discrete Stochastic Control: Finite-Sample Bounds

We consider a nonlinear discrete stochastic control system, and our goal is to design a feedback control policy in order to lead the system to a prespecified state. We adopt a stochastic approximation viewpoint of this problem. It is known that by solving the corresponding continuous-time deterministic system, and using the resulting feedback control policy, one ensures almost sure convergence to the prespecified state in the discrete system. In this paper, we adopt such a control mechanism and provide its finite-sample convergence bounds whenever a Lyapunov function is known for the continuous system. In particular, we consider four cases based on whether the Lyapunov function for the continuous system gives exponential or sub-exponential rates and based on whether it is smooth or not. We provide the finite-time bounds in all cases. Our proof relies on constructing a Lyapunov function for the discrete system based on the given Lyapunov function for the continuous system. We do this by appropriately smoothing the given function using the Moreau envelope. We present numerical experiments corresponding to the various cases, which validate the rates we establish.

math.OC↗

arXiv2025

Convergence of Natural Policy Gradient for a Family of Infinite-State Queueing MDPs

A wide variety of queueing systems can be naturally modeled as infinite-state Markov Decision Processes (MDPs). In the reinforcement learning (RL) context, a variety of algorithms have been developed to learn and optimize these MDPs. At the heart of many popular policy-gradient based learning algorithms, such as natural actor-critic, TRPO, and PPO, lies the Natural Policy Gradient (NPG) policy optimization algorithm. Convergence results for these RL algorithms rest on convergence results for the NPG algorithm. However, all existing results on the convergence of the NPG algorithm are limited to finite-state settings. We study a general class of queueing MDPs, and prove a $O(1/\sqrt{T})$ convergence rate for the NPG algorithm, if the NPG algorithm is initialized with the MaxWeight policy. This is the first convergence rate bound for the NPG algorithm for a general class of infinite-state average-reward MDPs. Moreover, our result applies to a beyond the queueing setting to any countably-infinite MDP satisfying certain mild structural assumptions, given a sufficiently good initial policy. Key to our result are state-dependent bounds on the relative value function achieved by the iterate policies of the NPG algorithm.

cs.LG↗

arXiv2025

Tight Finite Time Bounds of Two-Time-Scale Linear Stochastic Approximation with Markovian Noise

Stochastic approximation (SA) is an iterative algorithm for finding the fixed point of an operator using noisy samples and widely used in optimization and Reinforcement Learning (RL). The noise in RL exhibits a Markovian structure, and in some cases, such as gradient temporal difference (GTD) methods, SA is employed in a two-time-scale framework. This combination introduces significant theoretical challenges for analysis. We derive an upper bound on the error for the iterations of linear two-time-scale SA with Markovian noise. We demonstrate that the mean squared error decreases as $trace (Σ^y)/k + o(1/k)$ where $k$ is the number of iterates, and $Σ^y$ is an appropriately defined covariance matrix. A key feature of our bounds is that the leading term, $Σ^y$, exactly matches with the covariance in the Central Limit Theorem (CLT) for the two-time-scale SA, and we call them tight finite-time bounds. We illustrate their use in RL by establishing sample complexity for off-policy algorithms, TDC, GTD, and GTD2. A special case of linear two-time-scale SA that is extensively studied is linear SA with Polyak-Ruppert averaging. We present tight finite time bounds corresponding to the covariance matrix of the CLT. Such bounds can be used to study TD-learning with Polyak-Ruppert averaging.

cs.LG↗

arXiv2025

Power-of-$d$ Choices Load Balancing in the Sub-Halfin Whitt Regime

We consider the load balancing system under Poisson arrivals, exponential services, and homogeneous servers. Upon arrival, a job is to be routed to one of the servers, where it is queued until service. We consider the Power-of-$d$ choices routing algorithm, which chooses the queue with minimum length among $d$ randomly sampled queues. We study this system in the many-server heavy-traffic regime where the number of servers goes to infinity simultaneously when the load approaches the capacity. In particular, we consider a sequence of systems with $n$ servers and the arrival rate is given by $λ=n-n^{1-γ}$ for some $γ\in (0, 0.5)$, known as the sub-Halfin-Whitt regime. It was shown by [Liu Ying (2020)] that under Power-of-$d$ choices routing with $d \geq n^γ\log n$, the queue length behaves similarly to that of JSQ and that there are asymptotically zero queueing delays. The focus of this paper is to characterize the behavior when $d$ is below this threshold. We obtain high probability bounds on the queue lengths for various values of $d$ and large enough $n$. In particular, we show that when $d$ grows polynomially in $n$ but slower than in [Liu Ying (2020)], i.e., if $d$ is $Θ\left((n^γ\log n)^{1/m})\right)$ for some integer $m>1$, then the asymptotic queue length is $m$ with high probability. Moreover, if $d$ grows polylog in $n$, i.e., slower than any polynomial, but is at least $Ω(\log (n)^3)$, the queue length blows up to infinity asymptotically. We obtain these results by using an iterative state space collapse approach. We first establish a weak state-space collapse (SSC) on the queue lengths. Then, we bootstrap on weak SSC to iteratively narrow down the region of the collapse. After enough steps, this inductive refinement provides the bounds we seek. We establish these sequences of collapse using Lyapunov drift arguments.

math.PR↗