arXiv subjects
Publications and source records attributed to Vivek S. Borkar.
Using a martingale concentration inequality, concentration bounds `from time $n_0$ on' are derived for stochastic approximation algorithms with contractive maps and both martingale difference and Markov noises. These are applied to reinforcement learning algorithms, in particular to asynchronous Q-learning and TD(0).
We consider the problem of scheduling packet transmissions in a wireless network of users while minimizing the energy consumed and the transmission delay. A challenge is that transmissions of users that are close to each other mutually interfere, while users that are far apart can transmit simultaneously without much interference. Each user has a queue of packets that are transmitted on a single channel and mutually non interfering users reuse the spectrum. Using the theory of Whittle index for cost minimizing restless bandits, we design four index-based policies and compare their performance with that of the well-known policies: Slotted ALOHA, maximum weight scheduling, quadratic Lyapunov drift, Cella and Cesa Bianchi algorithm, and two Whittle index based policies from a recently published paper. We make the code used to perform our simulations publicly available, so that it can be used for future work by the research community at large.
In this work, we show that for the martingale problem for a class of degenerate diffusions with bounded continuous drift and diffusion coefficients, the small noise limit of non-degenerate approximations leads to a unique Feller limit. The proof uses the theory of viscosity solutions applied to the associated backward Kolmogorov equations. Under appropriate conditions on drift and diffusion coefficients, we will establish a comparison principle and a one-one correspondence between Feller solutions to the martingale problem and continuous viscosity solutions of the associated Kolmogorov equation. This work can be considered as an extension to the work of V. S. Borkar and K. S. Kumar (2010).
A novel reinforcement learning algorithm is introduced for multiarmed restless bandits with average reward, using the paradigms of Q-learning and Whittle index. Specifically, we leverage the structure of the Whittle index policy to reduce the search space of Q-learning, resulting in major computational gains. Rigorous convergence analysis is provided, supported by numerical experiments. The numerical experiments show excellent empirical performance of the proposed scheme.
We consider a prospect theoretic version of the classical Q-learning algorithm for discounted reward Markov decision processes, wherein the controller perceives a distorted and noisy future reward, modeled by a nonlinearity that accentuates gains and underrepresents losses relative to a reference point. We analyze the asymptotic behavior of the scheme by analyzing its limiting differential equation and using the theory of monotone dynamical systems to infer its asymptotic behavior. Specifically, we show convergence to equilibria, and establish some qualitative facts about the equilibria themselves.
We introduce a model of graph-constrained dynamic choice with reinforcement modeled by positively $α$-homogeneous rewards. We show that its empirical process, which can be written as a stochastic approximation recursion with Markov noise, has the same probability law as a certain vertex reinforced random walk. We use this equivalence to show that for $α> 0$, the asymptotic outcome concentrates around the optimum in a certain limiting sense when `annealed' by letting $α\uparrow\infty$ slowly.
We derive equivalent linear and dynamic programs for infinite horizon risk-sensitive control for minimization of the asymptotic growth rate of the cumulative cost.
The non-population conserving SIR (SIR-NC) model to describe the spread of infections in a community is proposed and studied. Unlike the standard SIR model, SIR-NC does not assume population conservation. Although similar in form to the standard SIR, SIR-NC admits a closed form solution while allowing us to model mortality, and also provides different, and arguably a more realistic, interpretation of the model parameters. Numerical comparisons of this SIR-NC model with the standard, population conserving, SIR model are provided. Extensions to include imported infections, interacting communities, and models that include births and deaths are presented and analyzed. Several numerical examples are also presented to illustrate these models. Two control problems for the SIR-NC epidemic model are presented. First we consider the continuous time model predictive control in which the cost function variables correspond to the levels of lockdown, the level of testing and quarantine, and the number of infections. We also include a switching cost for moving between lockdown levels. A discrete time version that is more amenable to computation is then presented along with numerical illustrations. We then consider a multi-objective and multi-community control where we can define multiple cost functions on the different communities and obtain the minimum cost control to keep the value function corresponding to these control objectives below a prescribed threshold.
We address the variational formulation of the risk-sensitive reward problem for non-degenerate diffusions on $\mathbb{R}^d$ controlled through the drift. We establish a variational formula on the whole space and also show that the risk-sensitive value equals the generalized principal eigenvalue of the semilinear operator. This can be viewed as a controlled version of the variational formulas for principal eigenvalues of diffusion operators arising in large deviations. We also revisit the average risk-sensitive minimization problem and by employing a gradient estimate developed in this paper, we extend earlier results to unbounded drifts and running costs.
We study the problem of scheduling packet transmissions with the aim of minimizing the energy consumption and data transmission delay of users in a wireless network in which spatial reuse of spectrum is employed. We approach this problem using the theory of Whittle index for cost minimizing restless bandits, which has been used to effectively solve problems in a variety of applications. We design two Whittle index based policies the first by treating the graph representing the network as a clique and the second based on interference constraints derived from the original graph. We evaluate the performance of these two policies via extensive simulations, in terms of average cost and packets dropped, and show that they outperform the well-known Slotted ALOHA and maximum weight scheduling algorithms.
The main result in this paper is a variational formula for the exit rate from a bounded domain for a diffusion process in terms of the stationary law of the diffusion constrained to remain in this domain forever. Related results on the geometric ergodicity of the controlled Q-process are also presented.
A multiplicative relative value iteration algorithm for solving the dynamic programming equation for the risk-sensitive control problem is studied for discrete time controlled Markov chains with a compact Polish state space, and controlled diffusions in on the whole Euclidean space. The main result is a proof of convergence to the desired limit in each case.
We formulate and study the infinite dimensional linear programming (LP) problem associated with the deterministic discrete time long-run average criterion optimal control problem. Along with its dual, this LP problem allows one to characterize the optimal value of the optimal control problem. The novelty of our approach is that we focus on the general case wherein the optimal value may depend on the initial condition of the system.
Given an ODE and its perturbation, the Alekseev formula expresses the solutions of the latter in terms related to the former. By exploiting this formula and a new concentration inequality for martingale-differences, we develop a novel approach for analyzing nonlinear Stochastic Approximation (SA). This approach is useful for studying a SA's behaviour close to a Locally Asymptotically Stable Equilibrium (LASE) of its limiting ODE; this LASE need not be the limiting ODE's only attractor. As an application, we obtain a new concentration bound for nonlinear SA. That is, given $ε>0$ and that the current iterate is in a neighbourhood of a LASE, we provide an estimate for i.) the time required to hit the $ε-$ball of this LASE, and ii.) the probability that after this time the iterates are indeed within this $ε-$ball and stay there thereafter. The latter estimate can also be viewed as the `lock-in' probability. Compared to related results, our concentration bound is tighter and holds under significantly weaker assumptions. In particular, our bound applies even when the stepsizes are not square-summable. Despite the weaker hypothesis, we show that the celebrated Kushner-Clark lemma continues to hold. %
This is an overview of the work of the authors and their collaborators on the characterization of risk sensitive costs and rewards in terms of an abstract Collatz-Wielandt formula and in case of rewards, also a controlled version of the Donsker-Varadhan formula. For the finite state and action case, this leads to useful linear and dynamic programming formulations in the reducible case.
We propose a dynamic formulation of file-sharing networks in terms of an average cost Markov decision process with constraints. By analyzing a Whittle-like relaxation thereof, we propose an index policy in the spirit of Whittle and compare it by simulations with other natural heuristics.
We propose a new simple and natural algorithm for learning the optimal Q-value function of a discounted-cost Markov Decision Process (MDP) when the transition kernels are unknown. Unlike the classical learning algorithms for MDPs, such as Q-learning and actor-critic algorithms, this algorithm doesn't depend on a stochastic approximation-based method. We show that our algorithm, which we call the empirical Q-value iteration (EQVI) algorithm, converges to the optimal Q-value function. We also give a rate of convergence or a non-asymptotic sample complexity bound, and also show that an asynchronous (or online) version of the algorithm will also work. Preliminary experimental results suggest a faster rate of convergence to a ball park estimate for our algorithm compared to stochastic approximation-based algorithms.
We consider a simultaneous small noise limit for a singularly perturbed coupled diffusion described by \begin{eqnarray*} dX^{\varepsilon}_t &=& b(X^{\varepsilon}_t, Y^{\varepsilon}_t)dt + \varepsilon^αdB_t, dY^{\varepsilon}_t &=& - \frac{1}{\varepsilon} \nabla_yU(X^{\varepsilon}_t, Y^{\varepsilon}_t)dt + \frac{s(\varepsilon)}{\sqrt{\varepsilon}} dW_t, \end{eqnarray*} where $B_t, W_t$ are independent Brownian motions on ${\mathbb R}^d$ and ${\mathbb R}^m$ respectively, $b : \mathbb{R}^d \times \mathbb{R}^m \rightarrow \mathbb{R}^d$, $U : \mathbb{R}^d \times \mathbb{R}^m \rightarrow \mathbb{R}$ and $s :(0,\infty) \rightarrow (0,\infty)$. We impose regularity assumptions on $b$, $U$ and let $0 < α< 1.$ When $s(\varepsilon)$ goes to zero slower than a prescribed rate as $\varepsilon \rightarrow 0$, we characterize all weak limit points of $X^{\varepsilon}$, as $\varepsilon \rightarrow 0$, as solutions to a differential equation driven by a measurable vector field. Under an additional assumption on the behaviour of $U(x, \cdot)$ at its global minima we characterize all limit points as Filippov solutions to the differential equation.