arXiv subjects
Publications and source records attributed to Sameer Kamal.
The problem of controlling a finite state Markov chain in the presence of an adversary so as to ensure desired performance levels for a vector of objectives is cast in the framework of Blackwell approachability. Relying on an elementary two time scale construction a control scheme is proposed which ensures almost sure convergence to the desired set regardless of the adversarial actions.
It is shown that under standard hypotheses, if stochastic approximation iterates remain tight, they converge with probability one to what their o.d.e. limit suggests. A simple test for tightness (and therefore a.s. convergence) is provided. Further, estimates on lock-in probability, i.e., the probability of convergence to a specific attractor of the o.d.e. limit given that the iterates visit its domain of attraction, and sample complexity, i.e., the number of steps needed to be within a prescribed neighborhood of the desired limit set with a prescribed probability, are also provided. The latter improve significantly upon existing results in that they require a much weaker condition on the martingale difference noise.
A scheme for stabilizing stochastic approximation iterates by adaptively scaling the step sizes is proposed and analyzed. This scheme leads to the same limiting differential equation as the original scheme and therefore has the same limiting behavior, while avoiding the difficulties associated with projection schemes. The proof technique requires only that the limiting o.d.e. descend a certain Lyapunov function outside an arbitrarily large bounded set.