A Concentration Bound for LSPE($λ$)
The popular LSPE($λ$) algorithm for policy evaluation is revisited to derive a concentration bound that gives high probability performance guarantees from some time on.
cs.LG↗
arXiv subjects
Siddharth Chandak
Publications and source records attributed to Siddharth Chandak.
Concentration of Contractive Stochastic Approximation and Reinforcement Learning
Using a martingale concentration inequality, concentration bounds `from time $n_0$ on' are derived for stochastic approximation algorithms with contractive maps and both martingale difference and Markov noises. These are applied to reinforcement learning algorithms, in particular to asynchronous Q-learning and TD(0).
cs.LG↗
Prospect-theoretic Q-learning
We consider a prospect theoretic version of the classical Q-learning algorithm for discounted reward Markov decision processes, wherein the controller perceives a distorted and noisy future reward, modeled by a nonlinearity that accentuates gains and underrepresents losses relative to a reference point. We analyze the asymptotic behavior of the scheme by analyzing its limiting differential equation and using the theory of monotone dynamical systems to infer its asymptotic behavior. Specifically, we show convergence to equilibria, and establish some qualitative facts about the equilibria themselves.
eess.SY↗
Hidden Markov Model-Based Encoding for Time-Correlated IoT Sources
As the use of Internet of Things (IoT) devices for monitoring purposes becomes ubiquitous, the efficiency of sensor communication is a major issue for the modern Internet. Channel coding is less efficient for extremely short packets, and traditional techniques that rely on source compression require extensive signaling or pre-existing knowledge of the source dynamics. In this work, we propose an encoding and decoding scheme that learns source dynamics online using a Hidden Markov Model (HMM), puncturing a short packet code to outperform existing compression-based approaches. Our approach shows significant performance improvements for sources that are highly correlated in time, with no additional complexity on the sender side.
cs.NI↗