Universal Reinforcement Learning

Vivek F. Farias; Ciamac C. Moallemi; Tsachy Weissman; Benjamin Van Roy

Home/Archive/arXiv

arXiv · 0707.3087

Universal Reinforcement Learning

Vivek F. Farias·Ciamac C. Moallemi·Tsachy Weissman·Benjamin Van Roy

View original source ↗Read source document ↗Export metadata ↓

Abstract

We consider an agent interacting with an unmodeled environment. At each time, the agent makes an observation, takes an action, and incurs a cost. Its actions can influence future observations and costs. The goal is to minimize the long-term average cost. We propose a novel algorithm, known as the active LZ algorithm, for optimal control based on ideas from the Lempel-Ziv scheme for universal data compression and prediction. We establish that, under the active LZ algorithm, if there exists an integer $K$ such that the future is conditionally independent of the past given a window of $K$ consecutive actions and observations, then the average cost converges to the optimum. Experimental results involving the game of Rock-Paper-Scissors illustrate merits of the algorithm.

Explore related subjects

cs.IT cs.LG math.IT

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Vivek F. Farias, Ciamac C. Moallemi, Tsachy Weissman, Benjamin Van Roy. 2009-07-22. Universal Reinforcement Learning. https://arxiv.org/abs/0707.3087

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Gaussian Mixture Models (GMMs) are widely used statistical models for representing multi-modal data distributions, with numerous applications in data mining, pattern recognition, data simulation, and machine learning. However, recent research has shown that releasing GMM parameters poses significant privacy risks, potentially exposing sensitive information about the underlying data. In this paper, we address the challenge of releasing GMM parameters while ensuring differential privacy (DP) guarantees. Specifically, we focus on the privacy protection of mixture weights, component means, and covariance matrices. We propose to use Kullback-Leibler (KL) divergence as a utility metric to assess the accuracy of the released GMM, as it captures the joint impact of noise perturbation on all the model parameters. We develop a KL-divergence-driven optimization framework for differentially private GMM release that jointly designs privacy-budget allocation and parameter perturbations. Through theoretical analysis, we quantify the effects of privacy budget allocation and perturbation statistics on the DP guarantee, and derive a tractable expression for evaluating KL divergence. We formulate and solve an optimization problem to minimize the KL divergence between the released and fitted non-private models, subject to a given $(ε, δ)$-DP constraint. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach achieves strong privacy guarantees while maintaining high utility.

cs.IT↗

arXiv2026

Direction-Shift Keying for High-Mobility mmWave Communications: Geometry-Based Resilience to Short Channel Coherence Times and Phase Noise

Communication reliability in high-mobility millimeter-wave (mmWave) systems is fundamentally challenged by rapid channel variations, manifested in short channel coherence times, and phase noise, which necessitate frequent channel estimation and phase tracking. To overcome these limitations, we introduce Direction-Shift Keying (DSK), a geometry-based modulation scheme that exploits the relative stability of propagation directions rather than the rapidly varying amplitude and phase of the wireless channel. DSK conveys information through distinguishable directions of arrival (DoAs), enabled by a distributed antenna system (DAS), thereby shifting the communication principle from channel-dependent to geometry-dependent signaling. We first derive the structure of the optimal DSK detector for a mobile device equipped with an arbitrary number of antennas. We then establish the fundamental law governing the temporal stability of propagation directions, termed Direction Coherence Time (DCT), and show that DCT scales as $d/v$, where $d$ is the transmitter--receiver distance and $v$ is the relative velocity. In contrast, conventional Channel Coherence Time (CCT) scales as $λ/v$, where $λ$ denotes the wavelength, revealing a coherence-time advantage on the order of $d/λ$, potentially exceeding four orders of magnitude in mmWave systems. Furthermore, we establish the inherent invariance of DSK to common phase rotations, providing resilience to common oscillator phase noise without explicit phase tracking. Numerical results corroborate the theoretical findings and demonstrate the potential of geometry-based signaling for reliable high-mobility mmWave communications.

cs.IT↗

arXiv2026

Coverage Region Analysis and Optimization of FIRES-Assisted NOMA and OMA Systems

Fluid integrated reflecting and emitting surfaces (FIRES) are investigated. In these metasurfaces, each subarea hosts an active element capable of simultaneous transmission and reflection, phase, and geometric positioning control within the subarea. We develop a coverage-centric system model for the two-user downlink scenario (one user per half-space) under spatially correlated Rician fading and imperfect phase control. First, we derive closed-form far-field line-of-sight (LoS) coverage bounds that reveal the effects of aperture size, base station (BS) distance, transmit power, energy-splitting (ES), and phase errors. Protocol-aware corollaries are then presented for both orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA), including conditions for successful successive interference cancellation (SIC). Second, we formulate coverage maximization as a bi-level optimization problem consisting of (i) an outer search over FIRES element positions, selecting one active preset per subarea under minimum-spacing constraints, and (ii) an inner resource allocation problem tailored to the multiple-access scheme, which is one-dimensional for OMA and a small convex program for NOMA. The proposed framework explicitly accounts for target rate constraints, ES conservation, power budgets, geometric placement limits, and decoding-order feasibility. Extensive simulations demonstrate that FIRES, by jointly exploiting geometric repositioning and passive energy control, substantially enlarges the coverage region compared with a conventional simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) under the same element budget. Furthermore, NOMA yields additional coverage gains when feasible. The analytical coverage bounds closely match the simulation results and quantify the robustness of FIRES to phase-control imperfections.

cs.IT↗