Near-Optimal Reinforcement Learning for Constrained Recurrence Objectives

Abstract

Recurrence objectives, where a target region must be visited infinitely often, are a fundamental class of specifications for Markov decision processes (MDPs) and form the core of $\omega$-regular and linear temporal logic (LTL) objectives. We study constrained recurrence objectives, a natural extension of recurrence objectives with probabilistic constraints capable of modelling safety or fairness requirements. We first study the structure of optimal policies, showing that constrained recurrence requires different policy classes than those sufficient for other constrained MDP formalisms. In particular, we show that every feasible instance admits an optimal mixture of two stochastic stationary policies, as well as an optimal mixture of two deterministic stationary policies over a one-bit augmented MDP. We then study the generative-model reinforcement learning setting and propose an algorithm that first identifies the maximal end-component decomposition of the MDP, then reduces constrained recurrence to a constrained average reward problem for a collapsed MDP. Moreover, we establish a $\tilde{\mathcal{O}}(1/p+B/\varepsilon^2)$ sample complexity guarantee per state-action pair, where $p$ bounds certain non-zero transition probabilities and $B$ bounds transient time. Finally, we prove a nearly matching lower bound, showing that $1/p$ dependence, unlike the average-reward setting, is unavoidable.