arXiv · 2611.11091
Uncertainty-Guided Reward Labeling for Reinforcement Learning under Limited Feedback
Abstract
In reinforcement learning from limited feedback (RLLF), only a small fraction of an offline dataset can be labeled with rewards, and the central question is which samples should be labeled to learn a high-performing policy from the resulting partially labeled dataset. Prior work formalized this as a reward-selection problem by focusing on the selection stage while treating downstream policy learning as a black box, in a regime where queried rewards are not retained for reward-model training. We instead study the retained-label setting, where queried rewards can be stored and used to fit a reward model before policy learning. We bound the suboptimality of the learned policy in terms of two sources of error: one arising from offline RL on a fixed, offline dataset, and the other from uncertainty in the reward model. Since reward selection cannot change the offline dataset, the limited labeling budget must be used to strategically reduce reward uncertainty. Motivated by RLLF's observation that useful rewards tend to keep the agent on high-return trajectories, we introduce Successor-Guided Uncertainty REduction (SURE), a novel method that uses successor features to select rewards that are both uncertain and associated with states from which high-valued states are reachable. We formally derive SURE from a bound-induced design objective and characterize its exact one-step marginal gain. Empirically, SURE reaches near full-feedback performance with few reward labels across a variety of domains and offline-RL backbones, yielding an effective method for feedback-efficient reinforcement learning.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Renhao Zhang, Shreyas Chaudhari, Bruno Castro da Silva. 2026-11-08. Uncertainty-Guided Reward Labeling for Reinforcement Learning under Limited Feedback. https://searcharxiv.com/abs/2611.11091
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.