arXiv · 2611.12643
Simulation-aided Reinforcement Learning with Control Variates
Abstract
Reinforcement Learning (RL) algorithms often rely on data collected from a target environment, which can be expensive and noisy. Although simulators can generate large amounts of additional data, standard simulation mixing approaches, which directly combine simulated and real experience, can degrade performance due to simulator inaccuracies, especially when they bias the solution away from the optimal policy. We propose a conservative simulation-aided learning method based on control variates, where simulated data is used solely to reduce variance in the learning process rather than directly shaping the learning target. We develop our approach for Policy-Evaluation, which is a fundamental building block of RL algorithms. Then we combine it within the actor-critic framework towards policy improvement, as well as extend it towards offline RL scenarios. Experiments on multiple MuJoCo benchmark tasks demonstrate improvements over standard simulation mixing approaches, highlighting control variates as an effective tool for simulation-aided reinforcement learning.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Omer Hemo, Ido Sefi, Mirco Mutti, Aviv Tamar, Kfir Y. Levy. 2026-11-09. Simulation-aided Reinforcement Learning with Control Variates. https://searcharxiv.com/abs/2611.12643
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.