Reinforcement learning in pursuit-evasion differential game: safety, stability and robustness

Abstract

Safety and stability are two critical concerns in pursuit-evasion (PE) problems in an obstacle-rich environment. Most existing works combine control barrier functions (CBFs) and reinforcement learning (RL) to provide an efficient and safe solution. However, they do not consider the presence of disturbances, such as wind gust and actuator fault, which may exist in many practical applications. This paper integrates CBFs and a sliding mode control (SMC) term into RL to simultaneously address safety, stability, and robustness to disturbances. However, this integration is significantly challenging due to the strong coupling between the CBF and SMC terms. Inspired by Stackelberg game, we handle the coupling issue by proposing a hierarchical design scheme where SMC and safe control terms interact with each other in a leader-follower manner. Specifically, the CBF controller, acting as the leader, enforces safety independently of the SMC design; while the SMC term, as the follower, is designed based on the CBF controller. We then formulate the PE problem as a zero-sum game and propose a safe robust RL framework to learn the min-max strategy online. A sufficient condition is provided under which the proposed algorithm remains effective even when constraints are conflicting. Simulation results demonstrate the effectiveness of the proposed safe robust RL framework.