Deterministic policy gradient based optimal control with probabilistic constraints

Abstract

This paper studies a deep deterministic policy gradient (DDPG) based actor critic (AC) reinforcement learning (RL) technique to control a linear discrete-time system with a quadratic control cost while ensuring a constraint on the probability of potentially risky or undesirable events. The proposed methodology can be applied to both known and unknown system models with minor adjustments to the reward structure (negative cost). The problem is formulated by considering the average expected quadratic cost of the states and inputs over an infinite time horizon. Risky or undesirable events are represented as functions of the states at the next time step exceeding a user-defined limit. Two strategies are employed to manage the probabilistic constraint in scenarios of known and unknown system models. In the case of a known system model, the probabilistic constraint is replaced with an upper bound, such as the Chernoff bound. For unknown system models, the expected value of the indicator function of the occurrence of the risky or undesirable event is used. We have adopted a deterministic policy gradient (DPG) based AC method to derive a parameterised optimal policy. Extensive numerical simulations are performed using a second- and a fourth-order system, and the proposed method is compared with the standard risk-neutral linear quadratic regulator (LQR) and a chance-constrained model predictive control (MPC) method. The results demonstrate the effectiveness of the proposed approach in both known and unknown system model scenarios.