Deep Reinforcement Learning with Smith-Predictor Delay Compensation

Abstract

Time delays can degrade feedback performance and destabilize a closed loop that is stable without delay. This paper studies Smith-predictor-assisted reinforcement learning for systems with constant input delays. A deep deterministic policy gradient (DDPG) actor is trained on a nominal delay-free environment and deployed with a Smith predictor using a fixed nominal delay. For linear controllers, we derive exact closed-loop maps under plant and delay mismatch and provide small-gain and $ν$-gap robustness conditions. A separate delay-independent Razumikhin condition gives a sufficient stability test for a fixed Lipschitz state-feedback policy. We also study output-error delay identification from a fixed input--output record, deriving the loss gradient and establishing projected-flow descent and local exponential convergence under exact model matching and an excitation condition. These identification results do not imply stability of an online adaptive predictor. Numerical experiments retained from the conference study compare PID, LQR, and DDPG controllers on stable and open-loop unstable first-order plants with valve stiction. The results illustrate the benefit of predictor compensation in the tested configurations and the sensitivity of standalone DDPG to delay and plant changes.