Reinforcement Learning versus PDE Backstepping and PI Control for Congested Freeway Traffic

Abstract

We develop reinforcement learning (RL) boundary controllers to mitigate stop-and-go traffic congestion on a freeway segment. The traffic dynamics of the freeway segment are governed by a macroscopic Aw-Rascle-Zhang (ARZ) model, consisting of $2\times 2$ quasi-linear partial differential equations (PDEs) for traffic density and velocity. Boundary stabilization of the linearized ARZ PDE model has been solved by PDE backstepping, guaranteeing spatial $L^2$ norm regulation of the traffic state to uniform density and velocity and ensuring that traffic oscillations are suppressed. Collocated Proportional (P) and Proportional-Integral (PI) controllers also provide stability guarantees under certain restricted conditions, and are always applicable as model-free control options through gain tuning by trail and error, or by model-free optimization. Although these approaches are mathematically elegant, the stabilization result only holds locally and is usually affected by the change of model parameters. Therefore, we reformulate the PDE boundary control problem as a RL problem that pursues stabilization without knowing the system dynamics, simply by observing the state values. The proximal policy optimization, a neural network-based policy gradient algorithm, is employed to obtain RL controllers by interacting with a numerical simulator of the ARZ PDE. Being stabilization-inspired, the RL state-feedback boundary controllers are compared and evaluated against the rigorously stabilizing controllers in two cases: (i) in a system with perfect knowledge of the traffic flow dynamics, and then (ii) in one with only partial knowledge. We obtain RL controllers that nearly recover the performance of the backstepping, P, and PI controllers with perfect knowledge and outperform them in some cases with partial knowledge.