A hierarchical resource-efficient deep policy gradient method for continuous-time optimal control problems
In this paper, we propose an efficient implementation of a deep policy gradient method (PGM) for optimal control problems in continuous time. For continuous-time problems that require a fine time discretization to achieve a desired accuracy, the proposed method improves time efficiency and performance by strategically allocating computational resources across scales, i.e., the number of trajectories, the granularity of time discretization, and the complexity of the neural network architecture. The main idea of the paper is to avoid committing to a fine time discretization. At first, we train a policy, modeled by a neural network, for a discretized optimal control problem in a coarse time scale. Then, we only discretize more in the time intervals where there are indications that the coarse scheme is not accurate enough and train a new policy in the finer scale. We then proceed to refine the time grid further to achieve better accuracy in the new smaller time intervals. Our theoretical result indicates how the new schedule for allocation of resources in different time scales can lead to efficiency. We conclude the paper by numerical experiments on a linear-quadratic stochastic optimal control problem and an optimal execution problem from quantitative finance.