Robust Markov decision processes under parametric transition distributions

Abstract

This paper considers robust Markov decision processes under parametric transition distributions. We assume that the true transition distribution is uniquely specified by some parametric distribution, and explicitly enforce that the worst-case distribution from the model is uniquely specified by a distribution in the same parametric family. After formulating the parametric robust model, we focus on developing algorithms for carrying out the robust Bellman updates required to complete robust value iteration. We first formulate the update as a linear program by discretising the ambiguity set. Since this model scales poorly with problem size and requires large amounts of pre-computation, we develop two additional algorithms for solving the robust Bellman update. Firstly, we present a cutting surface algorithm for solving this linear program in a shorter time. This algorithm requires the same pre-computation, but only ever solves the linear program over small subsets of the ambiguity set. Secondly, we present a novel projection-based bisection search algorithm that completely eliminates the need for discretisation and does not require any pre-computation. We test our algorithms extensively on a dynamic multi-period newsvendor problem under binomial and Poisson demands. In addition, we compare our methods with the non-parametric phi-divergence based methods from the literature. We show that our projection-based algorithm completes robust value iteration significantly faster than our other two parametric algorithms, and also faster than its non-parametric equivalent.