Stability Analysis of an Integrated Multistage Stochastic Programming and Markov Decision Process Problem

Abstract

In this paper, we consider an integrated MSP-MDP framework which captures features of Markov decision process (MDP) and multistage stochastic programming (MSP). The integrated framework allows one to study a dynamic decision-making process that involves both transition of system states and dynamic change of the stochastic environment affected respectively by potential endogenous uncertainties and exogenous uncertainties. The integrated model differs from classical MDP models by taking into account the effect of history-dependent exogenous uncertainty and distinguishes itself from standard MSP models by explicitly considering transition of states between stages. We begin by deriving dynamic nested reformulation of the problem and the Lipschitz continuity and convexity of the stage-wise optimal value functions. We then move on to investigate stability of the problem in terms of the optimal value and the set of optimal solutions under the perturbations of the probability distributions of the endogenous uncertainty and the exogenous uncertainty. Specifically, we quantify the effects of the perturbation of the two uncertainties on the optimal values and optimal solutions by deriving the error bounds in terms of Kantorovich metric and Fortet-Mourier metric of the probability distributions of the respective uncertainties. These results differ from the existing stability results established in terms of the filtration distance \cite{heitsch2009scenario} or the nested distance \cite{pflug2012distance}. We use some examples to explain the differences via tightness of the error bounds and applicability of the stability results. The results complement the existing stability results and provide new theoretical grounding for emerging integrated MSP-MDP models.