A Bayesian Composite Risk Approach for Stochastic Optimal Control and Markov Decision Processes

Abstract

Inspired by Shapiro et al.~\cite{shapiro2023episodic}, we consider a stochastic optimal control (SOC) and Markov decision process (MDP) where the risks arising from epistemic and aleatoric uncertainties are assessed using Bayesian composite risk (BCR) measures (Qian et al.~\cite{qian2019composite}). The time dependence of the risk measures allows us to capture the decision maker's (DM) dynamic risk preferences opportunely as increasing information about both uncertainties is obtained. This makes the new BCR-SOC/MDP model more flexible than conventional risk-averse SOC/MDP models. Unlike \cite{shapiro2023episodic} where the control/action at each episode is based on the current state alone, the new model allows the control to depend on the probability distribution of the epistemic uncertainty, which reflects the fact that in many practical instances the cumulative information about epistemic uncertainty often affects the DM's belief about the future aleatoric uncertainty and hence the DM's action \cite{strens2000bayesian}. The new modeling paradigm incorporates several existing SOC/MDP models including distributionally robust SOC/MDP models and Bayes-adaptive MDP models and generates so-called preference robust SOC/MDP models. Moreover, we derive conditions under which the BCR-SOC/MDP model is well-defined, demonstrate that finite-horizon BCR-SOC/MDP models can be solved using dynamic programming techniques, and extend the discussion to the infinite-horizon case. By using Bellman equations, we show that under some standard conditions, asymptotic convergence of the optimal values and optimal actions as the episodic variable goes to infinity is achieved. Finally, we carry out numerical tests on a finite horizon spread betting problem and an inventory control problem and show the effectiveness of the proposed model and numerical schemes.