Tomas Prieto-Rumeau | Searcharxiv

arXiv2025

Absorbing Markov Decision Processes: Geometric Properties and Sufficiency of Finite Mixtures of Deterministic Policies

In this paper we investigate several geometric properties of the set of occupancy measures. In particular, we analyse the structure of the faces generated by a given occupancy measure, together with their relative algebraic interior. We also determine the affine hulls of these faces and describe the associated parallel linear subspaces. It is shown that these structures can be fully characterised in terms of the parameters that define the underlying Markov decision process (MDP). Moreover, we establish that the class of finite mixtures of deterministic stationary policies constitutes a sufficient class of policies for uniformly absorbing MDPs with a measurable state space and multiple criteria. We also provide a characterisation of the minimal order required for a finite mixture of deterministic stationary policies to represent the performance vector of an arbitrary policy.

math.OC↗

arXiv2025

On the Feinberg-Piunovskiy Theorem and its extension to chattering policies

The Feinberg-Piunovskiy Theorem established in [14, Theorem 3.8] asserts that for a discrete-time uniformly absorbing and atomless Markov Decision Process (MDP) with Borel state space and multiple criteria, the family of deterministic stationary policies is a sufficient class of policies. In this paper, we study some related problems and some extensions. In particular dropping the atomless hypothesis, we establish that the set of chattering stationary policies is a sufficient class of policies for uniformly absorbing MDPs with measurable state space and multiple criteria. We also prove the Feinberg-Piunovskiy Theorem in the context of a measurable state space in two different ways that differ from \cite{piunovskiy19}. In particular, we show that the sufficiency of chattering stationary policies directly yields the sufficiency of deterministics stationary policies for atomless models. Our approach is partially based on the analysis of extreme points of certain convex sets of occupation measures satisfying integral type constraints. We show that for a uniformly absorbing model an extreme point of such sets is necessarily given by occupation measures induced by chattering stationary policies of order $d+1$ where $d$ is the dimension of the vector of constraints. When in addition the model $\mathsf{M}$ is atomless, then the extreme points of this constrained set of occupation measures are precisely the occupation measures generated by deterministic stationary policies satisfying these constraints.

math.PR↗