On Convergence of Value Iteration for a Class of Total Cost Markov Decision Processes

Abstract

We consider a general class of total cost Markov decision processes (MDP) in which the one-stage costs can have arbitrary signs, but the sum of the negative parts of the one-stage costs is finite for all policies and all initial states. We refer to this class as the General Convergence (GC for short) total cost model, and we study the convergence of value iteration for the GC model, in the Borel MDP framework with universally measurable policies. Our main results include: (i) convergence of value iteration when starting from certain functions above the optimal cost function; (ii) convergence of transfinite value iteration starting from zero, in the special case where the optimal cost function is nonnegative; and (iii) partial convergence of value iteration starting from zero, for a subset of initial states. These results extend several previously known results about the convergence of value iteration for either positive costs problems or GC total cost problems. In particular, the first result on convergence of value iteration from above extends a theorem of van der Wal for the GC model. The second result relates to Maitra and Sudderth's analysis of transfinite value iteration for the positive costs model. It suggests connections between the two total cost models when the optimal cost function is nonnegative, and it leads to additional results on the convergence of ordinary non-transfinite value iteration, with a suitably defined dynamic programming operator, for finite state or finite control GC problems. The third result on partial convergence of value iteration is motivated by Whittle's bridging condition for the positive costs model, and provides a novel extension of the bridging condition to the GC model, where there are no sign constraints on the costs.