arXiv subjects

Eugene A. Feinberg

Publications and source records attributed to Eugene A. Feinberg.

47 records · Page 3Linked to original sources

arXiv2014

Partially Observable Total-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper describes sufficient conditions for the existence of optimal policies for Partially Observable Markov Decision Processes (POMDPs) with Borel state, observation, and action sets and with the expected total costs. Action sets may not be compact and one-step cost functions may be unbounded. The introduced conditions are also sufficient for the validity of optimality equations, semi-continuity of value functions, and convergence of value iterations to optimal values. Since POMDPs can be reduced to Completely Observable Markov Decision Processes (COMDPs), whose states are posterior state distributions, this paper focuses on the validity of the above mentioned optimality properties for COMDPs. The central question is whether transition probabilities for a COMDP are weakly continuous. We introduce sufficient conditions for this and show that the transition probabilities for a COMDP are weakly continuous, if transition probabilities of the underlying Markov Decision Process are weakly continuous and observation probabilities for the POMDP are continuous in the total variation. Moreover, the continuity in the total variation of the observation probabilities cannot be weakened to setwise continuity. The results are illustrated with counterexamples and examples.

math.OC↗

arXiv2013

The Value Iteration Algorithm is Not Strongly Polynomial for Discounted Dynamic Programming

This note provides a simple example demonstrating that, if exact computations are allowed, the number of iterations required for the value iteration algorithm to find an optimal policy for discounted dynamic programming problems may grow arbitrarily quickly with the size of the problem. In particular, the number of iterations can be exponential in the number of actions. Thus, unlike policy iterations, the value iteration algorithm is not strongly polynomial for discounted dynamic programming.

cs.AI↗

arXiv2013

Fatou's Lemma for Weakly Converging Probabilities

Fatou's lemma states under appropriate conditions that the integral of the lower limit of a sequence of functions is not greater than the lower limit of the integrals. This note describes similar inequalities when, instead of a single measure, the functions are integrated with respect to different measures that form a weakly convergent sequence.

math.PR↗

arXiv2013

Berge's Maximum Theorem for Noncompact Image Sets

This note generalizes Berge's maximum theorem to noncompact image sets. It is also clarifies the results from E.A. Feinberg, P.O. Kasyanov, N.V. Zadoianchuk, "Berge's theorem for noncompact image sets," J. Math. Anal. Appl. 397(1)(2013), pp. 255-259 on the extension to noncompact image sets of another Berge's theorem, that states semi-continuity of value functions. Here we explain that the notion of a $\K$-inf-compact function introduced there is applicable to metrizable topological spaces and to more general compactly generated topological spaces. For Hausdorff topological spaces we introduce the notion of a $\K\N$-inf-compact function ($\N$ stands for "nets" in $\K$-inf-compactness), which coincides with $\K$-inf-compactness for compactly generated and, in particular, for metrizable topological spaces.

math.GN↗

arXiv2013

On solutions of Kolmogorov's equations for jump Markov processes

This paper studies three ways to construct a nonhomogeneous jump Markov process: (i) via a compensator of the random measure of a multivariate point process, (ii) as a minimal solution of the backward Kolmogorov equation, and (iii) as a minimal solution of the forward Kolmogorov equation. The main conclusion of this paper is that, for a given measurable transition intensity, commonly called a Q-function, all these constructions define the same transition function. If this transition function is regular, that is, the probability of accumulation of jumps is zero, then this transition function is the unique solution of the backward and forward Kolmogorov equations. For continuous Q-functions, Kolmogorov equations were studied in Feller's seminal paper. In particular, this paper extends Feller's results for continuous Q-functions to measurable Q-functions and provides additional results.

math.PR↗

arXiv2012

The Multi-Armed Bandit, with Constraints

The early sections of this paper present an analysis of a Markov decision model that is known as the multi-armed bandit under the assumption that the utility function of the decision maker is either linear or exponential. The analysis includes efficient procedures for computing the expected utility associated with the use of a priority policy and for identifying a priority policy that is optimal. The methodology in these sections is novel, building on the use of elementary row operations. In the later sections of this paper, the analysis is adapted to accommodate constraints that link the bandits.

math.OC↗

arXiv2012

Berge's Theorem for Noncompact Image Sets

For an upper semi-continuous set-valued mapping from one topological space to another and for a lower semi-continuous function defined on the product of these spaces, Berge's theorem states lower semi-continuity of the minimum of this function taken over the image sets. It assumes that the image sets are compact. For Hausdorff topological spaces, this paper extends Berge's theorem to set-valued mappings with possible noncompact image sets and studies relevant properties of minima.

math.GN↗

arXiv2012

Average-Cost Markov Decision Processes with Weakly Continuous Transition Probabilities

This paper presents sufficient conditions for the existence of stationary optimal policies for average-cost Markov Decision Processes with Borel state and action sets and with weakly continuous transition probabilities. The one-step cost functions may be unbounded, and action sets may be noncompact. The main contributions of this paper are: (i) general sufficient conditions for the existence of stationary discount-optimal and average-cost optimal policies and descriptions of properties of value functions and sets of optimal actions, (ii) a sufficient condition for the average-cost optimality of a stationary policy in the form of optimality inequalities, and (iii) approximations of average-cost optimal actions by discount-optimal actions.

math.OC↗

arXiv2011

Extension of Lyapunov's Convexity Theorem to Subranges

Consider a measurable space with a finite vector measure. This measure defines a mapping of the $σ$-field into a Euclidean space. According to Lyapunov's convexity theorem, the range of this mapping is compact and, if the measure is atomless, this range is convex. Similar ranges are also defined for measurable subsets of the space. We show that the union of the ranges of all subsets having the same given vector measure is also compact and, if the measure is atomless, it is convex. We further provide a geometrically constructed convex compactum in the Euclidean space that contains this union. The equality of these two sets, that holds for two-dimensional measures, can be violated in higher dimensions.

math.PR↗

arXiv2011

On Maximal Ranges of Vector Measures for Subsets and Purification of Transition Probabilities

Consider a measurable space with an atomless finite vector measure. This measure defines a mapping of the $σ$-field into an Euclidean space. According to the Lyapunov convexity theorem, the range of this mapping is a convex compactum. Similar ranges are also defined for measurable subsets of the space. Two subsets with the same vector measure may have different ranges. We investigate the question whether, among all the subsets having the same given vector measure, there always exists a set with the maximal range of the vector measure. The answer to this question is positive for two-dimensional vector measures and negative for higher dimensions. We use the existence of maximal ranges to strengthen the Dvoretzky-Wald-Wolfowitz purification theorem for the case of two measures.

math.PR↗

arXiv2007

Buffer Insertion for Bridges and Optimal Buffer Sizing for Communication Sub-System of Systems-on-Chip

We have presented an optimal buffer sizing and buffer insertion methodology which uses stochastic models of the architecture and Continuous Time Markov Decision Processes CTMDPs. Such a methodology is useful in managing the scarce buffer resources available on chip as compared to network based data communication which can have large buffer space. The modeling of this problem in terms of a CT-MDP framework lead to a nonlinear formulation due to usage of bridges in the bus architecture. We present a methodology to split the problem into several smaller though linear systems and we then solve these subsystems.

cs.AR↗