Searcharxiv⌕ Search

arXiv subjects

Eugene A. Feinberg

Publications and source records attributed to Eugene A. Feinberg.

At least 37 records · Page 2Linked to original sources
arXiv2019

Fatou's Lemma in Its Classic Form and Lebesgue's Convergence Theorems for Varying Measures with Applications to MDPs

The classic Fatou lemma states that the lower limit of a sequence of integrals of functions is greater or equal than the integral of the lower limit. It is known that Fatou's lemma for a sequence of weakly converging measures states a weaker inequality because the integral of the lower limit is replaced with the integral of the lower limit in two parameters, where the second parameter is the argument of the functions. This paper provides sufficient conditions when Fatou's lemma holds in its classic form for a sequence of weakly converging measures. The functions can take both positive and negative values. The paper also provides similar results for sequences of setwise converging measures. It also provides Lebesgue's and monotone convergence theorems for sequences of weakly and setwise converging measures. The obtained results are used to prove broad sufficient conditions for the validity of optimality equations for average-cost Markov decision processes.

math.PR↗
arXiv2019

Constrained discounted Markov decision processes with Borel state spaces

We study discrete-time discounted constrained Markov decision processes (CMDPs) on Borel spaces with unbounded reward functions. In our approach the transition probability functions are weakly or set-wise continuous. The reward functions are upper semicontinuous in state-action pairs or semicontinuous in actions. Our aim is to study models with unbounded reward functions, which are often encountered in applications, e.g., in consumption/investment problems. We provide some general assumptions under which the optimization problems in CMDPs are solvable in the class of stationary randomized policies. Then, we indicate that if the initial distribution and transition probabilities are non-atomic, then using a general purification result of Feinberg and Piunovskiy, stationary optimal policies can be deterministic. Our main results are illustrated by five examples.

math.OC↗
arXiv2019

Fatou's Lemma for Weakly Converging Measures under the Uniform Integrability Condition

This note describes Fatou's lemma and Lebesgue's dominated convergence theorem for a sequence of measures converging weakly to a finite measure and for a sequence of functions whose negative parts are uniformly integrable with respect to these measures. The note also provides new formulations of uniform Fatou's lemma, uniform Lebesgue convergence theorem, the Dunford-Pettis theorem, and the fundamental theorem for Young measures based on the equivalence of uniform integrability and the apparently weaker property of asymptotic uniform integrability for sequences of functions and finite measures.

math.CA↗
arXiv2018

Sufficiency of Deterministic Policies for Atomless Discounted and Uniformly Absorbing MDPs with Multiple Criteria

This paper studies Markov Decision Processes (MDPs) with atomless initial state distributions and atomless transition probabilities. Such MDPs are called atomless. The initial state distribution is considered to be fixed. We show that for discounted MDPs with bounded one-step reward vector-functions, for each policy there exists a deterministic (that is, nonrandomized and stationary) policy with the same performance vector. This fact is proved in the paper for a more general class of uniformly absorbing MDPs with expected total costs, and then it is extended under certain assumptions to MDPs with unbounded rewards. For problems with multiple criteria and constraints, the results of this paper imply that for atomless MDPs studied in this paper it is sufficient to consider only deterministic policies, while without the atomless assumption it is well-known that randomized policies can outperform deterministic ones. We also provide an example of an MDP demonstrating that, if a vector measure is defined on a standard Borel space, then Lyapunov's convexity theorem is a special case of the described results.

math.OC↗
arXiv2018

An example showing that A-lower semi-continuity is essential for minimax continuity theorems

Recently Feinberg et al. [arXiv:1609.03990] established results on continuity properties of minimax values and solution sets for a function of two variables depending on a parameter. Such minimax problems appear in games with perfect information, when the second player knows the move of the first one, in turn-based games, and in robust optimization. Some of the results in [arXiv:1609.03990] are proved under the assumption that the multifunction, defining the domains of the second variable, is $A$-lower semi-continuous. The $A$-lower semi-continuity property is stronger than lower semi-continuity, but in several important cases these properties coincide. This note provides an example demonstrating that in general the $A$-lower semi-continuity assumption cannot be relaxed to lower semi-continuity.

math.OC↗
arXiv2017

Reduction of total-cost and average-cost MDPs with weakly continuous transition probabilities to discounted MDPs

This note describes sufficient conditions under which total-cost and average-cost Markov decision processes (MDPs) with general state and action spaces, and with weakly continuous transition probabilities, can be reduced to discounted MDPs. For undiscounted problems, these reductions imply the validity of optimality equations and the existence of stationary optimal policies. The reductions also provide methods for computing optimal policies. The results are applied to a capacitated inventory control problem with fixed costs and lost sales.

math.OC↗
arXiv2017

Stochastic Setup-Cost Inventory Model with Backorders and Quasiconvex Cost Functions

In this paper we study a periodic-review single-commodity setup-cost inventory model with backorders and holding/backlog costs satisfying quasiconvexity assumptions. We show that the Markov decision process for this inventory model satisfies the assumptions that lead to the validity of optimality equations for discounted and average-cost problems and to the existence of optimal $(s,S)$ policies. In particular, we prove the equicontinuity of the family of discounted value functions and the convergence of optimal discounted lower thresholds to the optimal average-cost one for some sequences of discount factors converging to $1.$ If an arbitrary nonnegative amount of inventory can be ordered, we establish stronger convergence properties: (i) the optimal discounted lower thresholds $s_α$ converge to optimal average-cost lower threshold $s;$ and (ii) the discounted relative value functions converge to average-cost relative value function. These convergence results previously were known only for subsequences of discount factors even for problems with convex holding/backlog costs. The results of this paper hold for problems with deterministic positive lead times.

math.OC↗
arXiv2017

Continuity of Equilibria for Two-Person Zero-Sum Games with Noncompact Action Sets and Unbounded Payoffs

This paper extends Berge's maximum theorem for possibly noncompact action sets and unbounded cost functions to minimax problems and studies applications of these extensions to two-player zero-sum games with possibly noncompact action sets and unbounded payoffs. For games with perfect information, also known under the name of turn-based games, this paper establishes continuity properties of value functions and solution multifunctions. For games with simultaneous moves, it provides results on the existence of lopsided values (the values in the asymmetric form) and solutions. This paper also establishes continuity properties of the lopsided values and solution multifunctions.

math.OC↗
arXiv2017

Structure of Optimal Solutions to Periodic-Review Total-Cost Inventory Control Models with Convex Costs and Backorders for all Values of Discount Factors

This paper describes the structure of optimal policies for discounted periodic-review single-commodity total-cost inventory control problems with fixed ordering costs for finite and infinite horizons. There are known conditions in the literature for optimality of $(s_t,S_t)$ policies for finite-horizon problems and the optimality of $(s,S)$ policies for infinite-horizon problems. The results of this paper cover the situation, when such assumption may not hold. This paper describes a parameter, which, together with the value of the discount factor and the horizon length, defines the structure of an optimal policy. For the infinite horizon, depending on the values of this parameter and the discount factor, an optimal policy either is an $(s,S)$ policy or never orders inventory. For a finite horizon, depending on the values of this parameter, the discount factor, and the horizon length, there are three possible structures of an optimal policy: (i) it is an $(s_t,S_t)$ policy, (ii) it is an $(s_t,S_t)$ policy at earlier stages and then does not order inventory, or (iii) it never orders inventory. The paper also establishes continuity of optimal value functions and describes alternative optimal actions at states $s_t$ and $s.$

math.OC↗
arXiv2017

On the Reduction of Total-Cost and Average-Cost MDPs to Discounted MDPs

This paper provides conditions under which total-cost and average-cost Markov decision processes (MDPs) can be reduced to discounted ones. Results are given for transient total-cost MDPs with tran- sition rates whose values may be greater than one, as well as for average-cost MDPs with transition probabilities satisfying the condition that there is a state such that the expected time to reach it is uniformly bounded for all initial states and stationary policies. In particular, these reductions imply sufficient conditions for the validity of optimality equations and the existence of stationary optimal poli- cies for MDPs with undiscounted total cost and average-cost criteria. When the state and action sets are finite, these reductions lead to linear programming formulations and complexity estimates for MDPs under the aforementioned criteria.

math.OC↗
arXiv2017

On the Convergence of Optimal Actions for Markov Decision Processes and the Optimality of $(s,S)$ Inventory Policies

This paper studies convergence properties of optimal values and actions for discounted and average-cost Markov Decision Processes (MDPs) with weakly continuous transition probabilities and applies these properties to the stochastic periodic-review inventory control problem with backorders, positive setup costs, and convex holding/backordering costs. The following results are established for MDPs with possibly noncompact action sets and unbounded cost functions: (i) convergence of value iterations to optimal values for discounted problems with possibly non-zero terminal costs, (ii) convergence of optimal finite-horizon actions to optimal infinite-horizon actions for total discounted costs, as the time horizon tends to infinity, and (iii) convergence of optimal discount-cost actions to optimal average-cost actions for infinite-horizon problems, as the discount factor tends to 1. Being applied to the setup-cost inventory control problem, the general results on MDPs imply the optimality of $(s,S)$ policies and convergence properties of optimal thresholds. In particular this paper analyzes the setup-cost inventory control problem without two assumptions often used in the literature: (a) the demand is either discrete or continuous or (b) the backordering cost is higher than the cost of backordered inventory if the amount of backordered inventory is large.

math.OC↗
arXiv2016

Kolmogorov's Equations for Jump Markov Processes with Unbounded Jump Rates

As well-known, transition probabilities of jump Markov processes satisfy Kolmogorov's backward and forward equations. In the seminal 1940 paper, William Feller investigated solutions of Kolmogorov's equations for jump Markov processes. Recently the authors solved the problem studied by Feller and showed that the minimal solution of Kolmogorov's backward and forward equations is the transition probability of the corresponding jump Markov process if the transition rate at each state is bounded. This paper presents more general results. For Kolmogorov's backward equation, the sufficient condition for the described property of the minimal solution is that the transition rate at each state is locally integrable, and for Kolmogorov's forward equation the corresponding sufficient condition is that the transition rate at each state is locally bounded.

math.PR↗
arXiv2016

On the Optimality Equation for Average Cost Markov Decision Processes and its Validity for Inventory Control

As is well known, average-cost optimality inequalities imply the existence of stationary optimal policies for Markov Decision Processes with average costs per unit time, and these inequalities hold under broad natural conditions. This paper provides sufficient conditions for the validity of the average-cost optimality equation for an infinite state problem with weakly continuous transition probabilities and with possibly unbounded one-step costs and noncompact action sets. These conditions also imply the convergence of sequences of discounted relative value functions to average-cost relative value functions and the continuity of average-cost relative value functions. As shown in the paper, the classic periodic-review inventory control problem satisfies these conditions. Therefore, the optimality inequality holds in the form of an equality with a continuous average-cost relative value function for this problem. In addition, the $K$-convexity of discounted relative value functions and their convergence to average-cost relative value functions, when the discount factor increases to 1, imply the $K$-convexity of average-cost relative value functions. This implies that average-cost optimal $(s,S)$ policies for the inventory control problem can be derived from the average-cost optimality equation.

math.OC↗
arXiv2016

Optimality Conditions for Inventory Control

This tutorial describes recently developed general optimality conditions for Markov Decision Processes that have significant applications to inventory control. In particular, these conditions imply the validity of optimality equations and inequalities. They also imply the convergence of value iteration algorithms. For total discounted-cost problems only two mild conditions on the continuity of transition probabilities and lower semi-continuity of one-step costs are needed. For average-cost problems, a single additional assumption on the finiteness of relative values is required. The general results are applied to periodic-review inventory control problems with discounted and average-cost criteria without any assumptions on demand distributions. The case of partially observable states is also discussed.

math.OC↗
arXiv2015

Uniform Fatou's Lemma

Fatou's lemma is a classic fact in real analysis that states that the limit inferior of integrals of functions is greater than or equal to the integral of the inferior limit. This paper introduces a stronger inequality that holds uniformly for integrals on measurable subsets of a measurable space. The necessary and sufficient condition, under which this inequality holds for a sequence of finite measures converging in total variation, is provided. This statement is called the uniform Fatou's lemma, and it holds under the minor assumption that all the integrals are well-defined. The uniform Fatou's lemma improves the classic Fatou's lemma in the following directions: the uniform Fatou's lemma states a more precise inequality, it provides the necessary and sufficient condition, and it deals with variable measures. Various corollaries of the uniform Fatou's lemma are formulated. The examples in this paper demonstrate that: (a) the uniform Fatou's lemma may indeed provide a more accurate inequality than the classic Fatou's lemma; (b) the uniform Fatou's lemma does not hold if convergence of measures in total variation is relaxed to setwise convergence.

math.FA↗
arXiv2014

Continuity of Minima: Local Results

This paper compares and generalizes Berge's maximum theorem for noncompact image sets established in Feinberg, Kasyanov and Voorneveld (2014) and the local maximum theorem established in Bonnans and Shapiro (2000).

math.OC↗
arXiv2014

Convergence of Probability Measures and Markov Decision Models with Incomplete Information

This paper deals with three major types of convergence of probability measures on metric spaces: weak convergence, setwise converges, and convergence in the total variation. First, it describes and compares necessary and sufficient conditions for these types of convergence, some of which are well-known, in terms of convergence of probabilities of open and closed sets and, for the probabilities on the real line, in terms of convergence of distribution functions. Second, it provides % convenient criteria for weak and setwise convergence of probability measures and continuity of stochastic kernels in terms of convergence of probabilities defined on the base of the topology generated by the metric. Third, it provides applications to control of Partially Observable Markov Decision Processes and, in particular, to Markov Decision Models with incomplete information.

math.PR↗