arXiv subjects

François Dufour

Publications and source records attributed to François Dufour.

At least 19 recordsLinked to original sources

arXiv2025

Absorbing Markov Decision Processes

In this paper, we study discrete-time absorbing Markov Decision Processes (MDP) with measurable state space and Borel action space with a given initial distribution. For such models, solutions to the characteristic equation that are not occupation measures may exist. Several necessary and sufficient conditions are provided to guarantee that any solution to the characteristic equation is an occupation measure. Under the so-called continuity-compactness conditions, it is shown that the set of occupation measures is compact in the weak-strong topology if and only if the model is uniformly absorbing. Finally, it is shown that the occupation measures are characterized by the characteristic equation and an additional condition. Several examples are provided to illustrate our results.

math.OC↗

arXiv2024

A note on weak compactness of occupation measures for an absorbing Markov decision process

We consider an absorbing Markov decision process with Borel state and action spaces. We study conditions under which the MDP is uniformly absorbing and the set of occupation measures of the MDP is compact in the usual weak topology. These include suitable continuity requirements on the transition kernel and conditions on the dynamics of the system at the boundary of the absorbing set. We generalize previously known results and give an answer to some conjectures that have been mentioned in the related literature.

math.PR↗

arXiv2023

Nash equilibria for total expected reward absorbing Markov games: the constrained and unconstrained cases

We consider a nonzero-sum N-player Markov game on an abstract measurable state space with compact metric action spaces. The payoff functions are bounded Carathéodory functions and the transitions of the system are assumed to have a density function satisfying some continuity conditions. The optimality criterion of the players is given by a total expected payoff on an infinite discrete-time horizon. Under the condition that the game model is absorbing, we establish the existence of Markov strategies that are a noncooperative equilibrium in the family of all history-dependent strategies of the players for both the constrained and the unconstrained problems, We obtain, as a particular case of results, the existence of Nash equilibria for discounted constrained and unconstrained game models.

math.OC↗

arXiv2021

Stationary Markov Nash equilibria for nonzero-sum constrained ARAT Markov games

We consider a nonzero-sum Markov game on an abstract measurable state space with compact metric action spaces. The goal of each player is to maximize his respective discounted payoff function under the condition that some constraints on a discounted payoff are satisfied. We are interested in the existence of a Nash or noncooperative equilibrium. Under suitable conditions, which include absolute continuity of the transitions with respect to some reference probability measure, additivity of the payoffs and the transition probabilities (ARAT condition), and continuity in action of the payoff functions and the density function of the transitions of the system, we establish the existence of a constrained stationary Markov Nash equilibrium, that is, the existence of stationary Markov strategies for each of the players yielding an optimal profile within the class of all history-dependent profiles.

math.OC↗

arXiv2018

On the expected total reward with unbounded returns for Markov decision processes

We consider a discrete-time Markov decision process with Borel state and action spaces. The performance criterion is to maximize a total expected {utility determined by unbounded return function. It is shown the existence of optimal strategies under general conditions allowing the reward function to be unbounded both from above and below and the action sets available at each step to the decision maker to be not necessarily compact. To deal with unbounded reward functions, a new characterization for the weak convergence of probability measures is derived. Our results are illustrated by examples.

math.PR↗

arXiv2018

Controlling IL-7 injections in HIV-infected patients

Immune interventions consisting in repeated injection are broadly used as they are thought to improve the quantity and the quality of the immune response. However, they also raised several questions that remains unanswered, in particular the number of injections to make or the delay to respect between different injections to acheive this goal. Practical and financial considerations add constraints to these questions, especially in the framework of human studies. We specifically focus here on the use of interleukine-7 (IL-7) injections in HIV-infected patients under antiretroviral treatment, but still unable to restore normal levels of CD4+ T lymphocytes. Clinical trials have already shown that repeated cycles of injections of IL-7 could help maintaining CD4+ T lymphocytes levels over the limit of 500 cells per microL, by affecting proliferation and survival of CD4+ T cells. We then aim at answering the question : how to maintain a patient's level of CD4+ T lymphocytes by using a minimum number of injections (ie optimizing the strategy of injections) ? Based on mechanistic models that were previously developed for the dynamics of CD4+ T lymphocytes in this context, we model the process by a piecewise deterministic Markov model. We then address the question by using some recently established theory on impulse control problem in order to develop a numerical tool determining the optimal strategy. Results are obtained on a reduced model, as a proof of concept : the method allows to defined an optimal strategy for a given patient. This method could applied to optimize injections schedules in clinical trials.

math.OC↗

arXiv2017

Stochastic Control of Observer Trajectories in Bearings-only Tracking with Acoustic Signal Propagation Optimization

We present in this paper a numerical method which computes the optimal trajectory of a underwater vehicle subject to some mission objectives. The method is applied to a submarine whose goal is to best detect one or several targets, or/and to minimize its own detection range perceived by the other targets. The signal considered is acoustic propagation attenuation. Our approach is based on dynamic programming of a finite horizon Markov decision process. A quantization method is applied to fully discretize the problem and allows a numerically tractable solution. Different scenarios are considered. We suppose at first that the position and the velocity of the targets are known and in the second we suppose that they are unknown and estimated by a Kalman type filter in a context of bearings-only tracking.

math.OC↗

arXiv2016

Optimal strategies for impulse control of piecewise deterministic Markov processes

This paper deals with the general discounted impulse control problem of a piecewise deterministic Markov process. We investigate a new family of epsilon-optimal strategies. The construction of such strategies is explicit and only necessitates the previous knowledge of the cost of the no-impulse strategy. In particular, it does not require the resolution of auxiliary optimal stopping problem or the computation of the value function at each point of the state space. This approach is based on the iteration of a single-jump-or-intervention operator associated to the piecewise deterministic Markov process.

math.PR↗

arXiv2016

Partially observed optimal stopping problem for discrete-time Markov processes

This paper is dedicated to the investigation of a new numerical method to approximate the optimal stopping problem for a discrete-time continuous state space Markov chain under partial observations. It is based on a two-step discretization procedure based on optimal quantization. First,we discretize the state space of the unobserved variable by quantizing an underlying reference measure. Then we jointly discretize the resulting approximate filter and the observation process. We obtain a fully computable approximation of the value function with explicit error bounds for its convergence towards the true value fonction.

math.OC↗

arXiv2016

Optimization of a launcher integration process: a Markov decision process approach

This paper is dedicated to the numerical study of the optimization of an industrial launcher integration process. It is an original case of inventory-production system where a calendar plays a crucial role. The process is modeled using the Markov Decision Processes (MDPs) framework. Classical optimization procedures for MDPs cannot be used because of specificities of the transition law and cost function. Two simulation-based algorithms are tuned to fit this special case. We obtain a non trivial optimal policy that can be applied in practice and significantly outperforms reference policies.

math.OC↗

arXiv2014

A Linear Programming Formulation for Constrained Discounted Continuous Control for Piecewise Deterministic Markov Processes

This papers deals with the constrained discounted control of piecewise deterministic Markov process (PDMPs) in general Borel spaces. The control variable acts on the jump rate and transition measure, and the goal is to minimize the total expected discounted cost, composed of positive running and boundary costs, while satisfying some constraints also in this form. The basic idea is, by using the special features of the PDMPs, to re-write the problem via an embedded discrete-time Markov chain associated to the PDMP and re-formulate the problem as an infinite dimensional linear programming (LP) problem, via the occupation measures associated to the discrete-time process. It is important to stress however that our new discrete-time problem is not in the same framework of a general constrained discrete-time Markov Decision Process and, due to that, some conditions are required to get the equivalence between the continuous-time problem and the LP formulation. We provide in the sequel sufficient conditions for the solvability of the associated LP problem, based on a generalization of Theorem 4.1 in [8]. In the Appendix we present the proof of this generalization which, we believe, is of interest on its own. The paper is concluded with some examples to illustrate the obtained results.

math.OC↗

arXiv2014

Impulsive control for continuous-time Markov Decision Processes

The objective of this work is to study continuous-time Markov decision processes on a general Borel state space with both impulsive and continuous controls for the infinite-time horizon discounted cost. The continuous-time controlled process is shown to be non explosive under appropriate hypotheses. The so-called Bellman equation associated to this control problem is studied. Sufficient conditions ensuring the existence and the uniqueness of a bounded measurable solution to this optimality equation are provided. Moreover, it is shown that the value function of the optimization problem under consideration satisfies this optimality equation. Sufficient conditions are also presented to ensure on one hand the existence of an optimal control strategy and on the other hand the existence of an $\varepsilon$-optimal control strategy. The decomposition of the state space in two disjoint subsets is exhibited where roughly speaking, one should apply a gradual action or an impulsive action correspondingly to get an optimal or $\varepsilon$-optimal strategy. An interesting consequence of our previous results is as follows: the set of strategies that allow interventions at time $t=0$ and only immediately after natural jumps is a sufficient set for the control problem under consideration.

math.OC↗

arXiv2013

Limits on the Number of Galactic Young Supernova Remnants Emitting in the Decay Lines of 44Ti

We revise the assumptions of the parameters involved in predicting the number of supernova remnants detectable in the nuclear lines of the decay chain of 44Ti. Specifically, we consider the distribution of the supernova progenitors, the supernova rate in the Galaxy, the ratios of supernova types, the Galactic production of 44Ti, and the 44Ti yield from supernovae of different types, to derive credible bounds on the expected number of detectable remnants. We find that, within 1 sigma uncertainty, the Galaxy should contain an average of 5.1+2.4-2.0 remnants detectable to a survey with a 44Ti decay line flux limit of 10E-5 photons/cm2/s, with a probability of detecting a single remnant of (2.7+10.0-2.4)%, and an expected number of detections between 2 and 9 remnants, making the single detection of Cas A unlikely but consistent with our models. Our results show that the probability of detecting the brightest 44Ti flux source at the high absolute Galactic longitude of Cas A or above is ~10%. Using the detected flux of Cas A, we attempt to constrain the Galactic supernova rate and Galactic production of 44Ti, but find the detection to be only weakly informative. We conclude that even future surveys having 200 times more sensitivity than state-of-the art surveys can be guaranteed to detect only a few new remnants, with an expected number of detections between 8 and 21 at a limiting 44Ti decay flux of 10E-7 photons/cm2/s.

astro-ph.HE↗

arXiv2013

Optimal stopping for partially observed piecewise-deterministic Markov processes

This paper deals with the optimal stopping problem under partial observation for piecewise-deterministic Markov processes. We first obtain a recursive formulation of the optimal filter process and derive the dynamic programming equation of the partially observed optimal stopping problem. Then, we propose a numerical method, based on the quantization of the discrete-time filter process and the inter-jump times, to approximate the value function and to compute an actual $ε$-optimal stopping time. We prove the convergence of the algorithms and bound the rates of convergence.

math.PR↗

arXiv2012

Nonparametric estimation of the jump rate for non-homogeneous marked renewal processes

This paper is devoted to the nonparametric estimation of the jump rate and the cumulative rate for a general class of non-homogeneous marked renewal processes, defined on a separable metric space. In our framework, the estimation needs only one observation of the process within a long time. Our approach is based on a generalization of the multiplicative intensity model, introduced by Aalen in the seventies. We provide consistent estimators of these two functions, under some assumptions related to the ergodicity of an embedded chain and the characteristics of the process. The paper is illustrated by a numerical example.

math.ST↗

arXiv2012

Nonparametric estimation of the conditional distribution of the inter-jumping times for piecewise-deterministic Markov processes

This paper presents a nonparametric method for estimating the conditional density associated to the jump rate of a piecewise-deterministic Markov process. In our framework, the estimation needs only one observation of the process within a long time interval. Our method relies on a generalization of Aalen's multiplicative intensity model. We prove the uniform consistency of our estimator, under some reasonable assumptions related to the primitive characteristics of the process. A simulation example illustrates the behavior of our estimator.

math.ST↗

arXiv2012

Numerical method for expectations of piecewise-determistic Markov processes

We present a numerical method to compute expectations of functionals of a piecewise-deterministic Markov process. We discuss time dependent functionals as well as deterministic time horizon problems. Our approach is based on the quantization of an underlying discrete-time Markov chain. We obtain bounds for the rate of convergence of the algorithm. The approximation we propose is easily computable and is flexible with respect to some of the parameters defining the problem. Two examples illustrate the paper.

math.PR↗

arXiv2011

Numerical method for impulse control of Piecewise Deterministic Markov Processes

This paper presents a numerical method to calculate the value function for a general discounted impulse control problem for piecewise deterministic Markov processes. Our approach is based on a quantization technique for the underlying Markov chain defined by the post jump location and inter-arrival time. Convergence results are obtained and more importantly we are able to give a convergence rate of the algorithm. The paper is illustrated by a numerical example.

math.PR↗