The Law of Large Numbers and CLT for Non-stationary Markov Jump Processes Exhibiting Time-of-Day Effects

Abstract

In this paper, we develop a general law of large numbers and central limit theorem for cumulative reward processes associated with finite state Markov jump processes with non-stationary transition rates. Such models commonly arise in service operations and manufacturing applications in which time-of-day, day-of-week, and secular effects are of first-order importance in predicting system behavior. Our theorems allow for non-stationary reward environments that continuously accumulate reward, while also including contributions from non-stationary lump-sum rewards of random size that are collected at either jump times of the underlying process, jump times of a Poisson process modulated by the underlying process, or scheduled deterministic times. As part of our development, we also obtain a new central limit theorem for the special case in which the jump process transition rates and reward structure are periodic (as may occur over a weekly time interval), as well as for jump process models with resetting. We include a simulation study illustrating the quality of our CLT approximations for several non-stationary stochastic models.