Verifiable Conditions for the Irreducibility and Aperiodicity of Markov Chains by Analyzing Underlying Deterministic Models

Abstract

We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1} = F(\Phi_k, \alpha(\Phi_k, U_{k+1}))$ where the function $F$ is $C^1$ while $\alpha$ is typically discontinuous and $\{U_k: k \in \mathbb{Z}_{> 0} \}$ is an independent and identically distributed process. We assume that for all $x$, the random variable $\alpha(x, U_1)$ admits a density $p_x$ such that $(x, w) \mapsto p_x(w)$ is lower semi-continuous. We generalize and extend previous results that connect properties of the underlying deterministic control model to provide conditions for the chain to be $\varphi$-irreducible and aperiodic. By building on those results, we show that if a rank condition on the controllability matrix is satisfied for all $x$, there is equivalence between the existence of a globally attracting state for the control model and $\varphi$-irreducibility of the Markov chain. Additionally, under the same rank condition on the controllability matrix, we prove that there is equivalence between the existence of a steadily attracting state and the $\varphi$-irreducibility and aperiodicity of the chain. The notion of steadily attracting state is new. Those results hold under considerably weaker assumptions on the model than previous ones that would require $(x,u) \mapsto F(x,\alpha(x,u))$ to be $C^\infty$ (while it can be discontinuous here). Additionally the establishment of a necessary and sufficient condition for the $\varphi$-irreducibility and aperiodicity without a structural assumption on the control set is novel---even for Markov chains where $(x,u) \mapsto F(x,\alpha(x,u))$ is $C^\infty$. We illustrate that the conditions are easy to verify on a non-trivial and non-artificial example of Markov chain arising in the context of adaptive stochastic search algorithms to optimize continuous functions in a black-box scenario.