Continuous Policy and Value Iteration for Stochastic Control Problems with Inhomogeneous Synchronous Coupling

Abstract

We propose a continuous policy-value iteration framework for infinite horizon stochastic control problems, in which the policy evolves through Langevin type dynamics driven by the gradient of the inhomogeneous Hamiltonian, while the value function is updated simultaneously with optimal policy iterations, avoiding directly solving its corresponding PDE at each iteration. The iteration scheme enjoys policy improvement under suitable regularity and admissibility assumptions. We show the exponential contraction of the joint policy value dynamics in Wasserstein-2 distance under the Lipschitz condition in expectation via the inhomogeneous synchronous coupling and verify these conditions for linear quadratic problems. Numerical experiments on linear quadratic models and the optimal consumption-investment problems in an incomplete market factor model illustrate policy improvement, convergence toward optimal policies, and the effect of decaying exploration noise.