arXiv · 2611.11073
A Globally Convergent Line-Search Quasi-Newton Method for Discounted Markov Decision Processes
Abstract
In this paper, we develop a line-search quasi-Newton (LSQN) method for discounted Markov decision processes (MDPs). The method maintains an inverse model to approximate the inverse of the current policy Jacobian and refines it through an inner learning procedure that preserves existing secant conditions and achieves the prescribed accuracy in at most $n$ queries in exact arithmetic for an MDP with $n$ states, even when the initial inverse model is singular. A sign-based direction correction combined with an Armijo line search ensures monotonic decrease of the Bellman residual norm across policy changes without requiring a value-iteration fallback. We prove global Q-linear residual convergence and local Q-superlinear convergence of the value iterates from arbitrary finite initial value vectors and inverse models, without requiring a unique optimal policy. Numerical experiments show competitive performance against nine baselines, while public benchmarks demonstrate that the inner learning procedure typically requires far fewer than $n$ learning queries.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Wangzhi Zhou, Yuanqiu Mo, Wenwu Yu. 2026-11-08. A Globally Convergent Line-Search Quasi-Newton Method for Discounted Markov Decision Processes. https://searcharxiv.com/abs/2611.11073
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.