arXiv subjects

Ali D. Kara

Publications and source records attributed to Ali D. Kara.

5 recordsLinked to original sources

arXiv2026

Mean-Field Control with a Common Hidden State under Decentralized Observations

We study optimal control of a system with multiple decision makers who share a common hidden state and receive fully decentralized observations through identical channels. The dynamics of the hidden state and the cost incurred by the agents depend on the agents' actions only through their empirical distribution. In the limit problem with infinitely many agents, the problem reduces to a single agent control problem where the agent affects the hidden state dynamics via the conditional law of the actions given the past values of the hidden state process. We formulate this problem as a deterministic measure valued control problem over the space of policies and provide a dynamic programming recursion. We first show that for the limiting problem randomization over the control actions is necessary for optimality. However, randomization over the selection of policies (i.e., mixture policies) is not required. We then show that the optimal symmetric policies designed for the infinite population problem are near optimal for the finite population problem. In particular, we establish convergence rates that decay with number of agents as $\frac{1}{\sqrt{N}}$, and grow exponentially with the memory length used in the policy.

math.OC↗

arXiv2026

Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes

This paper develops a quantized Q-learning algorithm for the optimal control of controlled diffusion processes on $\mathbb{R}^d$ under both discounted and ergodic (average) cost criteria. We first establish near-optimality of finite-state MDP approximations to discrete-time discretizations of the diffusion, then introduce a quantized Q-learning scheme and prove its almost-sure convergence to near-optimal policies for the finite MDP. These policies, when interpolated to continuous time, are shown to be near-optimal for the original diffusion model under discounted costs and -- via a vanishing-discount argument -- also under ergodic costs for sufficiently small discount factors. The analysis applies under mild conditions (Lipschitz dynamics, non-degeneracy, bounded continuous costs, and Lyapunov stability for ergodic case) without requiring prior knowledge of the system dynamics or restrictions on control policies (beyond admissibility). Our results complement recent work on continuous-time reinforcement learning for diffusions by providing explicit near-optimality rates and extending rigorous guarantees both for discounted cost and ergodic cost criteria for diffusions with unbounded state space.

math.OC↗

arXiv2025

Quantizer Design for Finite Model Approximations, Model Learning, and Quantized Q-Learning for MDPs with Unbounded Spaces

In this paper, for Markov decision processes (MDPs) with unbounded state spaces we present refined upper bounds presented in [Kara et. al. JMLR'23] on finite model approximation errors via optimizing the quantizers used for finite model approximations. We also consider implications on quantizer design for quantized Q-learning and empirical model learning, and the performance of policies obtained via Q-learning where the quantized state is treated as the state itself. We highlight the distinctions between planning, where approximating MDPs can be independently designed, and learning (either via Q-learning or empirical model learning), where approximating MDPs are restricted to be defined by invariant measures of Markov chains under exploration policies, leading to significant subtleties on quantizer design performance, even though asymptotic near optimality can be established under both setups. In particular, under Lyapunov growth conditions, we obtain explicit upper bounds which decay to zero as the number of bins approaches infinity

math.OC↗

arXiv2025

Learning with Linear Function Approximations in Mean-Field Control

The paper focuses on mean-field type multi-agent control problems with finite state and action spaces where the dynamics and cost structures are symmetric and homogeneous, and are affected by the distribution of the agents. A standard solution method for these problems is to consider the infinite population limit as an approximation and use symmetric solutions of the limit problem to achieve near optimality. The control policies, and in particular the dynamics, depend on the population distribution in the finite population setting, or the marginal distribution of the state variable of a representative agent for the infinite population setting. Hence, learning and planning for these control problems generally require estimating the reaction of the system to all possible state distributions of the agents. To overcome this issue, we consider linear function approximation for the control problem and provide coordinated and independent learning methods. We rigorously establish error upper bounds for the performance of learned solutions. The performance gap stems from (i) the mismatch due to estimating the true model with a linear one, and (ii) using the infinite population solution in the finite population problem as an approximate control. The provided upper bounds quantify the impact of these error sources on the overall performance.

math.OC↗

arXiv2024

Infinite Horizon Average Cost Optimality Criteria for Mean-Field Control

We study mean-field control problems in discrete-time under the infinite horizon average cost optimality criteria. We focus on both the finite population and the infinite population setups. We show the existence of a solution to the average cost optimality equation (ACOE) and the existence of optimal stationary Markov policies for finite population problems under (i) a minorization condition that provides geometric ergodicity on the collective state process of the agents, and (ii) under standard Lipschitz continuity assumptions on the stage-wise cost and transition function of the agents when the Lipschitz constant of the transition function satisfies a certain bound. For the infinite population problem, we establish the existence of a solution to the ACOE, and the existence of optimal policies under the continuity assumptions on the cost and the transition functions. Finally, we relate the finite population and infinite population control problems: (i) we prove that the optimal value of the finite population problem converges to the optimal value of the infinite population problem as the number of agents grows to infinity; (ii) we show that the accumulation points of the finite population optimal solution corresponds to an optimal solution for the infinite population problem, and finally (iii), we show that one can use the solution of the infinite population problem for the finite population problem symmetrically across the agents to achieve near optimal performance when the population is sufficiently large.

math.OC↗