arXiv subjects

David Šiška

Publications and source records attributed to David Šiška.

At least 19 recordsLinked to original sources

arXiv2026

Mirror descent for constrained stochastic control problems

Mirror descent is a well established tool for solving convex optimization problems with convex constraints. This article introduces continuous-time mirror descent dynamics for approximating optimal Markov controls for stochastic control problems with the action space being bounded and convex. We show that if the Hamiltonian is uniformly convex in its action variable then mirror descent converges linearly while if it is uniformly strongly convex relative to an appropriate Bregman divergence, then the mirror flow converges exponentially. The two fundamental difficulties that must be overcome to prove such results are: first, the inherent lack of convexity of the map from Markov controls to the corresponding value function. Second, maintaining sufficient regularity of the value function and the Markov controls along the mirror descent updates. The first issue is handled using the performance difference lemma, while the second using careful Sobolev space estimates for the solutions of the associated linear PDEs. Finally, we complement the theoretical analysis with numerical experiments. We provide a model-based algorithm which uses finite difference discretization and discrete mirror descent updates. Empirical results confirm the theoretically predicted convergence rates. We also provide a model-free algorithm which uses martingale differences to learn the advantage rate function and applies this in the mirror descent updates of the policy.

math.OC↗

arXiv2026

Linear convergence of proximal descent schemes on the Wasserstein space

We investigate proximal descent methods, inspired by the minimizing movement scheme introduced by Jordan, Kinderlehrer and Otto, for optimizing entropy-regularized functionals on the Wasserstein space. We establish linear convergence under flat convexity assumptions, thereby relaxing the common reliance on geodesic convexity. Our analysis circumvents the need for discrete-time adaptations of the Evolution Variational Inequality (EVI). Instead, we leverage a uniform logarithmic Sobolev inequality (LSI) and the entropy ``sandwich" lemma, extending the analysis from arXiv:2201.10469 and arXiv:2202.01009. The major challenge in the proof via LSI is to show that the relative Fisher information is well-defined at every step of the scheme. Since the relative entropy is not Wasserstein differentiable, we prove that along the scheme the iterates belong to a certain class of Sobolev regularity, and hence the relative entropy has a unique Wasserstein sub-gradient, and that the relative Fisher information is indeed finite.

math.OC↗

arXiv2026

Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learning

Entropy regularization is widely used in continuous-time reinforcement learning (RL) to reduce sensitivity to environmental perturbations, yet its robustness benefits lack a rigorous theoretical foundation. This paper establishes the first robustness guarantees for entropy-regularized continuous-time Markov decision processes. We show that maximizing an entropy-regularized objective yields a lower bound on a worst-case robust RL problem with joint reward and transition perturbations. We analytically characterize the induced robust sets and prove that they expand monotonically with the regularization strength, justifying the empirical observation that stronger entropy improves robustness. In contrast to prior discrete-time analyses, our results remove the intractable state-distribution entropy term and provide guarantees invariant to action frequency. Experiments on queueing network control and market making confirm our theory, showing that entropy-regularized policies outperform greedy and $ε$-greedy baselines under dynamics perturbations.

math.OC↗

arXiv2026

Convergence of an actor-critic gradient flow for entropy regularised MDPs in general spaces

We prove the stability and global convergence of a coupled actor-critic gradient flow for infinite-horizon and entropy-regularised Markov decision processes (MDPs) in continuous state and action space with linear function approximation under Q-function realisability. We consider a version of the actor critic gradient flow where the critic is updated using temporal difference (TD) learning while the policy is updated using a policy mirror descent method on a separate timescale. For general action spaces, the relative entropy regularizer is unbounded and thus it is not clear a priori that the actor-critc flow does not suffer from finite-time blow-up. Therefore we first demonstrate stability which in turn enables us obtain a convergence rate of the actor critic flow to the optimal regularised value function. The arguments presented show that timescale separation is crucial for stability and convergence in this setting.

math.OC↗

arXiv2026

Global linear convergence of entropy-regularized softmax policy gradient beyond tabular MDPs

We study the global convergence of policy gradient for infinite-horizon entropy-regularized Markov decision processes (MDPs) with continuous state and action spaces. We consider log-linear softmax policies with linear function approximation, which extend the tabular softmax parameterization while retaining a tractable policy class. Under $Q^π_τ$-realizability for the regularized state-action value function, we first establish a non-uniform Polyak--Łojasiewicz (PŁ) inequality. The non-uniformity arises through degeneracy of constants associated with the policy geometry, namely the Fisher information matrix or an uncentered feature covariance matrix. We then identify two feature regimes under which this non-uniform constant can be bounded along the gradient flow. For full-affine-span features, we prove radial unboundedness of the KL regularizer and show that the smallest eigenvalue of the Fisher information matrix remains bounded below by an initialization-dependent positive constant. For simplex-valued features, we prove an analogous radial unboundedness result in the subspace orthogonal to the all-ones vector and obtain a uniform lower bound for the smallest eigenvalue of the uncentered covariance matrix. These results imply global linear convergence of the regularized objective along the gradient flow, i.e. suboptimality decaying as $\mathcal{O}(e^{-Ct})$ for some $C>0$. Our analysis extends the global convergence theory of entropy-regularized softmax policy gradient beyond the tabular setting of Agarwal et al. (2020); Bhandari and Russo (2024); Mei et al. (2020).

cs.LG↗

arXiv2026

A note on convergence of Wasserstein policy optimization

Wasserstein Policy Optimization (WPO) is a recently proposed reinforcement learning algorithm that leverages Wasserstein gradient flows to optimize stochastic policies in continuous action spaces. Despite its empirical success, the theoretical convergence properties of WPO in environments with continuous state and action spaces have yet to be fully established. In this note, we argue that WPO within the framework of entropy-regularised Markov Decision Processes converges linearly. This is done by leveraging recent advances in mean-field analysis for convergence of gradient flows using log-Sobole inequalities. Assuming existence of sufficiently regular solution to the gradient flow equation we demonstrate monotonic energy dissipation along the flow and establish a local log-Sobolev inequality. Ultimately, these properties allow us to argue that the value function should converge linearly to the global optimum.

cs.LG↗

arXiv2026

Mirror descent actor-critic methods for entropy regularised MDPs in general spaces: stability and convergence

We provide theoretical guarantees for convergence of discrete-time policy mirror descent with inexact advantage functions updated using temporal difference (TD) learning for entropy regularised MDPs in Polish state and action spaces. We rigorously derive sufficient conditions under which the single-loop actor-critic scheme is stable and convergent. To weaken these conditions, we introduce a variant that performs multiple TD steps per policy update and derive an explicit lower bound on the number of TD steps required to ensure stability. Finally, we establish sub-linear convergence when the number of TD steps grows logarithmically with the number of policy updates, and linear convergence when it grows linearly under a concentrability assumption.

math.OC↗

arXiv2026

PPO in the Fisher-Rao geometry

Proximal Policy Optimization (PPO) is widely used in reinforcement learning due to its strong empirical performance, yet it lacks formal guarantees for policy improvement and convergence. PPO's clipped surrogate objective is motivated by a lower bound on linearization of the value function in flat geometry setting. We derive a tighter surrogate objective and introduce Fisher-Rao PPO (FR-PPO) by leveraging the Fisher-Rao (FR) geometry. Our scheme provides strong theoretical guarantees, including monotonic policy improvement. In the direct parametrization setting, we show that FR-PPO achieves sub-linear convergence with no dependence on action or state space dimensions, and for parametrized policies we further obtain sub-linear convergence up to the compatible function approximation error. Finally, although our primary focus is theoretical, we also demonstrate empirically that FR-PPO performs well across a range of standard reinforcement learning tasks.

cs.LG↗

arXiv2025

Mirror Descent for Stochastic Control Problems with Measure-valued Controls

This paper studies the convergence of the mirror descent algorithm for finite horizon stochastic control problems with measure-valued control processes. The control objective involves a convex regularisation function, denoted as $h$, with regularisation strength determined by the weight $τ\ge 0$. The setting covers regularised relaxed control problems. Under suitable conditions, we establish the relative smoothness and convexity of the control objective with respect to the Bregman divergence of $h$, and prove linear convergence of the algorithm for $τ=0$ and exponential convergence for $τ>0$. The results apply to common regularisers including relative entropy, $χ^2$-divergence, and entropic Wasserstein costs. This validates recent reinforcement learning heuristics that adding regularisation accelerates the convergence of gradient methods. The proof exploits careful regularity estimates of backward stochastic differential equations in the bounded mean oscillation norm.

math.OC↗

arXiv2025

Logarithmic regret in the ergodic Avellaneda-Stoikov market making model

We analyse the regret arising from learning the price sensitivity parameter $κ$ of liquidity takers in the ergodic version of the Avellaneda-Stoikov market making model. We show that a learning algorithm based on a maximum-likelihood estimator for the parameter achieves the regret upper bound of order $\ln^2 T$ in expectation. To obtain the result we need two key ingredients. The first is the twice differentiability of the ergodic constant under the misspecified parameter in the Hamilton-Jacobi-Bellman (HJB) equation with respect to $κ$, which leads to a second--order performance gap. The second is the learning rate of the regularised maximum-likelihood estimator which is obtained from concentration inequalities for Bernoulli signals. Numerical experiments confirm the convergence and the robustness of the proposed algorithm.

math.OC↗

arXiv2025

Entropy annealing for policy mirror descent in continuous time and space

Entropy regularization has been widely used in policy optimization algorithms to enhance exploration and the robustness of the optimal control; however it also introduces an additional regularization bias. This work quantifies the impact of entropy regularization on the convergence of policy gradient methods for stochastic exit time control problems. We analyze a continuous-time policy mirror descent dynamics, which updates the policy based on the gradient of an entropy-regularized value function and adjusts the strength of entropy regularization as the algorithm progresses. We prove that with a fixed entropy level, the mirror descent dynamics converges exponentially to the optimal solution of the regularized problem. We further show that when the entropy level decays at suitable polynomial rates, the annealed flow converges to the solution of the unregularized problem at a rate of $\mathcal O(1/S)$ for discrete action spaces and, under suitable conditions, at a rate of $\mathcal O(1/\sqrt{S})$ for general action spaces, with $S$ being the gradient flow running time. The technical challenge lies in analyzing the gradient flow in the infinite-dimensional space of Markov kernels for nonconvex objectives. This paper explains how entropy regularization improves policy optimization, even with the true gradient, from the perspective of convergence rate.

math.OC↗

arXiv2024

Ergodic optimal liquidations in DeFi

We address the liquidation problem arising from the credit risk management in decentralised finance (DeFi) by formulating it as an ergodic optimal control problem. In decentralised derivatives exchanges, liquidation is triggered whenever the parties fail to maintain sufficient collateral for their open positions. Consequently, effectively managing and liquidating disposal of positions accrued through liquidations is a critical concern for decentralised derivatives exchanges. By simplifying the model (linear temporary and permanent price impacts, simplified cash balance dynamics), we derive the closed-form solutions for the optimal liquidation strategies, which balance immediate executions with the temporary and permanent price impacts, and the optimal long-term average reward. Numerical simulations further highlight the effectiveness of the proposed optimal strategy and demonstrate that the simplified model closely approximates the original market environment. Finally, we provide the method for calibrating the parameters in the model from the available data.

q-fin.TR↗

arXiv2024

Gradient Flows for Regularized Stochastic Control Problems

This paper studies stochastic control problems with the action space taken to be probability measures, with the objective penalised by the relative entropy. We identify suitable metric space on which we construct a gradient flow for the measure-valued control process, in the set of admissible controls, along which the cost functional is guaranteed to decrease. It is shown that any invariant measure of this gradient flow satisfies the Pontryagin optimality principle. If the problem we work with is sufficiently convex, the gradient flow converges exponentially fast. Furthermore, the optimal measure-valued control process admits a Bayesian interpretation which means that one can incorporate prior knowledge when solving such stochastic control problems. This work is motivated by a desire to extend the theoretical underpinning for the convergence of stochastic gradient type algorithms widely employed in the reinforcement learning community to solve control problems.

math.OC↗

arXiv2023

The Modified MSA, a Gradient Flow and Convergence

The modified Method of Successive Approximations (MSA) is an iterative scheme for approximating solutions to stochastic control problems in continuous time based on Pontryagin Optimality Principle which, starting with an initial open loop control, solves the forward equation, the backward adjoint equation and then performs a static minimization step. We observe that this is an implicit Euler scheme for a gradient flow system. We prove that appropriate interpolations of the iterates of the modified MSA converge to a gradient flow with rate $τ$. We then study the convergence of this gradient flow as time goes to infinity. In the general (non-convex) case we prove that the gradient term itself converges to zero. This is a consequence of an energy identity which shows that the optimization objective decreases along the gradient flow. Moreover, in the convex case, when Pontryagin Optimality Principle provides a sufficient condition for optimality, we prove that the optimization objective converges at rate $\tfrac{1}{S}$ to its optimal value and at exponential rate under strong convexity. The main technical difficulties lie in obtaining appropriate properties of the Hamiltonian (growth, continuity). These are obtained by utilising the theory of Bounded Mean Oscillation (BMO) martingales required for estimates on the adjoint Backward Stochastic Differential Equation (BSDE).

math.OC↗

arXiv2023

Inefficiency of CFMs: hedging perspective and agent-based simulations

We investigate whether the fee income from trades on the CFM is sufficient for the liquidity providers to hedge away the exposure to market risk. We first analyse this problem through the lens of continuous-time financial mathematics and derive an upper bound for not-arbitrage fee income that would make CFM efficient and liquidity provision fair. We then evaluate our findings by performing multi-agent simulations by varying CFM fees, market volatility, and rate of arrival of liquidity takers. We observe that, on average, fee income generated from liquidity provision is insufficient to compensate for market risk.

q-fin.MF↗

arXiv2022

Decaying derivative estimates for functions of solutions to non-autonomous SDEs

We produce uniform and decaying bounds in time for derivatives of the solution to the backwards Kolmogorov equation associated to a stochastic processes governed by a time dependent dynamics. These hold under assumptions over the integrability properties in finite time of the derivatives of the transition density associated to the process, together with the assumption of remaining close over all $[0,\infty)$, or decaying in time, to some static measure. We moreover provide examples which satisfy such a set of assumptions. Finally, the results are interpreted in the McKean-Vlasov context for monotonic coefficients by introducing an auxiliary non-autonomous stochastic process.

math.PR↗

arXiv2022

Convergence of Policy Gradient for Entropy Regularized MDPs with Neural Network Approximation in the Mean-Field Regime

We study the global convergence of policy gradient for infinite-horizon, continuous state and action space, and entropy-regularized Markov decision processes (MDPs). We consider a softmax policy with (one-hidden layer) neural network approximation in a mean-field regime. Additional entropic regularization in the associated mean-field probability measure is added, and the corresponding gradient flow is studied in the 2-Wasserstein metric. We show that the objective function is increasing along the gradient flow. Further, we prove that if the regularization in terms of the mean-field measure is sufficient, the gradient flow converges exponentially fast to the unique stationary solution, which is the unique maximizer of the regularized MDP objective. Lastly, we study the sensitivity of the value function along the gradient flow with respect to regularization parameters and the initial condition. Our results rely on the careful analysis of the non-linear Fokker-Planck-Kolmogorov equation and extend the pioneering work of Mei et al. 2020 and Agarwal et al. 2020, which quantify the global convergence rate of policy gradient for entropy-regularized MDPs in the tabular setting.

math.OC↗

arXiv2021

Mean-Field Neural ODEs via Relaxed Optimal Control

We develop a framework for the analysis of deep neural networks and neural ODE models that are trained with stochastic gradient algorithms. We do that by identifying the connections between control theory, deep learning and theory of statistical sampling. We derive Pontryagin's optimality principle and study the corresponding gradient flow in the form of Mean-Field Langevin dynamics (MFLD) for solving relaxed data-driven control problems. Subsequently, we study uniform-in-time propagation of chaos of time-discretised MFLD. We derive explicit convergence rate in terms of the learning rate, the number of particles/model parameters and the number of iterations of the gradient algorithm. In addition, we study the error arising when using a finite training data set and thus provide quantitive bounds on the generalisation error. Crucially, the obtained rates are dimension-independent. This is possible by exploiting the regularity of the model with respect to the measure over the parameter space.

math.PR↗