arXiv subjects

Reza Asad

Publications and source records attributed to Reza Asad.

5 recordsLinked to original sources

arXiv2026

Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes

Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and construct "implicit" policies without explicit parameterization. Such policies are computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable $\textit{logit-matching}$ regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that the resulting algorithm achieves $\widetilde{\mathcal{O}}(\epsilon^{-4})$ and $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity in the on-policy and off-policy setting, respectively. Our results match prior theoretical work in achieving the state-of-the-art sample complexity, while our algorithm is more aligned with practice.

cs.LG↗

arXiv2025

Dissecting Discrete Soft Actor-Critic: Limitations and Principled Alternatives

While Soft Actor-Critic (SAC) is highly effective in continuous control, its discrete counterpart (DSAC) performs poorly on challenging discrete-action domains such as Atari. Consequently, starting from DSAC, we revisit the design of actor-critic methods in this setting. First, we determine that the coupling between the actor and critic entropy is the primary reason behind the poor performance of DSAC. We demonstrate that by merely decoupling these components, DSAC's performance significantly improves. Motivated by this insight, we introduce a flexible off-policy actor-critic framework that subsumes DSAC as a special case and yields novel objectives. Our framework allows using an m-step Bellman operator for the critic update, and instantiates the actor objective by combining standard policy optimization methods with entropy regularization. Theoretically, we prove that the proposed methods can guarantee convergence to the optimal regularized value function in the tabular setting, generalizing the results in prior work. Empirically, we evaluate the proposed objectives on standard Atari games. Our ablations indicate that, unlike DSAC, these objectives, including novel ones, perform robustly even without entropy regularization or explicit exploration mechanisms.

cs.LG↗

arXiv2024

Fast Convergence of Softmax Policy Mirror Ascent

Natural policy gradient (NPG) is a common policy optimization algorithm and can be viewed as mirror ascent in the space of probabilities. Recently, Vaswani et al. [2021] introduced a policy gradient method that corresponds to mirror ascent in the dual space of logits. We refine this algorithm, removing its need for a normalization across actions and analyze the resulting method (referred to as SPMA). For tabular MDPs, we prove that SPMA with a constant step-size matches the linear convergence of NPG and achieves a faster convergence than constant step-size (accelerated) softmax policy gradient. To handle large state-action spaces, we extend SPMA to use a log-linear policy parameterization. Unlike that for NPG, generalizing SPMA to the linear function approximation (FA) setting does not require compatible function approximation. Unlike MDPO, a practical generalization of NPG, SPMA with linear FA only requires solving convex softmax classification problems. We prove that SPMA achieves linear convergence to the neighbourhood of the optimal value function. We extend SPMA to handle non-linear FA and evaluate its empirical performance on the MuJoCo and Atari benchmarks. Our results demonstrate that SPMA consistently achieves similar or better performance compared to MDPO, PPO and TRPO.

cs.LG↗

arXiv2020

Steiner symmetrization along a certain equidistributed sequence of directions

This note reports the results of an undergraduate research project from the year 2013-14, concerning the convergence of iterated Steiner symmetrizations in the plane. The directions of symmetrization are chosen according to the Van der Corput sequence, a classical example of a sequence that is equidistributed on the unit circle with low discrepancy. It is shown here that the resulting iteration of Steiner symmetrizations converges to the symmetric decreasing rearrangement. The proof exploits the self-similarity of the sequence of angular increments, using the technique of competing symmetries.

math.MG↗

arXiv2011

Embedded Eigenvalues and the Nonlinear Schrodinger Equation

A common challenge to proving asymptotic stability of solitary waves is understanding the spectrum of the operator associated with the linearized flow. The existence of eigenvalues can inhibit the dispersive estimates key to proving stability. Following the work of Marzuola & Simpson, we prove the absence of embedded eigenvalues for a collection of nonlinear Schrodinger equations, including some one and three dimensional supercritical equations, and the three dimensional cubic-quintic equation. Our results also rule out nonzero eigenvalues within the spectral gap and, in 3D, endpoint resonances. The proof is computer assisted as it depends on the sign of certain inner products which do not readily admit analytic representations. Our source code is available for verification at http://www.math.toronto.edu/simpson/files/spec_prop_asad_simpson_code.zip.

math.SP↗