arXiv subjects

Ruoqi Zhang

Publications and source records attributed to Ruoqi Zhang.

12 recordsLinked to original sources

arXiv2026

Comparison of Dirichlet forms for stable-like random walks on groups of polynomial volume growth

In earlier works, we studied natural examples of stable-like random walks on finitely generated nilpotent groups and, more generally, on groups of polynomial volume growth. These walks are driven by measures of radial type, coordinate-wise type, or convex combinations of such measures. In the present work, we explore when two such random walks have comparable behavior, as measured by the equivalence of their associated Dirichlet forms. We develop criteria for this equivalence in terms of both geometric and algebraic features of the driving measures and the group.

math.PR↗

arXiv2025

Examples of stable-like random walks on groups of polynomial growth

We consider several families of long jump random walks on groups of polynomial volume growth which are naturally expected to have a stable-like behavior. We then prove optimal pseudo-Poincar\'e inequalities for these walks. These pseudo-Poincar\'e inequalities allow us to show that the random walks in questions indeed have a stable-like behavior and to obtain detailed estimates.

math.PR↗

arXiv2025

Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensembles

Diffusion models have shown impressive performance in capturing complex and multi-modal action distributions for game agents, but their slow inference speed prevents practical deployment in real-time game environments. While consistency models offer a promising approach for one-step generation, they often suffer from training instability and performance degradation when applied to policy learning. In this paper, we present CPQE (Consistency Policy with Q-Ensembles), which combines consistency models with Q-ensembles to address these challenges.CPQE leverages uncertainty estimation through Q-ensembles to provide more reliable value function approximations, resulting in better training stability and improved performance compared to classic double Q-network methods. Our extensive experiments across multiple game scenarios demonstrate that CPQE achieves inference speeds of up to 60 Hz -- a significant improvement over state-of-the-art diffusion policies that operate at only 20 Hz -- while maintaining comparable performance to multi-step diffusion approaches. CPQE consistently outperforms state-of-the-art consistency model approaches, showing both higher rewards and enhanced training stability throughout the learning process. These results indicate that CPQE offers a practical solution for deploying diffusion-based policies in games and other real-time applications where both multi-modal behavior modeling and rapid inference are critical requirements.

cs.AI↗

arXiv2025

Convergence in On-line Learning of Static and Dynamic Systems

The paper derives analytical expressions for the asymptotic average updating direction of the adaptive moment generation (ADAM) algorithm when applied to recursive identification of nonlinear systems. It is proved that the standard hyper-parameter setting results in the same asymptotic average updating direction as a diagonally power normalized stochastic gradient algorithm. With the internal filtering turned off, the asymptotic average updating direction is instead equivalent to that of a sign-sign stochastic gradient algorithm. Global convergence to an invariant set follows, where a subset of parameters contain those that give a correct input-output description of the system. The paper also exploits a nonlinear dynamic model to embed structure in recurrent neural networks. A Monte-Carlo simulation study validates the results.

eess.SY↗

arXiv2024

Safe Output Feedback Improvement with Baselines

In data-driven control design, an important problem is to deal with uncertainty due to limited and noisy data. One way to do this is to use a min-max approach, which aims to minimize some design criteria for the worst-case scenario. However, a strategy based on this approach can lead to overly conservative controllers. To overcome this issue, we apply the idea of baseline regret, and it is seen that minimizing the baseline regret under model uncertainty can guarantee safe controller improvement with less conservatism and variance in the resulting controllers. To exemplify the use of baseline controllers, we focus on the output feedback setting and propose a two-step control design method; first, an uncertainty set is constructed by a data-driven system identification approach based on finite impulse response models; then a control design criterion based on model reference control is used. To solve the baseline regret optimization problem efficiently, we use a convex approximation of the criterion and apply the scenario approach in optimization. The numerical examples show that the inclusion of baseline regret indeed improves the performance and reduces the variance of the resulting controller.

eess.SY↗

arXiv2024

Frobenius objects in the category of relations

We give a characterization, in terms of simplicial sets, of Frobenius objects in the category of relations. This result generalizes a result of Heunen, Contreras, and Cattaneo showing that special dagger Frobenius objects in the category of relations are in correspondence with groupoids. As an additional example, we construct a Frobenius object in the category of relations whose elements are certain cohomology classes in a compact oriented Riemannian manifold.

math.CT↗

arXiv2024

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distribution into a standard Gaussian and then samples actions conditioned on the environment state with a corresponding reverse-time SDE, like a typical diffusion policy. We show that such an SDE has a solution that we can use to calculate the log probability of the policy, yielding an entropy regularizer that improves the exploration of offline datasets. To mitigate the impact of inaccurate value functions from out-of-distribution data points, we further propose to learn the lower confidence bound of Q-ensembles for more robust policy improvement. By combining the entropy-regularized diffusion policy with Q-ensembles in offline RL, our method achieves state-of-the-art performance on most tasks in D4RL benchmarks. Code is available at https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble.

cs.LG↗

arXiv2023

Risk-sensitive Actor-free Policy via Convex Optimization

Traditional reinforcement learning methods optimize agents without considering safety, potentially resulting in unintended consequences. In this paper, we propose an optimal actor-free policy that optimizes a risk-sensitive criterion based on the conditional value at risk. The risk-sensitive objective function is modeled using an input-convex neural network ensuring convexity with respect to the actions and enabling the identification of globally optimal actions through simple gradient-following methods. Experimental results demonstrate the efficacy of our approach in maintaining effective risk control.

cs.LG↗

arXiv2023

Observer-Feedback-Feedforward Controller Structures in Reinforcement Learning

The paper proposes the use of structured neural networks for reinforcement learning based nonlinear adaptive control. The focus is on partially observable systems, with separate neural networks for the state and feedforward observer and the state feedback and feedforward controller. The observer dynamics are modelled by recurrent neural networks while a standard network is used for the controller. As discussed in the paper, this leads to a separation of the observer dynamics to the recurrent neural network part, and the state feedback to the feedback and feedforward network. The structured approach reduces the computational complexity and gives the reinforcement learning based controller an {\em understandable} structure as compared to when one single neural network is used. As shown by simulation the proposed structure has the additional and main advantage that the training becomes significantly faster. Two ways to include feedforward structure are presented, one related to state feedback control and one related to classical feedforward control. The latter method introduces further structure with a separate recurrent neural network that processes only the measured disturbance. When evaluated with simulation on a nonlinear cascaded double tank process, the method with most structure performs the best, with excellent feedforward disturbance rejection gains.

eess.SY↗

arXiv2023

Robust nonlinear set-point control with reinforcement learning

There has recently been an increased interest in reinforcement learning for nonlinear control problems. However standard reinforcement learning algorithms can often struggle even on seemingly simple set-point control problems. This paper argues that three ideas can improve reinforcement learning methods even for highly nonlinear set-point control problems: 1) Make use of a prior feedback controller to aid amplitude exploration. 2) Use integrated errors. 3) Train on model ensembles. Together these ideas lead to more efficient training, and a trained set-point controller that is more robust to modelling errors and thus can be directly deployed to real-world nonlinear systems. The claim is supported by experiments with a real-world nonlinear cascaded tank process and a simulated strongly nonlinear pH-control system.

eess.SY↗

arXiv2023

Aiding reinforcement learning for set point control

While reinforcement learning has made great improvements, state-of-the-art algorithms can still struggle with seemingly simple set-point feedback control problems. One reason for this is that the learned controller may not be able to excite the system dynamics well enough initially, and therefore it can take a long time to get data that is informative enough to learn for good control. The paper contributes by augmentation of reinforcement learning with a simple guiding feedback controller, for example, a proportional controller. The key advantage in set point control is a much improved excitation that improves the convergence properties of the reinforcement learning controller significantly. This can be very important in real-world control where quick and accurate convergence is needed. The proposed method is evaluated with simulation and on a real-world double tank process with promising results.

eess.SY↗

arXiv2020

Unital operads, monoids, monads, and bar constructions

We give a description of unital operads in a symmetric monoidal category as monoids in a monoidal category of unital $\Lambda$-sequences. This is a new variant of Kelly's old description of operads as monoids in the monoidal category of symmetric sequences. The monads associated to unital operads are the ones of interest in iterated loop space theory and factorization homology, among many other applications. Our new description of unital operads allows an illuminating comparison between the two-sided monadic bar constructions used in such applications and "classical" monoidal two-sided bar constructions. It also allows a more conceptual understanding of the scanning map central to non-abelian Poincar\'e duality in factorization homology.

math.AT↗