arXiv subjects

Shangke Lyu

Publications and source records attributed to Shangke Lyu.

21 records · Page 2Linked to original sources

arXiv2025

GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation

With the rapid development of embodied artificial intelligence, significant progress has been made in vision-language-action (VLA) models for general robot decision-making. However, the majority of existing VLAs fail to account for the inevitable external perturbations encountered during deployment. These perturbations introduce unforeseen state information to the VLA, resulting in inaccurate actions and consequently, a significant decline in generalization performance. The classic internal model control (IMC) principle demonstrates that a closed-loop system with an internal model that includes external input signals can accurately track the reference input and effectively offset the disturbance. We propose a novel closed-loop VLA method GEVRM that integrates the IMC principle to enhance the robustness of robot visual manipulation. The text-guided video generation model in GEVRM can generate highly expressive future visual planning goals. Simultaneously, we evaluate perturbations by simulating responses, which are called internal embeddings and optimized through prototype contrastive learning. This allows the model to implicitly infer and distinguish perturbations from the external environment. The proposed GEVRM achieves state-of-the-art performance on both standard and perturbed CALVIN benchmarks and shows significant improvements in realistic robot tasks.

cs.RO↗

arXiv2024

GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robot

Multi-task robot learning holds significant importance in tackling diverse and complex scenarios. However, current approaches are hindered by performance issues and difficulties in collecting training datasets. In this paper, we propose GeRM (Generalist Robotic Model). We utilize offline reinforcement learning to optimize data utilization strategies to learn from both demonstrations and sub-optimal data, thus surpassing the limitations of human demonstrations. Thereafter, we employ a transformer-based VLA network to process multi-modal inputs and output actions. By introducing the Mixture-of-Experts structure, GeRM allows faster inference speed with higher whole model capacity, and thus resolves the issue of limited RL parameters, enhancing model performance in multi-task learning while controlling computational costs. Through a series of experiments, we demonstrate that GeRM outperforms other methods across all tasks, while also validating its efficiency in both training and inference processes. Additionally, we uncover its potential to acquire emergent skills. Additionally, we contribute the QUARD-Auto dataset, collected automatically to support our training approach and foster advancements in multi-task quadruped robot learning. This work presents a new paradigm for reducing the cost of collecting robot data and driving progress in the multi-task learning community. You can reach our project and video through the link: https://songwxuan.github.io/GeRM/ .

cs.RO↗

arXiv2017

Dynamic Modularity Approach to Adaptive Inner/Outer Loop Control of Robotic Systems

Modern applications of robotics typically involve a robot control system with an inner PI (proportional-integral) or PID (proportional-integral-derivative) control loop and an outer user-specified control loop. The existing outer loop controllers, however, do not take into consideration the dynamic effects of robots and their effectiveness relies on the ad hoc assumption that the inner PI or PID control loop is fast enough, and other torque-based control algorithms cannot be implemented in robotics with closed architecture. This paper investigates the adaptive control of robotic systems with an inner/outer loop structure, taking into full account the effects of the dynamics and the system uncertainties, and both the task-space control and joint-space control are considered. We propose a dynamic modularity approach to resolve this issue, and a class of adaptive outer loop control schemes is proposed and their role is to dynamically generate the joint velocity (or position) command for the low-level joint servoing loop. Without relying on the ad hoc assumption that the joint servoing is fast enough or the modification of the low-level joint controller structure, we rigorously show that the proposed outer loop controllers can ensure the stability and convergence of the closed-loop system. We also propose the outer loop versions of several standard joint-space direct/composite adaptive controllers for rigid or flexible-joint robots, and a promising conclusion may be that most torque-based adaptive controllers for robots can be designed to fit the inner/outer loop structure by using the new definition of the joint velocity (or position) command. Simulation results are provided to show the performance of various adaptive outer loop controllers, using a three-DOF (degree-of-freedom) manipulator, and experiment results using the UR10 robotic system are also presented.

eess.SY↗