arXiv subjects

Xinjun Sheng

Publications and source records attributed to Xinjun Sheng.

11 recordsLinked to original sources

arXiv2026

Hand-in-the-Loop: Improving VLA Policies for Dexterous Manipulation via Seamless Hand-Arm Intervention

Vision-Language-Action (VLA) models are prone to compounding errors in dexterous manipulation, where high-dimensional action spaces and contact-rich dynamics amplify small policy deviations over long horizons. While Interactive Imitation Learning (IIL) can refine policies through human correction data, applying it to high-degree-of-freedom (DoF) robotic hands remains challenging due to a command mismatch between human teleoperation and policy execution at the intervention moment, which causes abrupt robot-hand configuration changes, or "gesture jumps". We present Hand-in-the-Loop (HandITL), a seamless human-in-the-loop intervention method that blends human corrective intent with autonomous policy execution to avoid gesture jumps during bimanual dexterous manipulation. Compared with taking over control using direct teleoperation, HandITL reduces intervention jitter by 99.8% and preserves robust post-intervention manipulation, reducing grasp failures by 87.5% and mean completion time by 19.1%. We validate HandITL on tasks requiring bimanual coordination, tool use, and fine-grained long-horizon manipulation. When used to collect correction data for policy refinement, HandITL yields policies that outperform those trained with standard teleoperation data by 19% on average across three long-horizon dexterous tasks.

cs.RO↗

arXiv2025

Hierarchical Reinforcement Learning for Articulated Tool Manipulation with Multifingered Hand

Manipulating articulated tools, such as tweezers or scissors, has rarely been explored in previous research. Unlike rigid tools, articulated tools change their shape dynamically, creating unique challenges for dexterous robotic hands. In this work, we present a hierarchical, goal-conditioned reinforcement learning (GCRL) framework to improve the manipulation capabilities of anthropomorphic robotic hands using articulated tools. Our framework comprises two policy layers: (1) a low-level policy that enables the dexterous hand to manipulate the tool into various configurations for objects of different sizes, and (2) a high-level policy that defines the tool's goal state and controls the robotic arm for object-picking tasks. We employ an encoder, trained on synthetic pointclouds, to estimate the tool's affordance states--specifically, how different tool configurations (e.g., tweezer opening angles) enable grasping of objects of varying sizes--from input point clouds, thereby enabling precise tool manipulation. We also utilize a privilege-informed heuristic policy to generate replay buffer, improving the training efficiency of the high-level policy. We validate our approach through real-world experiments, showing that the robot can effectively manipulate a tweezer-like tool to grasp objects of diverse shapes and sizes with a 70.8 % success rate. This study highlights the potential of RL to advance dexterous robotic manipulation of articulated tools.

cs.RO↗

arXiv2024

Optimized Design of A Haptic Unit for Vibrotactile Amplitude Modulation

Communicating information to users is a crucial aspect of human-machine interaction. Vibrotactile feedback encodes information into spatiotemporal vibrations, enabling users to perceive tactile sensations. It offers advantages such as lightweight, wearability, and high stability, with broad applications in sensory substitution, virtual reality, education, and healthcare. However, existing haptic unit designs lack amplitude modulation capabilities, which limits their applications. This paper proposed an optimized design of the haptic unit from the perspective of vibration amplitude modulation. A modified elastic model was developed to describe the propagation and attenuation mechanisms of vibration in the skin. Based on the model, two types of hierarchical architectural design were proposed. The design incorporated various materials arranged in multiple layers to amplify or attenuate the vibration amplitude as it traveled through the structure. An experimental platform was built to evaluate the performance of the optimized design.

cs.RO↗

arXiv2023

Farthest Streamline Sampling for the Uniform Distribution of Forearm Muscle Fiber Tracts from Diffusion Tensor Imaging

Background: Diffusion tensor imaging (DTI) has been used to characterize forearm muscle architecture. Since only uniform sampling is performed for seed points rather than fiber tracts, the tracts may be unevenly distributed in the muscle volume. Purpose: To reconstruct uniformly distributed fiber tracts in human forearm by filtering the tracts from DTI. Assessment: Farthest streamline sampling (FSS) was proposed for filtering and compared with two conventional methods, i.e., two-dimensional sampling and three-dimensional sampling. The uniform coverage performance of the methods was evaluated by streamline coverage (SC) and the coefficient of variation of streamline density (SDCV). Architectural parameters were calculated for 17 forearm muscles. Anatomical correctness was verified by 1. visually assessing the fiber orientation, 2. checking whether the architectural parameters were within physiological ranges, and 3. classifying the architectural types. Results: FSS had the highest SC (0.93+0.04) and the lowest SDCV (0.34+0.06) among the three methods (P<0.05). FSS reduced the sampling of long tracts (10% reduction in fiber length, P<0.05), and the architectural parameters were within physiological ranges (two parameters with P<0.05). The fiber orientation of the tractography was visually consistent with that of the cadaveric specimen. The architectural types of 16 muscles were correctly classified, except for the palmaris longus, which had a linear arrangement of fiber endpoints (R2=0.95+0.02, P<0.001). Data Conclusion: FSS reconstructed more muscle regions and uniformly distributed fiber tracts. The tracts were anatomically correct, indicating the validity of fiber tracts. Key Words: diffusion tensor imaging; forearm muscles; architectural properties

eess.IV↗

arXiv2022

Perching on Moving Inclined Surfaces using Uncertainty Tolerant Planner and Thrust Regulation

Quadrotors with the ability to perch on moving inclined surfaces can save energy and extend their travel distance by leveraging ground vehicles. Achieving dynamic perching places high demands on the performance of trajectory planning and terminal state accuracy in SE(3). However, in the perching process, uncertainties in target surface prediction, tracking control and external disturbances may cause trajectory planning failure or lead to unacceptable terminal errors. To address these challenges, we first propose a trajectory planner that considers adaptation to uncertainties in target prediction and tracking control. To facilitate this work, the reachable set of quadrotors' states is first analyzed. The states whose reachable sets possess the largest coverage probability for uncertainty targets, are defined as optimal waypoints. Subsequently, an approach to seek local optimal waypoints for static and moving uncertainty targets is proposed. A real-time trajectory planner based on optimized waypoints is developed accordingly. Secondly, thrust regulation is also implemented in the terminal attitude tracking stage to handle external disturbances. When a quadrotor's attitude is commanded to align with target surfaces, the thrust is optimized to minimize terminal errors. This makes the terminal position and velocity be controlled in closed-loop manner. Therefore, the resistance to disturbances and terminal accuracy is improved. Extensive simulation experiments demonstrate that our methods can improve the accuracy of terminal states under uncertainties. The success rate is approximately increased by $50\%$ compared to the two-end planner without thrust regulation. Perching on the rear window of a car is also achieved using our proposed heterogeneous cooperation system outdoors. This validates the feasibility and practicality of our methods.

cs.RO↗

arXiv2022

Conditional Generative Models for Simulation of EMG During Naturalistic Movements

Numerical models of electromyographic (EMG) signals have provided a huge contribution to our fundamental understanding of human neurophysiology and remain a central pillar of motor neuroscience and the development of human-machine interfaces. However, whilst modern biophysical simulations based on finite element methods are highly accurate, they are extremely computationally expensive and thus are generally limited to modelling static systems such as isometrically contracting limbs. As a solution to this problem, we propose a transfer learning approach, in which a conditional generative model is trained to mimic the output of an advanced numerical model. To this end, we present BioMime, a conditional generative neural network trained adversarially to generate motor unit activation potential waveforms under a wide variety of volume conductor parameters. We demonstrate the ability of such a model to predictively interpolate between a much smaller number of numerical model's outputs with a high accuracy. Consequently, the computational load is dramatically reduced, which allows the rapid simulation of EMG signals during truly dynamic and naturalistic movements.

cs.LG↗

arXiv2022

A Novel Semi-Coupled Hierarchical Motion Planning Framework for Cooperative Transportation of Multiple Mobile Manipulators

Multiple mobile manipulators show superiority in the tasks requiring mobility and dexterity compared with a single robot, especially when manipulating/transporting bulky objects. However, closed-chain of the system, redundancy of each mobile manipulator and obstacles in the environment bring challenges to the motion planning problem. In this paper, we propose a novel semi-coupled hierarchical framework (SCHF), which decomposes the problem into two semi-coupled sub-problems.To be specific, the centralized layer plans the object's motion first and then the decentralized layer independently explores the redundancy of each robot in real-time. A notable feature is that the lower bound of the redundancy constraint metric is ensured besides the closed-chain and obstacle-avoidance constraints in the centralized layer, which ensures the object's motion can be executed by each robot in the decentralized layer. Simulated results show that the success rate and time cost of SCHF outperforms the fully centralized planner and fully decoupled hierarchical planner significantly. In addition, cluttered real-world experiments also show the feasibility of the SCHF in the transportation tasks. A video clip in various scenarios can be found at https://youtu.be/Y8ZrnspIuBg.

cs.RO↗

arXiv2022

Hitchhiker: A Quadrotor Aggressively Perching on a Moving Inclined Surface Using Compliant Suction Cup Gripper

Perching on {the surface} of moving objects, like vehicles, could extend the flight {time} and range of quadrotors. Suction cups are usually adopted for {surface attachment} due to their durability and large adhesive force. To seal on {a surfaces}, suction cups {must} be aligned with {the surface} and {possess proper relative tangential velocity}. {However, quadrotors' attitude and relative velocity errors would become significant when the object surface is moving and inclined. To address this problem, we proposed a real-time trajectory planning algorithm. The time-optimal aggressive trajectory is efficiently generated through multimodal search in a dynamic time-domain. The velocity errors relative to the moving surface are alleviated.} To further adapt to the residual errors, we design a compliant gripper using self-sealing cups. Multiple cups in different directions are integrated into a wheel-like mechanism to increase the tolerance to attitude errors. The wheel mechanism also eliminates the requirement of matching the attitude and tangential velocity. {Extensive tests are conducted to perch on static and moving surfaces at various inclinations.} Results demonstrate that our proposed system enables a quadrotor to reliably perch on moving inclined surfaces (up to $1.07m/s$ and $90^\circ$) with a success rate of $70\%$ or higher. {The efficacy of the trajectory planner is also validated. Our gripper has larger adaptability to attitude errors and tangential velocities than conventional suction cup grippers.} The success rate increases by 45\% in dynamic perches.

cs.RO↗

arXiv2021

An Efficient Egocentric Regulator for Continuous Targeting Problems of the Underactuated Quadrotor

Flying robots such as the quadrotor could provide an efficient approach for medical treatment or sensor placing of wild animals. In these applications, continuously targeting the moving animal is a crucial requirement. Due to the underactuated characteristics of the quadrotor and the coupled kinematics with the animal, nonlinear optimal tracking approaches, other than smooth feedback control, are required. However, with severe nonlinearities, it would be time-consuming to evaluate control inputs, and real-time tracking may not be achieved with generic optimizers onboard. To tackle this problem, a novel efficient egocentric regulation approach with high computational efficiency is proposed in this paper. Specifically, it directly formulates the optimal tracking problem in an egocentric manner regarding the quadrotor's body coordinates. Meanwhile, the nonlinearities of the system are peeled off through a mapping of the feedback states as well as control inputs, between the inertial and body coordinates. In this way, the proposed efficient egocentric regulator only requires solving a quadratic performance objective with linear constraints and then generate control inputs analytically. Comparative simulations and mimic biological experiment are carried out to verify the effectiveness and computational efficiency. Results demonstrate that the proposed control approach presents the highest and stablest computational efficiency than generic optimizers on different platforms. Particularly, on a commonly utilized onboard computer, our method can compute the control action in approximately 0.3 ms, which is on the order of 350 times faster than that of generic nonlinear optimizers, establishing a control frequency around 3000 Hz.

cs.RO↗

arXiv2020

An Active Sense and Avoid System for Flying Robots in Dynamic Environments

This paper investigates a novel active-sensing-based obstacle avoidance paradigm for flying robots in dynamic environments. Instead of fusing multiple sensors to enlarge the field of view (FOV), we introduce an alternative approach that utilizes a stereo camera with an independent rotational DOF to sense the obstacles actively. In particular, the sensing direction is planned heuristically by multiple objectives, including tracking dynamic obstacles, observing the heading direction, and exploring the previously unseen area. With the sensing result, a flight path is then planned based on real-time sampling and uncertainty-aware collision checking in the state space, which constitutes an active sense and avoid (ASAA) system. Experiments in both simulation and the real world demonstrate that this system can well cope with dynamic obstacles and abrupt goal direction changes. Since only one stereo camera is utilized, this system provides a low-cost and effective approach to overcome the FOV limitation in visual navigation.

cs.RO↗

arXiv2019

Learning to Navigate from Simulation via Spatial and Semantic Information Synthesis with Noise Model Embedding

While training an end-to-end navigation network in the real world is usually of high cost, simulation provides a safe and cheap environment in this training stage. However, training neural network models in simulation brings up the problem of how to effectively transfer the model from simulation to the real world (sim-to-real). In this work, we regard the environment representation as a crucial element in this transfer process and propose a visual information pyramid (VIP) model to systematically investigate a practical environment representation. A novel representation composed of spatial and semantic information synthesis is then established accordingly, where noise model embedding is particularly considered. To explore the effectiveness of this representation, we compared the performance with representations popularly used in the literature in both simulated and real-world scenarios. Results suggest that our environment representation stands out. Furthermore, an analysis on the feature map is implemented to investigate the effectiveness through inner reaction, which could be irradiative for future researches on end-to-end navigation.

cs.RO↗