arXiv subjects

Dandan Zhang

Publications and source records attributed to Dandan Zhang.

At least 55 records · Page 3Linked to original sources

arXiv2025

TacEva: A Performance Evaluation Framework For Vision-Based Tactile Sensors

Vision-Based Tactile Sensors (VBTSs) are widely used in robotic tasks because of the high spatial resolution they offer and their relatively low manufacturing costs. However, variations in their sensing mechanisms, structural dimension, and other parameters lead to significant performance disparities between existing VBTSs. This makes it challenging to optimize them for specific tasks, as both the initial choice and subsequent fine-tuning are hindered by the lack of standardized metrics. To address this issue, TacEva is introduced as a comprehensive evaluation framework for the quantitative analysis of VBTS performance. The framework defines a set of performance metrics that capture key characteristics in typical application scenarios. For each metric, a structured experimental pipeline is designed to ensure consistent and repeatable quantification. The framework is applied to multiple VBTSs with distinct sensing mechanisms, and the results demonstrate its ability to provide a thorough evaluation of each design and quantitative indicators for each performance dimension. This enables researchers to pre-select the most appropriate VBTS on a task by task basis, while also offering performance-guided insights into the optimization of VBTS design. A list of existing VBTS evaluation methods and additional evaluations can be found on our website: https://stevenoh2003.github.io/TacEva/

cs.RO↗

arXiv2025

Stability Analysis for Stochastic Hybrid Inclusions

Stochastic hybrid inclusions (SHIs) address situations with the stochastic continuous evolution in a stochastic differential inclusions and random jumps in the difference inclusions due to the forced (the state reaching a boundary in the state space) and/or spontaneous (the state vector may occur spontaneously) transitions. An obvious characteristic of SHIs is the non-uniqueness of random solutions, which can be ensured by the mild regularity conditions, as well as nominal robustness. Basic sufficient conditions for stability/recurrence in probability are usually expressed based on different types of Lyapunov functions, including Lagrange/Lyapunov/Lyapunov-Forster functions respectively for Lagrange/Lyapunov/asymptotical stability in probability and Foster/Lagrange-Forster functions for recurrence, (weaker) relaxed Lyapunov-based sufficient conditions including Matrosov-Foster functions and the stochastic invariance principle, as well as Lyapunov-based necessary and sufficient conditions for asymptotical stability in probability or recurrence (i.e.,converse theorems), etc. The converse theorems involving smooth Lyapunov functions are guaranteed by the sequential compactness and thus robustness. In addition, the uniformity property and causality are analyzed for the stabilities in probability. Hence, serving as a partial roadmap for the theoretical development of SHIs, also serving as inspiration, we anticipate that many of the open questions, including the prediction problem, the filtering problem and the control problem, will be resolved based on the techniques of SHIs.

eess.SY↗

arXiv2025

Attitude Control of Rigid Bodies: A Survey of Representations, Topological Obstructions, and Stabilization Techniques

This paper reviews the attitude control problems for rigid-body systems, starting from the attitude representation for rigid body kinematics. Highly redundant rotation matrix defines the attitude orientation globally and uniquely by 9 parameters, which is the most fundamental one, without any singularities; minimum 3-parameter Euler angles or (modified) Rodrigues parameters define the attitude orientation neither globally nor uniquely, but the former exhibits kinematical singularity and Gimbal lock, while the latter two exhibit geometrical singularity; once-redundant axis-angle or unit quaternion globally define the attitude rotation but not uniquely using 4 parameters, but the former is not appropriate to define very small or very large rotations, while the latter shows unwinding phenomenon despite of the reduced computation burden. In addition, we explore the relationships among those attitude representations, including the connections among Gimbal lock, unwinding phenomenon and a nowhere dense set of zero Lebesgue measure. Based on attitude representations, we analyze different attitude control laws, almost global control and global attitude control, nominal and general robustness, as well as the technique tools.

eess.SY↗

arXiv2025

Physics-Informed Machine Learning with Adaptive Grids for Optical Microrobot Depth Estimation

Optical microrobots actuated by optical tweezers (OT) offer great potential for biomedical applications such as cell manipulation and microscale assembly. These tasks demand accurate three-dimensional perception to ensure precise control in complex and dynamic biological environments. However, the transparent nature of microrobots and low-contrast microscopic imaging challenge conventional deep learning methods, which also require large annotated datasets that are costly to obtain. To address these challenges, we propose a physics-informed, data-efficient framework for depth estimation of optical microrobots. Our method augments convolutional feature extraction with physics-based focus metrics, such as entropy, Laplacian of Gaussian, and gradient sharpness, calculated using an adaptive grid strategy. This approach allocates finer grids over microrobot regions and coarser grids over background areas, enhancing depth sensitivity while reducing computational complexity. We evaluate our framework on multiple microrobot types and demonstrate significant improvements over baseline models. Specifically, our approach reduces mean squared error (MSE) by over 60% and improves the coefficient of determination (R^2) across all test cases. Notably, even when trained on only 20% of the available data, our model outperforms ResNet50 trained on the full dataset, highlighting its robustness under limited data conditions. Our code is available at: https://github.com/LannWei/CBS2025.

cs.RO↗

arXiv2025

Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models

Spatial reasoning is a core component of human cognition, enabling individuals to perceive, comprehend, and interact with the physical world. It relies on a nuanced understanding of spatial structures and inter-object relationships, serving as the foundation for complex reasoning and decision-making. To investigate whether current vision-language models (VLMs) exhibit similar capability, we introduce Jigsaw-Puzzles, a novel benchmark consisting of 1,100 carefully curated real-world images with high spatial complexity. Based on this dataset, we design five tasks to rigorously evaluate VLMs' spatial perception, structural understanding, and reasoning capabilities, while deliberately minimizing reliance on domain-specific knowledge to better isolate and assess the general spatial reasoning capability. We conduct a comprehensive evaluation across 24 state-of-the-art VLMs. The results show that even the strongest model, Gemini-2.5-Pro, achieves only 77.14% overall accuracy and performs particularly poorly on the Order Generation task, with only 30.00% accuracy, far below the performance exceeding 90% achieved by human participants. This persistent gap underscores the need for continued progress, positioning Jigsaw-Puzzles as a challenging and diagnostic benchmark for advancing spatial reasoning research in VLMs. Our project page is at https://zesen01.github.io/jigsaw-puzzles.

cs.AI↗

arXiv2025

Research Challenges and Progress in the End-to-End V2X Cooperative Autonomous Driving Competition

With the rapid advancement of autonomous driving technology, vehicle-to-everything (V2X) communication has emerged as a key enabler for extending perception range and enhancing driving safety by providing visibility beyond the line of sight. However, integrating multi-source sensor data from both ego-vehicles and infrastructure under real-world constraints, such as limited communication bandwidth and dynamic environments, presents significant technical challenges. To facilitate research in this area, we organized the End-to-End Autonomous Driving through V2X Cooperation Challenge, which features two tracks: cooperative temporal perception and cooperative end-to-end planning. Built on the UniV2X framework and the V2X-Seq-SPD dataset, the challenge attracted participation from over 30 teams worldwide and established a unified benchmark for evaluating cooperative driving systems. This paper describes the design and outcomes of the challenge, highlights key research problems including bandwidth-aware fusion, robust multi-agent planning, and heterogeneous sensor integration, and analyzes emerging technical trends among top-performing solutions. By addressing practical constraints in communication and data fusion, the challenge contributes to the development of scalable and reliable V2X-cooperative autonomous driving systems.

cs.RO↗

arXiv2025

TypeTele: Releasing Dexterity in Teleoperation by Dexterous Manipulation Types

Dexterous teleoperation plays a crucial role in robotic manipulation for real-world data collection and remote robot control. Previous dexterous teleoperation mostly relies on hand retargeting to closely mimic human hand postures. However, these approaches may fail to fully leverage the inherent dexterity of dexterous hands, which can execute unique actions through their structural advantages compared to human hands. To address this limitation, we propose TypeTele, a type-guided dexterous teleoperation system, which enables dexterous hands to perform actions that are not constrained by human motion patterns. This is achieved by introducing dexterous manipulation types into the teleoperation system, allowing operators to employ appropriate types to complete specific tasks. To support this system, we build an extensible dexterous manipulation type library to cover comprehensive dexterous postures used in manipulation tasks. During teleoperation, we employ a MLLM (Multi-modality Large Language Model)-assisted type retrieval module to identify the most suitable manipulation type based on the specific task and operator commands. Extensive experiments of real-world teleoperation and imitation learning demonstrate that the incorporation of manipulation types significantly takes full advantage of the dexterous robot's ability to perform diverse and complex tasks with higher success rates.

cs.RO↗

arXiv2025

A Stochastic Hybrid Approach to Decentralized Networked Control: Stochastic Network Delays and Poisson Pulsing Attacks

By designing the decentralized time-regularized (Zeno-free) event-triggered strategies for the state-feedback control law, this paper considers the stochastic stabilization of a class of networked control systems, where two sources of randomness exist in multiple decentralized networks that operate asynchronously and independently: the communication channels are constrained by the stochastic network delays and also by Poisson pulsing denial-of-service (Pp-DoS) attacks. The time delay in the network denotes the length from a transmission instant to the corresponding update instant, and is supposed to be a continuous random variable subject to certain continuous probability distribution; while the attacks' cardinal number is a discrete random variable supposed to be subject to Poisson distribution, so the inter-attack time, i.e., the time between two consecutive attack instants, is subject to exponential distribution. The considered system is modeled as a stochastic hybrid formalism, where the randomness enters through the jump map into the reset value (the inter-attack time directly related) of each triggered strategy. By only sampling/transmitting state measurements when needed and simultaneously by taking the specific medium access protocols into account, the designed event-triggered strategies are synthesized in a state-based and decentralized form, which are robust (tolerable well) to stochastic network delays, under different tradeoff-conditions between the minimum inter-event times, maximum allowable delays (i.e., potentially tolerable delays) and the frequencies of attacks. Using stochastic hybrid tools to combine attack-active parts with attack-over parts, the designed triggered strategies, if designed well according to the actual system needs, can tolerate (be resilient to) the Pp-DoS attacks and stochastic network delays without jeopardizing the stability and Zeno-freeness.

eess.SY↗

arXiv2025

Interactive OT Gym: A Reinforcement Learning-Based Interactive Optical tweezer (OT)-Driven Microrobotics Simulation Platform

Optical tweezers (OT) offer unparalleled capabilities for micromanipulation with submicron precision in biomedical applications. However, controlling conventional multi-trap OT to achieve cooperative manipulation of multiple complex-shaped microrobots in dynamic environments poses a significant challenge. To address this, we introduce Interactive OT Gym, a reinforcement learning (RL)-based simulation platform designed for OT-driven microrobotics. Our platform supports complex physical field simulations and integrates haptic feedback interfaces, RL modules, and context-aware shared control strategies tailored for OT-driven microrobot in cooperative biological object manipulation tasks. This integration allows for an adaptive blend of manual and autonomous control, enabling seamless transitions between human input and autonomous operation. We evaluated the effectiveness of our platform using a cell manipulation task. Experimental results show that our shared control system significantly improves micromanipulation performance, reducing task completion time by approximately 67% compared to using pure human or RL control alone and achieving a 100% success rate. With its high fidelity, interactivity, low cost, and high-speed simulation capabilities, Interactive OT Gym serves as a user-friendly training and testing environment for the development of advanced interactive OT-driven micromanipulation systems and control algorithms. For more details on the project, please see our website https://sites.google.com/view/otgym

cs.RO↗

arXiv2025

MagicGripper: A Multimodal Sensor-Integrated Gripper for Contact-Rich Robotic Manipulation

Contact-rich manipulation in unstructured environments demands precise, multimodal perception to enable robust and adaptive control. Vision-based tactile sensors (VBTSs) have emerged as an effective solution; however, conventional VBTSs often face challenges in achieving compact, multi-modal functionality due to hardware constraints and algorithmic complexity. In this work, we present MagicGripper, a multimodal sensor-integrated gripper designed for contact-rich robotic manipulation. Building on our prior design, MagicTac, we develop a compact variant, mini-MagicTac, which features a three-dimensional, multi-layered grid embedded in a soft elastomer. MagicGripper integrates mini-MagicTac, enabling high-resolution tactile feedback alongside proximity and visual sensing within a compact, gripper-compatible form factor. We conduct a thorough evaluation of mini-MagicTac's performance, demonstrating its capabilities in spatial resolution, contact localization, and force regression. We also assess its robustness across manufacturing variability, mechanical deformation, and sensing performance under real-world conditions. Furthermore, we validate the effectiveness of MagicGripper through three representative robotic tasks: a teleoperated assembly task, a contact-based alignment task, and an autonomous robotic grasping task. Across these experiments, MagicGripper exhibits reliable multimodal perception, accurate force estimation, and high adaptability to challenging manipulation scenarios. Our results highlight the potential of MagicGripper as a practical and versatile tool for embodied intelligence in complex, contact-rich environments.

cs.RO↗

arXiv2025

A Dataset and Benchmarks for Deep Learning-Based Optical Microrobot Pose and Depth Perception

Optical microrobots, manipulated via optical tweezers (OT), have broad applications in biomedicine. However, reliable pose and depth perception remain fundamental challenges due to the transparent or low-contrast nature of the microrobots, as well as the noisy and dynamic conditions of the microscale environments in which they operate. An open dataset is crucial for enabling reproducible research, facilitating benchmarking, and accelerating the development of perception models tailored to microscale challenges. Standardised evaluation enables consistent comparison across algorithms, ensuring objective benchmarking and facilitating reproducible research. Here, we introduce the OpTical MicroRobot dataset (OTMR), the first publicly available dataset designed to support microrobot perception under the optical microscope. OTMR contains 232,881 images spanning 18 microrobot types and 176 distinct poses. We benchmarked the performance of eight deep learning models, including architectures derived via neural architecture search (NAS), on two key tasks: pose classification and depth regression. Results indicated that Vision Transformer (ViT) achieve the highest accuracy in pose classification, while depth regression benefits from deeper architectures. Additionally, increasing the size of the training dataset leads to substantial improvements across both tasks, highlighting OTMR's potential as a foundational resource for robust and generalisable microrobot perception in complex microscale environments.

cs.RO↗

arXiv2025

Multi-Agent Systems for Robotic Autonomy with LLMs

Since the advent of Large Language Models (LLMs), various research based on such models have maintained significant academic attention and impact, especially in AI and robotics. In this paper, we propose a multi-agent framework with LLMs to construct an integrated system for robotic task analysis, mechanical design, and path generation. The framework includes three core agents: Task Analyst, Robot Designer, and Reinforcement Learning Designer. Outputs are formatted as multimodal results, such as code files or technical reports, for stronger understandability and usability. To evaluate generalizability comparatively, we conducted experiments with models from both GPT and DeepSeek. Results demonstrate that the proposed system can design feasible robots with control strategies when appropriate task inputs are provided, exhibiting substantial potential for enhancing the efficiency and accessibility of robotic system development in research and industrial applications.

cs.RO↗

arXiv2025

Evaluating Molecule Synthesizability via Retrosynthetic Planning and Reaction Prediction

A significant challenge in wet lab experiments with current drug design generative models is the trade-off between pharmacological properties and synthesizability. Molecules predicted to have highly desirable properties are often difficult to synthesize, while those that are easily synthesizable tend to exhibit less favorable properties. As a result, evaluating the synthesizability of molecules in general drug design scenarios remains a significant challenge in the field of drug discovery. The commonly used synthetic accessibility (SA) score aims to evaluate the ease of synthesizing generated molecules, but it falls short of guaranteeing that synthetic routes can actually be found. Inspired by recent advances in top-down synthetic route generation and forward reaction prediction, we propose a new, data-driven metric to evaluate molecule synthesizability. This novel metric leverages the synergistic duality between retrosynthetic planners and reaction predictors, both of which are trained on extensive reaction datasets. To demonstrate the efficacy of our metric, we conduct a comprehensive evaluation of round-trip scores across a range of representative molecule generative models.

cs.LG↗

arXiv2025

Coarse-to-Fine Learning for Multi-Pipette Localisation in Robot-Assisted In Vivo Patch-Clamp

In vivo image-guided multi-pipette patch-clamp is essential for studying cellular interactions and network dynamics in neuroscience. However, current procedures mainly rely on manual expertise, which limits accessibility and scalability. Robotic automation presents a promising solution, but achieving precise real-time detection of multiple pipettes remains a challenge. Existing methods focus on ex vivo experiments or single pipette use, making them inadequate for in vivo multi-pipette scenarios. To address these challenges, we propose a heatmap-augmented coarse-to-fine learning technique to facilitate multi-pipette real-time localisation for robot-assisted in vivo patch-clamp. More specifically, we introduce a Generative Adversarial Network (GAN)-based module to remove background noise and enhance pipette visibility. We then introduce a two-stage Transformer model that starts with predicting the coarse heatmap of the pipette tips, followed by the fine-grained coordination regression module for precise tip localisation. To ensure robust training, we use the Hungarian algorithm for optimal matching between the predicted and actual locations of tips. Experimental results demonstrate that our method achieved > 98% accuracy within 10 μm, and > 89% accuracy within 5 μm for the localisation of multi-pipette tips. The average MSE is 2.52 μm.

eess.IV↗

arXiv2025

Deep Reinforcement Learning-Based Semi-Autonomous Control for Magnetic Micro-robot Navigation with Immersive Manipulation

Magnetic micro-robots have demonstrated immense potential in biomedical applications, such as in vivo drug delivery, non-invasive diagnostics, and cell-based therapies, owing to their precise maneuverability and small size. However, current micromanipulation techniques often rely solely on a two-dimensional (2D) microscopic view as sensory feedback, while traditional control interfaces do not provide an intuitive manner for operators to manipulate micro-robots. These limitations increase the cognitive load on operators, who must interpret limited feedback and translate it into effective control actions. To address these challenges, we propose a Deep Reinforcement Learning-Based Semi-Autonomous Control (DRL-SC) framework for magnetic micro-robot navigation in a simulated microvascular system. Our framework integrates Mixed Reality (MR) to facilitate immersive manipulation of micro-robots, thereby enhancing situational awareness and control precision. Simulation and experimental results demonstrate that our approach significantly improves navigation efficiency, reduces control errors, and enhances the overall robustness of the system in simulated microvascular environments.

cs.RO↗

arXiv2025

Design and Benchmarking of A Multi-Modality Sensor for Robotic Manipulation with GAN-Based Cross-Modality Interpretation

In this paper, we present the design and benchmark of an innovative sensor, ViTacTip, which fulfills the demand for advanced multi-modal sensing in a compact design. A notable feature of ViTacTip is its transparent skin, which incorporates a `see-through-skin' mechanism. This mechanism aims at capturing detailed object features upon contact, significantly improving both vision-based and proximity perception capabilities. In parallel, the biomimetic tips embedded in the sensor's skin are designed to amplify contact details, thus substantially augmenting tactile and derived force perception abilities. To demonstrate the multi-modal capabilities of ViTacTip, we developed a multi-task learning model that enables simultaneous recognition of hardness, material, and textures. To assess the functionality and validate the versatility of ViTacTip, we conducted extensive benchmarking experiments, including object recognition, contact point detection, pose regression, and grating identification. To facilitate seamless switching between various sensing modalities, we employed a Generative Adversarial Network (GAN)-based approach. This method enhances the applicability of the ViTacTip sensor across diverse environments by enabling cross-modality interpretation.

cs.RO↗

arXiv2024

Enhanced third-harmonic generation and degenerate four-wave mixing in an all-dielectric metasurfaces via Brillouin zone folding-induced bound states in the continuum

Bound states in the continuum (BICs) exhibit significant electric field confinement capabilities and have recently been employed to enhance nonlinear optics response at the nanoscale. In this study, we achieve substantial enhancement of third-harmonic generation (THG) and degenerate four-wave mixing (dFWM) by implementing Brillouin zone folding-induced BICs (BZF-BICs) in an air-hole type nonlinear metasurface. By introducing gap perturbations within the metasurface, guided modes below the light line can be folded into the light cone, resulting in three resonant modes: guided resonances (GRs), $Γ$-BICs, and BZF-BICs. Through the eigenvalue analysis and multipole decompositions, we establish their excitation conditions. With their resonantly enhanced local field, we successfully boost both THG and dFWM under $x$- and $y$- polarizations within the same metasurfaces. The simulated results indicate that the BZF-BICs provide the most significant enhancement of third-order nonlinear optical responses, with the output power of THG to 10$^{-4}$ W and dFWM output power of 10$^{-2}$ W under a moderate input power density of 1 MW/cm$^{2}$. These findings demonstrate that the BZF-BICs can offer an effective pathway for chip-scale nonlinear optical applications.

physics.optics↗

arXiv2024

TacPalm: A Soft Gripper with a Biomimetic Optical Tactile Palm for Stable Precise Grasping

Manipulating fragile objects in environments such as homes and factories requires stable and gentle grasping along with precise and safe placement. Compared to traditional rigid grippers, the use of soft grippers reduces the control complexity and the risk of damaging objects. However, it is challenging to integrate camera-based optical tactile sensing into a soft gripper without compromising the flexibility and adaptability of the fingers, while also ensuring that the precision of tactile perception remains unaffected by passive deformations of the soft structure during object contact. In this paper, we demonstrate a modular soft two-fingered gripper with a 3D-printed optical tactile sensor (the TacTip) integrated in the palm. We propose a soft-grasping strategy that includes three functions: light contact detection, grasp pose adjustment and loss-of-contact detection, so that objects of different shapes and sizes can be grasped stably and placed precisely, which we test with both artificial and household objects. By sequentially implementing these three functions, the grasp success rate progressively improves from 45% without any functions, to 59% with light contact detection, 90% with grasp pose adjustment, and 97% with loss-of-contact detection, achieving a sub-millimeter placement precision. Overall, this work demonstrates the feasibility and utility of integrating optical tactile sensors into the palm of a soft gripper, of applicability to various types of soft manipulators. The proposed grasping strategy has potential applications in areas such as fragile product processing and home assistance.

cs.RO↗