arXiv subjects

Rüdiger Dillmann

Publications and source records attributed to Rüdiger Dillmann.

At least 19 recordsLinked to original sources

arXiv2026

A Practical Framework of Key Performance Indicators for Multi-Robot Lunar and Planetary Field Tests

Robotic prospecting for critical resources on the Moon, such as ilmenite, rare earth elements, and water ice, requires robust exploration methods given the diverse terrain and harsh environmental conditions. Although numerous analog field trials address these goals, comparing their results remains challenging because of differences in robot platforms and experimental setups. These missions typically assess performance using selected, scenario-specific engineering metrics that fail to establish a clear link between field performance and science-driven objectives. In this paper, we address this gap by deriving a structured framework of KPI from three realistic multi-robot lunar scenarios reflecting scientific objectives and operational constraints. Our framework emphasizes scenario-dependent priorities in efficiency, robustness, and precision, and is explicitly designed for practical applicability in field deployments. We validated the framework in a multi-robot field test and found it practical and easy to apply for efficiency- and robustness-related KPI, whereas precision-oriented KPI require reliable ground-truth data that is not always feasible to obtain in outdoor analog environments. Overall, we propose this framework as a common evaluation standard enabling consistent, goal-oriented comparison of multi-robot field trials and supporting systematic development of robotic systems for future planetary exploration.

cs.RO↗

arXiv2026

Constraint Manifold Exploration for Efficient Continuous Coverage Estimation

Many automated manufacturing processes rely on industrial robot arms to move process-specific tools along workpiece surfaces. In applications like grinding, sanding, spray painting, or inspection, they need to cover a workpiece fully while keeping their tools perpendicular to its surface. While there are approaches to generate trajectories for these applications, there are no sufficient methods for analyzing the feasibility of full surface coverage. This work proposes a sampling-based approach for continuous coverage estimation that explores reachable surface regions in the configuration space. We define an extended ambient configuration space that allows for the representation of tool position and orientation constraints. A continuation-based approach is used to explore it using two different sampling strategies. A thorough evaluation across different kinematics and environments analyzes their runtime and efficiency. This validates our ability to accurately and efficiently calculate surface coverage for complex surfaces in complicated environments.

cs.RO↗

arXiv2026

MOSAIC: Modular Scalable Autonomy for Intelligent Coordination of Heterogeneous Robotic Teams

Mobile robots have become indispensable for exploring hostile environments, such as in space or disaster relief scenarios, but often remain limited to teleoperation by a human operator. This restricts the deployment scale and requires near-continuous low-latency communication between the operator and the robot. We present MOSAIC: a scalable autonomy framework for multi-robot scientific exploration using a unified mission abstraction based on Points of Interest (POIs) and multiple layers of autonomy, enabling supervision by a single operator. The framework dynamically allocates exploration and measurement tasks based on each robot's capabilities, leveraging team-level redundancy and specialization to enable continuous operation. We validated the framework in a space-analog field experiment emulating a lunar prospecting scenario, involving a heterogeneous team of five robots and a single operator. Despite the complete failure of one robot during the mission, the team completed 82.3% of assigned tasks at an Autonomy Ratio of 86%, while the operator workload remained at only 78.2%. These results demonstrate that the proposed framework enables robust, scalable multi-robot scientific exploration with limited operator intervention. We further derive practical lessons learned in robot interoperability, networking architecture, team composition, and operator workload management to inform future multi-robot exploration missions.

cs.RO↗

arXiv2024

Behavior Tree Capabilities for Dynamic Multi-Robot Task Allocation with Heterogeneous Robot Teams

While individual robots are becoming increasingly capable, with new sensors and actuators, the complexity of expected missions increased exponentially in comparison. To cope with this complexity, heterogeneous teams of robots have become a significant research interest in recent years. Making effective use of the robots and their unique skills in a team is challenging. Dynamic runtime conditions often make static task allocations infeasible, therefore requiring a dynamic, capability-aware allocation of tasks to team members. To this end, we propose and implement a system that allows a user to specify missions using Bheavior Trees (BTs), which can then, at runtime, be dynamically allocated to the current robot team. The system allows to statically model an individual robot's capabilities within our ros_bt_py BT framework. It offers a runtime auction system to dynamically allocate tasks to the most capable robot in the current team. The system leverages utility values and pre-conditions to ensure that the allocation improves the overall mission execution quality while preventing faulty assignments. To evaluate the system, we simulated a find-and-decontaminate mission with a team of three heterogeneous robots and analyzed the utilization and overall mission times as metrics. Our results show that our system can improve the overall effectiveness of a team while allowing for intuitive mission specification and flexibility in the team composition.

cs.RO↗

arXiv2024

Efficient Gesture Recognition on Spiking Convolutional Networks Through Sensor Fusion of Event-Based and Depth Data

As intelligent systems become increasingly important in our daily lives, new ways of interaction are needed. Classical user interfaces pose issues for the physically impaired and are partially not practical or convenient. Gesture recognition is an alternative, but often not reactive enough when conventional cameras are used. This work proposes a Spiking Convolutional Neural Network, processing event- and depth data for gesture recognition. The network is simulated using the open-source neuromorphic computing framework LAVA for offline training and evaluation on an embedded system. For the evaluation three open source data sets are used. Since these do not represent the applied bi-modality, a new data set with synchronized event- and depth data was recorded. The results show the viability of temporal encoding on depth information and modality fusion, even on differently encoded data, to be beneficial to network performance and generalization capabilities.

cs.RO↗

arXiv2023

EfficientPPS: Part-aware Panoptic Segmentation of Transparent Objects for Robotic Manipulation

The use of autonomous robots for assistance tasks in hospitals has the potential to free up qualified staff and im-prove patient care. However, the ubiquity of deformable and transparent objects in hospital settings poses signif-icant challenges to vision-based perception systems. We present EfficientPPS, a neural architecture for part-aware panoptic segmentation that provides robots with semantically rich visual information for grasping and ma-nipulation tasks. We also present an unsupervised data collection and labelling method to reduce the need for human involvement in the training process. EfficientPPS is evaluated on a dataset containing real-world hospital objects and demonstrated to be robust and efficient in grasping transparent transfusion bags with a collaborative robot arm.

cs.RO↗

arXiv2023

HoLLiE C -- A Multifunctional Bimanual Mobile Robot Supporting Versatile Care Applications

Care robotics as a research field has developed a lot in recent years, driven by the rapidly increasing need for it. However, these technologies are mostly limited to a very concrete and usually relatively simple use case. The bimanual robot House of Living Labs intelligent Escort (HoLLiE) includes an omnidirectional mobile platform. This paper presents how HoLLiE is adapted, by flexible software and hardware modules, for different care applications. The design goal of HoLLiE was to be human-like but abstract enough to ensure a high level of acceptance, which is very advantageous for its use in hospitals. After a short retrospect of previous generations of HoLLiE, it is highlighted how the current version is equipped with a variety of additional sensors and actuators to allow a wide range of possible applications. Then, the software stack of HoLLiE is depicted, with the focus on navigation and force sensitive intention recognition.

cs.RO↗

arXiv2023

Implicit Shape Model Trees: Recognition of 3-D Indoor Scenes and Prediction of Object Poses for Mobile Robots

We present an approach for mobile robots to recognize scenes in object arrangements distributed across cluttered environments. Recognition is enabled by intertwining the robot's search for objects and the assignment of found objects to scenes. Our scene model called "Implicit Shape Model (ISM) trees" allows these two tasks to be solved jointly. This article presents novel algorithms for ISM trees to recognize scenes and predict poses of searched objects. We define scenes as object sets in which some objects are connected via 3-D spatial relations. In previous work, we recognized scenes with single ISMs. However, single ISMs are prone to false positives. As a remedy, we have developed ISM trees, a hierarchical model consisting of multiple ISMs. This article contributes a recognition algorithm that now enables the use of ISM trees for scene recognition. ISM trees should be ideally generated from human demonstrations of object arrangements. As a suitable algorithm was not available, we introduce such a generation algorithm. In line with the active vision paradigm, we combined scene recognition and object search in previous work. However, an efficient algorithm was lacking to make this combination effective. Physical experiments show that this is now overcome with a new algorithm achieving efficient combination through predicted object poses.

cs.RO↗

arXiv2023

Distributed Behavior Trees for Heterogeneous Robot Teams

Heterogeneous Robot Teams can provide a wide range of capabilities and therefore significant benefits when handling a mission. However, they also require new approaches to capability and mission definition that are not only suitable to handle heterogeneous capabilities but furthermore allow a combination or distribution of them with a coherent representation that is not limiting the individual robot. Behavior Trees offer many of the required properties, are growing in popularity for robot control and have been proposed for multirobot coordination, but always as separate behavior tree, defined in advance and without consideration for a changing team. In this paper, we propose a new behavior tree approach that is capable to handle complex real world robotic missions and is geared towards a distributed execution by providing built in functionalities for cost calculation, subtree distribution and data wiring. We present a formal definition, its open source implementation as ros_bt_py library and experimental verification of its capabilities.

cs.RO↗

arXiv2023

Motion Macro Programming on Assistive Robotic Manipulators: Three Skill Types for Everyday Tasks

Assistive robotic manipulators are becoming increasingly important for people with disabilities. Teleoperating the manipulator in mundane tasks is part of their daily lives. Instead of steering the robot through all actions, applying self-recorded motion macros could greatly facilitate repetitive tasks. Dynamic Movement Primitives (DMP) are a powerful method for skill learning via teleoperation. For this use case, however, they need simple heuristics to specify where to start, stop, and parameterize a skill without a background in computer science and academic sensor setups for autonomous perception. To achieve this goal, this paper provides the concept of local, global, and hybrid skills that form a modular basis for composing single-handed tasks of daily living. These skills are specified implicitly and can easily be programmed by users themselves, requiring only their basic robotic manipulator. The paper contributes all details for robot-agnostic implementations. Experiments validate the developed methods for exemplary tasks, such as scratching an itchy spot, sorting objects on a desk, and feeding a piggy bank with coins. The paper is accompanied by an open-source implementation at https://github.com/fzi-forschungszentrum-informatik/ArNe

cs.RO↗

arXiv2023

Inverse Kinematics with Forward Dynamics Solvers for Sampled Motion Tracking

Tracking Cartesian motion with end~effectors is a fundamental task in robot control. For motion that is not known in advance, the solvers must find fast solutions to the inverse kinematics (IK) problem for discretely sampled target poses. On joint control level, however, the robot's actuators operate in a continuous domain, requiring smooth transitions between individual states. In this work, we present a boost to the well-known Jacobian transpose method to address this goal, using the mass matrix of a virtually conditioned twin of the manipulator. Results on the UR10 show superior convergence and quality of our dynamics-based solver against the plain Jacobian method. Our algorithm is straightforward to implement as a controller, using common robotics libraries.

cs.RO↗

arXiv2023

Learning Human-Inspired Force Strategies for Robotic Assembly

The programming of robotic assembly tasks is a key component in manufacturing and automation. Force-sensitive assembly, however, often requires reactive strategies to handle slight changes in positioning and unforeseen part jamming. Learning such strategies from human performance is a promising approach, but faces two common challenges: the handling of low part clearances which is difficult to capture from demonstrations and learning intuitive strategies offline without access to the real hardware. We address these two challenges by learning probabilistic force strategies from data that are easily acquired offline in a robot-less simulation from human demonstrations with a joystick. We combine a Long Short Term Memory (LSTM) and a Mixture Density Network (MDN) to model human-inspired behavior in such a way that the learned strategies transfer easily onto real hardware. The experiments show a UR10e robot that completes a plastic assembly with clearances of less than 100 micrometers whose strategies were solely demonstrated in simulation.

cs.RO↗

arXiv2022

Reactive Neural Path Planning with Dynamic Obstacle Avoidance in a Condensed Configuration Space

We present a biologically inspired approach for path planning with dynamic obstacle avoidance. Path planning is performed in a condensed configuration space of a robot generated by self-organizing neural networks (SONN). The robot itself and static as well as dynamic obstacles are mapped from the Cartesian task space into the configuration space by precomputed kinematics. The condensed space represents a cognitive map of the environment, which is inspired by place cells and the concept of cognitive maps in mammalian brains. The generation of training data as well as the evaluation were performed on a real industrial robot accompanied by simulations. To evaluate the reactive collision-free online planning within a changing environment, a demonstrator was realized. Then, a comparative study regarding sample-based planners was carried out. So we could show that the robot is able to operate in dynamically changing environments and re-plan its motion trajectories within impressing 0.02 seconds, which proofs the real-time capability of our concept.

cs.RO↗

arXiv2022

A Walking Space Robot for On-Orbit Satellite Servicing: The ReCoBot

A key factor in the economic efficiency of satellites is their availability in orbit. Replacing standardized building blocks, such as empty fuel tanks or outdated electronic modules, could greatly extend the satellites' lifetime. This, however, requires flexible robots that can locomote on the surface of these satellites for optimal accessibility and manipulation. This paper introduces ReCoBot, a 7-axis walking space manipulator for locomotion and manipulation. The robot can connect to compatible structures with its symmetric ends and provides interfaces for manual teleoperation and motion planning with a constantly changing base and tip. We build on open-source robotics software and easily available components to evaluate the overall concept with an early stage demonstrator. The proposed manipulator has a length of 1.20 m and a weight of 10.4 kg and successfully locomotes over a satellite mockup in our lab environment.

cs.RO↗

arXiv2022

Comparing SONN Types for Efficient Robot Motion Planning in the Configuration Space

Motion planning in the configuration space (C-space) induces benefits, such as smooth trajectories. It becomes more complex as the degrees of freedom (DOF) increase. This is due to the direct relation between the dimensionality of the search space and the DOF. Self-organizing neural networks (SONN) and their famous candidate, the Self-Organizing Map, have been proven to be useful tools for C-space reduction while preserving its underlying topology, as presented in [29]. In this work, we extend our previous study with additional models and adapt the approach from human motion data towards robots' kinematics. The evaluation includes the best performant models from [29] and three additional SONN architectures, representing the consequent continuation of this previous work. Generated Trajectories, planned with the different SONN models, were successfully tested in a robot simulation.

cs.RO↗

arXiv2020

Virtual Forward Dynamics Models for Cartesian Robot Control

In industrial context, admittance control represents an important scheme in programming robots for interaction tasks with their environments. Those robots usually implement high-gain disturbance rejection on joint-level and hide direct access to the actuators behind velocity or position controlled interfaces. Using wrist force-torque sensors to add compliance to these systems, force-resolved control laws must map the control signals from Cartesian space to joint motion. Although forward dynamics algorithms would perfectly fit to that task description, their application to Cartesian robot control is not well researched. This paper proposes a general concept of virtual forward dynamics models for Cartesian robot control and investigates how the forward mapping behaves in comparison to well-established alternatives. Through decreasing the virtual system's link masses in comparison to the end effector, the virtual system becomes linear in the operational space dynamics. Experiments focus on stability and manipulability, particularly in singular configurations. Our results show that through this trick, forward dynamics can combine both benefits of the Jacobian inverse and the Jacobian transpose and, in this regard, outperforms the Damped Least Squares method.

cs.RO↗

arXiv2020

Contact Skill Imitation Learning for Robot-Independent Assembly Programming

Robotic automation is a key driver for the advancement of technology. The skills of human workers, however, are difficult to program and seem currently unmatched by technical systems. In this work we present a data-driven approach to extract and learn robot-independent contact skills from human demonstrations in simulation environments, using a Long Short Term Memory (LSTM) network. Our model learns to generate error-correcting sequences of forces and torques in task space from object-relative motion, which industrial robots carry out through a Cartesian force control scheme on the real setup. This scheme uses forward dynamics computation of a virtually conditioned twin of the manipulator to solve the inverse kinematics problem. We evaluate our methods with an assembly experiment, in which our algorithm handles part tilting and jamming in order to succeed. The results show that the skill is robust towards localization uncertainty in task space and across different joint configurations of the robot. With our approach, non-experts can easily program force-sensitive assembly tasks in a robot-independent way.

cs.RO↗

arXiv2019

Embodied Neuromorphic Vision with Event-Driven Random Backpropagation

Spike-based communication between biological neurons is sparse and unreliable. This enables the brain to process visual information from the eyes efficiently. Taking inspiration from biology, artificial spiking neural networks coupled with silicon retinas attempt to model these computations. Recent findings in machine learning allowed the derivation of a family of powerful synaptic plasticity rules approximating backpropagation for spiking networks. Are these rules capable of processing real-world visual sensory data? In this paper, we evaluate the performance of Event-Driven Random Back-Propagation (eRBP) at learning representations from event streams provided by a Dynamic Vision Sensor (DVS). First, we show that eRBP matches state-of-the-art performance on the DvsGesture dataset with the addition of a simple covert attention mechanism. By remapping visual receptive fields relatively to the center of the motion, this attention mechanism provides translation invariance at low computational cost compared to convolutions. Second, we successfully integrate eRBP in a real robotic setup, where a robotic arm grasps objects according to detected visual affordances. In this setup, visual information is actively sensed by a DVS mounted on a robotic head performing microsaccadic eye movements. We show that our method classifies affordances within 100ms after microsaccade onset, which is comparable to human performance reported in behavioral study. Our results suggest that advances in neuromorphic technology and plasticity rules enable the development of autonomous robots operating at high speed and low energy consumption.

cs.NE↗