arXiv subjects

Simon Schwaiger

Publications and source records attributed to Simon Schwaiger.

5 recordsLinked to original sources

arXiv2026

ReSiReg: Towards Spatially Consistent Semantics in Language-Conditioned Robotic Tasks

Vision-Language Models (VLMs) enable robots to follow open-language instructions. However, dense VLM embeddings have shown to be noisy and lack spatial consistency. This is problematic for robotic applications, which require simultaneous reasoning over semantics and 3D space. We examine spatial structure across recent VLMs and propose ReSiReg, a feature reconstruction method that uses spatially consistent VLM intermediates to improve dense language-grounded retrieval. ReSiReg clusters intermediates into visual prototypes, derives their language descriptors, and reconstructs each patch as a soft mixture of prototype-level language embeddings. We evaluate quantitatively on OVSS and 3D mapping across backbones, and qualitatively in real-world manipulation scenes. Quantitative results show improved dense retrieval; manipulation scenes show more spatially consistent target activations. We further provide a compact 25M dense VLM for robotic applications, substantially smaller than and competitive with ViT-B baselines. Available at https://resireg.github.io

cs.RO↗

arXiv2026

ART-VS: Adaptive Resolution Tiling for Vision Transformer Visual Servoing

Visual servoing with self-supervised Vision Transformer (ViT) features enables training-free robotic positioning with strong generalization, but faces a fundamental trade-off between robustness and precision. Coarse patch-level descriptors provide stable correspondences yet limit positioning accuracy. Increasing image resolution improves precision but yields only marginal robustness gains - under perturbation, high-resolution processing improves convergence success rate from 76.6% to just 81.0% despite 12x more ViT patches. Therefore, we propose Adaptive Resolution Tiling Visual Servoing (ART-VS), a two-phase method that adapts feature granularity to servoing progress: a coarse phase at native ViT resolution for stable alignment, then a tiled high-resolution phase that restricts matching to local neighborhoods improving positioning accuracy. Without any task-specific training, ART-VS achieves 95.4% convergence under perturbation, outperforming standard and full-resolution ViT-based servoing by 18.8 and 14.4 percentage points. Over the former it reduces positioning error by 53%, while running at over 10x higher speed and 27% lower VRAM than the latter. We validate ART-VS across three ViT backbones and demonstrate real-world category-level grasping of unseen object instances, achieving 95/100 on transparent bottles and 98/100 on shoes. Code available under https://art-vs.github.io/.

cs.RO↗

arXiv2025

OTAS: Open-vocabulary Token Alignment for Outdoor Segmentation

Understanding open-world semantics is critical for robotic planning and control, particularly in unstructured outdoor environments. Existing vision-language mapping approaches typically rely on object-centric segmentation priors, which often fail outdoors due to semantic ambiguities and indistinct class boundaries. We propose OTAS - an Open-vocabulary Token Alignment method for outdoor Segmentation. OTAS addresses the limitations of open-vocabulary segmentation models by extracting semantic structure directly from the output tokens of pre-trained vision models. By clustering semantically similar structures across single and multiple views and grounding them in language, OTAS reconstructs a geometrically consistent feature field that supports open-vocabulary segmentation queries. Our method operates in a zero-shot manner, without scene-specific fine-tuning, and achieves real-time performance of up to ~17 fps. On the Off-Road Freespace Detection dataset, OTAS yields a modest IoU improvement over fine-tuned and open-vocabulary 2D segmentation baselines. In 3D segmentation on TartanAir, it achieves up to a 151% relative IoU improvement compared to existing open-vocabulary mapping methods. Real-world reconstructions further demonstrate OTAS' applicability to robotic deployment. Code and a ROS 2 node are available at https://otas-segmentation.github.io/.

cs.RO↗

arXiv2024

UGV-CBRN: An Unmanned Ground Vehicle for Chemical, Biological, Radiological, and Nuclear Disaster Response

Robotic search and rescue (SAR) supports response teams by accelerating disaster assessment and by keeping operators away from hazardous environments. In the event of a chemical, biological, radiological, and nuclear (CBRN) disaster, robots are deployed to identify and locate radiation sources. Human responders then assess the situation and neutralize the danger. The presented system takes a step toward enhanced integration of robots into SAR teams. Integrating autonomous radiation mapping with semi-autonomous substance sampling and online analysis of the CBRN threat lets the human operator localize and assess the threat from a safe distance. Two LiDARs, an IMU, and a Geiger counter are used for mapping the surrounding area and localizing potential radiation sources. A mobile manipulator with six Degrees of Freedom manipulates valves and samples substances that are analyzed by an onboard Raman spectrometer. The human operator monitors the mission's progression from a remote location defining target locations and directing the semi-autonomous manipulation processes. Diverse recovery behaviours aid robot deployment, system state monitoring, as well as recovery of hard- and software. Field tests showcase the capabilities of the presented system during trials at the CBRN disaster response challenge European Robotics Hackathon (EnRicH). We provide recorded sensor data and implemented software through a GitHub repository: https://github.com/TW-Robotics/search-and-rescue-robot-2024.

cs.RO↗

arXiv2024

From Words to Poses: Enhancing Novel Object Pose Estimation with Vision Language Models

Robots are increasingly envisioned to interact in real-world scenarios, where they must continuously adapt to new situations. To detect and grasp novel objects, zero-shot pose estimators determine poses without prior knowledge. Recently, vision language models (VLMs) have shown considerable advances in robotics applications by establishing an understanding between language input and image input. In our work, we take advantage of VLMs zero-shot capabilities and translate this ability to 6D object pose estimation. We propose a novel framework for promptable zero-shot 6D object pose estimation using language embeddings. The idea is to derive a coarse location of an object based on the relevancy map of a language-embedded NeRF reconstruction and to compute the pose estimate with a point cloud registration method. Additionally, we provide an analysis of LERF's suitability for open-set object pose estimation. We examine hyperparameters, such as activation thresholds for relevancy maps and investigate the zero-shot capabilities on an instance- and category-level. Furthermore, we plan to conduct robotic grasping experiments in a real-world setting.

cs.CV↗