Searcharxiv⌕ Search

arXiv subjects

Jonas Frey

Publications and source records attributed to Jonas Frey.

53 records · Page 3Linked to original sources
arXiv2023

MEM: Multi-Modal Elevation Mapping for Robotics and Learning

Elevation maps are commonly used to represent the environment of mobile robots and are instrumental for locomotion and navigation tasks. However, pure geometric information is insufficient for many field applications that require appearance or semantic information, which limits their applicability to other platforms or domains. In this work, we extend a 2.5D robot-centric elevation mapping framework by fusing multi-modal information from multiple sources into a popular map representation. The framework allows inputting data contained in point clouds or images in a unified manner. To manage the different nature of the data, we also present a set of fusion algorithms that can be selected based on the information type and user requirements. Our system is designed to run on the GPU, making it real-time capable for various robotic and learning tasks. We demonstrate the capabilities of our framework by deploying it on multiple robots with varying sensor configurations and showcasing a range of applications that utilize multi-modal layers, including line detection, human detection, and colorization.

cs.RO↗
arXiv2023

SMUG Planner: A Safe Multi-Goal Planner for Mobile Robots in Challenging Environments

Robotic exploration or monitoring missions require mobile robots to autonomously and safely navigate between multiple target locations in potentially challenging environments. Currently, this type of multi-goal mission often relies on humans designing a set of actions for the robot to follow in the form of a path or waypoints. In this work, we consider the multi-goal problem of visiting a set of pre-defined targets, each of which could be visited from multiple potential locations. To increase autonomy in these missions, we propose a safe multi-goal (SMUG) planner that generates an optimal motion path to visit those targets. To increase safety and efficiency, we propose a hierarchical state validity checking scheme, which leverages robot-specific traversability learned in simulation. We use LazyPRM* with an informed sampler to accelerate collision-free path generation. Our iterative dynamic programming algorithm enables the planner to generate a path visiting more than ten targets within seconds. Moreover, the proposed hierarchical state validity checking scheme reduces the planning time by 30% compared to pure volumetric collision checking and increases safety by avoiding high-risk regions. We deploy the SMUG planner on the quadruped robot ANYmal and show its capability to guide the robot in multi-goal missions fully autonomously on rough terrain.

cs.RO↗
arXiv2023

Fast Traversability Estimation for Wild Visual Navigation

Natural environments such as forests and grasslands are challenging for robotic navigation because of the false perception of rigid obstacles from high grass, twigs, or bushes. In this work, we propose Wild Visual Navigation (WVN), an online self-supervised learning system for traversability estimation which uses only vision. The system is able to continuously adapt from a short human demonstration in the field. It leverages high-dimensional features from self-supervised visual transformer models, with an online scheme for supervision generation that runs in real-time on the robot. We demonstrate the advantages of our approach with experiments and ablation studies in challenging environments in forests, parks, and grasslands. Our system is able to bootstrap the traversable terrain segmentation in less than 5 min of in-field training time, enabling the robot to navigate in complex outdoor terrains - negotiating obstacles in high grass as well as a 1.4 km footpath following. While our experiments were executed with a quadruped robot, ANYmal, the approach presented can generalize to any ground robot.

cs.RO↗
arXiv2023

Seeing Through the Grass: Semantic Pointcloud Filter for Support Surface Learning

Mobile ground robots require perceiving and understanding their surrounding support surface to move around autonomously and safely. The support surface is commonly estimated based on exteroceptive depth measurements, e.g., from LiDARs. However, the measured depth fails to align with the true support surface in the presence of high grass or other penetrable vegetation. In this work, we present the Semantic Pointcloud Filter (SPF), a Convolutional Neural Network (CNN) that learns to adjust LiDAR measurements to align with the underlying support surface. The SPF is trained in a semi-self-supervised manner and takes as an input a LiDAR pointcloud and RGB image. The network predicts a binary segmentation mask that identifies the specific points requiring adjustment, along with estimating their corresponding depth values. To train the segmentation task, 300 distinct images are manually labeled into rigid and non-rigid terrain. The depth estimation task is trained in a self-supervised manner by utilizing the future footholds of the robot to estimate the support surface based on a Gaussian process. Our method can correctly adjust the support surface prior to interacting with the terrain and is extensively tested on the quadruped robot ANYmal. We show the qualitative benefits of SPF in natural environments for elevation mapping and traversability estimation compared to using raw sensor measurements and existing smoothing methods. Quantitative analysis is performed in various natural environments, and an improvement by 48% RMSE is achieved within a meadow terrain.

cs.RO↗
arXiv2023

Unsupervised Continual Semantic Adaptation through Neural Rendering

An increasing amount of applications rely on data-driven models that are deployed for perception tasks across a sequence of scenes. Due to the mismatch between training and deployment data, adapting the model on the new scenes is often crucial to obtain good performance. In this work, we study continual multi-scene adaptation for the task of semantic segmentation, assuming that no ground-truth labels are available during deployment and that performance on the previous scenes should be maintained. We propose training a Semantic-NeRF network for each scene by fusing the predictions of a segmentation model and then using the view-consistent rendered semantic labels as pseudo-labels to adapt the model. Through joint training with the segmentation model, the Semantic-NeRF model effectively enables 2D-3D knowledge transfer. Furthermore, due to its compact size, it can be stored in a long-term memory and subsequently used to render data from arbitrary viewpoints to reduce forgetting. We evaluate our approach on ScanNet, where we outperform both a voxel-based baseline and a state-of-the-art unsupervised domain adaptation method.

cs.CV↗
arXiv2023

Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions

Learning diverse skills is one of the main challenges in robotics. To this end, imitation learning approaches have achieved impressive results. These methods require explicitly labeled datasets or assume consistent skill execution to enable learning and active control of individual behaviors, which limits their applicability. In this work, we propose a cooperative adversarial method for obtaining single versatile policies with controllable skill sets from unlabeled datasets containing diverse state transition patterns by maximizing their discriminability. Moreover, we show that by utilizing unsupervised skill discovery in the generative adversarial imitation learning framework, novel and useful skills emerge with successful task fulfillment. Finally, the obtained versatile policies are tested on an agile quadruped robot called Solo 8 and present faithful replications of diverse skills encoded in the demonstrations.

cs.RO↗
arXiv2022

Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations

Learning agile skills is one of the main challenges in robotics. To this end, reinforcement learning approaches have achieved impressive results. These methods require explicit task information in terms of a reward function or an expert that can be queried in simulation to provide a target control output, which limits their applicability. In this work, we propose a generative adversarial method for inferring reward functions from partial and potentially physically incompatible demonstrations for successful skill acquirement where reference or expert demonstrations are not easily accessible. Moreover, we show that by using a Wasserstein GAN formulation and transitions from demonstrations with rough and partial information as input, we are able to extract policies that are robust and capable of imitating demonstrated behaviors. Finally, the obtained skills such as a backflip are tested on an agile quadruped robot called Solo 8 and present faithful replication of hand-held human demonstrations.

cs.RO↗
arXiv2022

Locomotion Policy Guided Traversability Learning using Volumetric Representations of Complex Environments

Despite the progress in legged robotic locomotion, autonomous navigation in unknown environments remains an open problem. Ideally, the navigation system utilizes the full potential of the robots' locomotion capabilities while operating within safety limits under uncertainty. The robot must sense and analyze the traversability of the surrounding terrain, which depends on the hardware, locomotion control, and terrain properties. It may contain information about the risk, energy, or time consumption needed to traverse the terrain. To avoid hand-crafted traversability cost functions we propose to collect traversability information about the robot and locomotion policy by simulating the traversal over randomly generated terrains using a physics simulator. Thousand of robots are simulated in parallel controlled by the same locomotion policy used in reality to acquire 57 years of real-world locomotion experience equivalent. For deployment on the real robot, a sparse convolutional network is trained to predict the simulated traversability cost, which is tailored to the deployed locomotion policy, from an entirely geometric representation of the environment in the form of a 3D voxel-occupancy map. This representation avoids the need for commonly used elevation maps, which are error-prone in the presence of overhanging obstacles and multi-floor or low-ceiling scenarios. The effectiveness of the proposed traversability prediction network is demonstrated for path planning for the legged robot ANYmal in various indoor and natural environments.

cs.RO↗
arXiv2022

Continual Adaptation of Semantic Segmentation using Complementary 2D-3D Data Representations

Semantic segmentation networks are usually pre-trained once and not updated during deployment. As a consequence, misclassifications commonly occur if the distribution of the training data deviates from the one encountered during the robot's operation. We propose to mitigate this problem by adapting the neural network to the robot's environment during deployment, without any need for external supervision. Leveraging complementary data representations, we generate a supervision signal, by probabilistically accumulating consecutive 2D semantic predictions in a volumetric 3D map. We then train the network on renderings of the accumulated semantic map, effectively resolving ambiguities and enforcing multi-view consistency through the 3D representation. In contrast to scene adaptation methods, we aim to retain the previously-learned knowledge, and therefore employ a continual learning experience replay strategy to adapt the network. Through extensive experimental evaluation, we show successful adaptation to real-world indoor scenes both on the ScanNet dataset and on in-house data recorded with an RGB-D sensor. Our method increases the segmentation accuracy on average by 9.9% compared to the fixed pre-trained neural network, while retaining knowledge from the pre-training dataset.

cs.RO↗
arXiv2022

Categories of partial equivalence relations as localizations

We construct a category of fibrant objects $\mathbb{C}\langle P\rangle$ in the sense of K. Brown from any indexed frame (a kind of indexed poset generalizing triposes) $P$, and show that its homotopy category is the Barr-exact category $\mathbb{C}[P]$ of partial equivalence relations and compatible functional relations. In particular this gives a presentation of realizability toposes as homotopy categories. We give criteria for the existence of left and right derived functors to functors $\mathbb{C}\langleΦ\rangle : \mathbb{C}\langle P\rangle\to \mathbb{C}\langle Q\rangle$ induced by finite-meet-preserving transformations $Φ : P \to Q$ between indexed frames.

math.CT↗
arXiv2018

Characterizing partitioned assemblies and realizability toposes

We give simple characterizations of the category PAsm(A) of partitioned assemblies, and of the realizability topos RT(A) over a partial combinatory algebra A. This answers the question for an 'extensional characterization' of realizability toposes.

math.CT↗
arXiv2018

Impredicative Encodings of (Higher) Inductive Types

Postulating an impredicative universe in dependent type theory allows System F style encodings of finitary inductive types, but these fail to satisfy the relevant η-equalities and consequently do not admit dependent eliminators. To recover η and dependent elimination, we present a method to construct refinements of these impredicative encodings, using ideas from homotopy type theory. We then extend our method to construct impredicative encodings of some higher inductive types, such as 1-truncation and the unit circle S1.

cs.LO↗
arXiv2015

Realizability Toposes from Specifications

We investigate a framework of Krivine realizability with I/O effects, and present a method of associating realizability models to specifications on the I/O behavior of processes, by using adequate interpretations of the central concepts of `pole' and `proof-like term'. This method does in particular allow to associate realizability models to computable functions. Following recent work of Streicher and others we show how these models give rise to triposes and toposes.

math.LO↗
arXiv2014

Ordered combinatory algebras and realizability

We consider different classes of combinatory structures related to Krivine realizability. We show, in the precise sense that they give rise to the same class of triposes, that they are equivalent for the purpose of modeling higher-order logic. We center our attentions in the role of a special kind of Ordered Combinatory Algebras-- that we call the "Krivine ordered combinatory algebras" ($\mathcal{KOCA}$s)-- that we propose as the foundational pillars for the categorical perspective of Krivine's classical realizability as presented by Streicher. Our procedure is the following: we show that each of the considered combinatory structures gives rise to an indexed preorder, and describe a way to transform the different structures into each other that preserves the associated indexed preorders up to equivalence. Since all structures give rise to the same indexed preorders, we only prove that they are triposes once: for the class of $\mathcal{KOCA}$s. We finish showing that in $\mathcal{KOCA}$s, one can define realizability in every higher-order language and in particular in higher-order arithmetic.

math.LO↗
arXiv2014

A fibrational study of realizability toposes

This is the author's PhD thesis. It is a contribution to categorical logic, in particular to the theory of realizability toposes. While the tools of categorical logic have proven very successful in analyzing and organizing proof theoretic realizability interpretations, it was remarked by experts (notably Peter Johnstone) that the field of realizability toposes itself was not clearly delineated, and lacked a powerful theory analogous to that of Grothendieck toposes. The present work sets out to remedy this situation to a certain extent. We argue that realizability toposes are best understood using Grothendieck fibrations, and develop a framework of fibrational cocompletions, which allows to view certain constructions from realizability in precise analogy to constructions of presheaf and sheaf toposes. Using these techniques, and a class of posetal fibrations that we call uniform preorders, we are able to give an extensional characterization of partial combinatory algebras and of the realizability toposes that are constructed from these algebras. Striving to develop the analogy to Grothendieck toposes further, we outline how to apply our techniques on arbitrary base toposes, and give a decomposition theorem for constant objects functors induced by triposes, analogous to the known decompositions of geometric morphisms. Finally, we sketch an approach of how to find a unified framework for Grothendieck toposes and realizability toposes, based on the observation that uniform preorders can be identified with preorders internal to a category of sheaves.

math.CT↗
arXiv2011

A 2-Categorical Analysis of the Tripos-to-Topos Construction

We characterize the tripos-to-topos construction of Hyland, Johnstone and Pitts as a biadjunction in a bicategory enriched category of equipment-like structures. These abstract concepts are necessary to handle the presence of oplax constructs --- the construction is only oplax functorial on certain classes of cartesian functors between triposes. A by-product of our analysis is the decomposition of the tripos-to-topos construction into two steps, the intermediate step being a weakened version of quasitoposes.

math.CT↗