arXiv subjects

Xiaotong Chen

Publications and source records attributed to Xiaotong Chen.

32 records · Page 2Linked to original sources

arXiv2022

TransNet: Category-Level Transparent Object Pose Estimation

Transparent objects present multiple distinct challenges to visual perception systems. First, their lack of distinguishing visual features makes transparent objects harder to detect and localize than opaque objects. Even humans find certain transparent surfaces with little specular reflection or refraction, e.g. glass doors, difficult to perceive. A second challenge is that common depth sensors typically used for opaque object perception cannot obtain accurate depth measurements on transparent objects due to their unique reflective properties. Stemming from these challenges, we observe that transparent object instances within the same category (e.g. cups) look more similar to each other than to ordinary opaque objects of that same category. Given this observation, the present paper sets out to explore the possibility of category-level transparent object pose estimation rather than instance-level pose estimation. We propose TransNet, a two-stage pipeline that learns to estimate category-level transparent object pose using localized depth completion and surface normal estimation. TransNet is evaluated in terms of pose estimation accuracy on a recent, large-scale transparent object dataset and compared to a state-of-the-art category-level pose estimation approach. Results from this comparison demonstrate that TransNet achieves improved pose estimation accuracy on transparent objects and key findings from the included ablation studies suggest future directions for performance improvements.

cs.CV↗

arXiv2022

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

Benefiting from language flexibility and compositionality, humans naturally intend to use language to command an embodied agent for complex tasks such as navigation and object manipulation. In this work, we aim to fill the blank of the last mile of embodied agents -- object manipulation by following human guidance, e.g., "move the red mug next to the box while keeping it upright." To this end, we introduce an Automatic Manipulation Solver (AMSolver) system and build a Vision-and-Language Manipulation benchmark (VLMbench) based on it, containing various language instructions on categorized robotic manipulation tasks. Specifically, modular rule-based task templates are created to automatically generate robot demonstrations with language instructions, consisting of diverse object shapes and appearances, action types, and motion constraints. We also develop a keypoint-based model 6D-CLIPort to deal with multi-view observations and language input and output a sequence of 6 degrees of freedom (DoF) actions. We hope the new simulator and benchmark will facilitate future research on language-guided robotic manipulation.

cs.RO↗

arXiv2022

ProgressLabeller: Visual Data Stream Annotation for Training Object-Centric 3D Perception

Visual perception tasks often require vast amounts of labelled data, including 3D poses and image space segmentation masks. The process of creating such training data sets can prove difficult or time-intensive to scale up to efficacy for general use. Consider the task of pose estimation for rigid objects. Deep neural network based approaches have shown good performance when trained on large, public datasets. However, adapting these networks for other novel objects, or fine-tuning existing models for different environments, requires significant time investment to generate newly labelled instances. Towards this end, we propose ProgressLabeller as a method for more efficiently generating large amounts of 6D pose training data from color images sequences for custom scenes in a scalable manner. ProgressLabeller is intended to also support transparent or translucent objects, for which the previous methods based on depth dense reconstruction will fail. We demonstrate the effectiveness of ProgressLabeller by rapidly create a dataset of over 1M samples with which we fine-tune a state-of-the-art pose estimation network in order to markedly improve the downstream robotic grasp success rates. ProgressLabeller is open-source at https://github.com/huijieZH/ProgressLabeller.

cs.RO↗

arXiv2022

ClearPose: Large-scale Transparent Object Dataset and Benchmark

Transparent objects are ubiquitous in household settings and pose distinct challenges for visual sensing and perception systems. The optical properties of transparent objects leave conventional 3D sensors alone unreliable for object depth and pose estimation. These challenges are highlighted by the shortage of large-scale RGB-Depth datasets focusing on transparent objects in real-world settings. In this work, we contribute a large-scale real-world RGB-Depth transparent object dataset named ClearPose to serve as a benchmark dataset for segmentation, scene-level depth completion and object-centric pose estimation tasks. The ClearPose dataset contains over 350K labeled real-world RGB-Depth frames and 5M instance annotations covering 63 household objects. The dataset includes object categories commonly used in daily life under various lighting and occluding conditions as well as challenging test scenarios such as cases of occlusion by opaque or translucent objects, non-planar orientations, presence of liquids, etc. We benchmark several state-of-the-art depth completion and object pose estimation deep neural networks on ClearPose. The dataset and benchmarking source code is available at https://github.com/opipari/ClearPose.

cs.CV↗

arXiv2022

PatchTrack: Multiple Object Tracking Using Frame Patches

Object motion and object appearance are commonly used information in multiple object tracking (MOT) applications, either for associating detections across frames in tracking-by-detection methods or direct track predictions for joint-detection-and-tracking methods. However, not only are these two types of information often considered separately, but also they do not help optimize the usage of visual information from the current frame of interest directly. In this paper, we present PatchTrack, a Transformer-based joint-detection-and-tracking system that predicts tracks using patches of the current frame of interest. We use the Kalman filter to predict the locations of existing tracks in the current frame from the previous frame. Patches cropped from the predicted bounding boxes are sent to the Transformer decoder to infer new tracks. By utilizing both object motion and object appearance information encoded in patches, the proposed method pays more attention to where new tracks are more likely to occur. We show the effectiveness of PatchTrack on recent MOT benchmarks, including MOT16 (MOTA 73.71%, IDF1 65.77%) and MOT17 (MOTA 73.59%, IDF1 65.23%). The results are published on https://motchallenge.net/method/MOT=4725&chl=10.

cs.CV↗

arXiv2020

An ADMM-LAP method for total variation myopic deconvolution of adaptive optics retinal images

Adaptive optics (AO) corrected ood imaging of the retina is a popular technique for studying the retinal structure and function in the living eye. However, the raw retinal images are usually of poor contrast and the interpretation of such images requires image deconvolution. Different from standard deconvolution problems where the point spread function (PSF) is completely known, the PSF in these retinal imaging problems is only partially known which leads to the more complicated myopic (mildly blind) deconvolution problem. In this paper, we propose an efficient numerical scheme for solving this myopic deconvolution problem with total variational (TV) regularization. First, we apply the alternating direction method of multipliers (ADMM) to tackle the TV regularizer. Specifically, we reformulate the TV problem as an equivalent equality constrained problem where the objective function is separable, and then minimize the augmented Lagrangian function by alternating between two (separated) blocks of unknowns to obtain the solution. Due to the structure of the retinal images, the subproblems with respect to the fidelity term appearing within each ADMM iteration are tightly coupled and a variation of the Linearize And Project (LAP) method is designed to solve these subproblems efficiently. The proposed method is called the ADMM-LAP method. Theoretically, we establish the subsequence convergence of the ADMM-LAP method to a stationary point. Both the theoretical complexity analysis and numerical results are provided to demonstrate the efficiency of the ADMM-LAP method.

math.OC↗

arXiv2020

Design, Control, and Applications of a Soft Robotic Arm

This paper presents the design, control, and applications of a multi-segment soft robotic arm. In order to design a soft arm with large load capacity, several design principles are proposed by analyzing two kinds of buckling issues, under which we present a novel structure named Honeycomb Pneumatic Networks (HPN). Parameter optimization method, based on finite element method (FEM), is proposed to optimize HPN Arm design parameters. Through a quick fabrication process, several prototypes with different performance are made, one of which can achieve the transverse load capacity of 3 kg under 3 bar pressure. Next, considering different internal and external conditions, we develop three controllers according to different model precision. Specifically, based on accurate model, an open-loop controller is realized by combining piece-wise constant curvature (PCC) modeling method and machine learning method. Based on inaccurate model, a feedback controller, using estimated Jacobian, is realized in 3D space. A model-free controller, using reinforcement learning to learn a control policy rather than a model, is realized in 2D plane, with minimal training data. Then, these three control methods are compared on a same experiment platform to explore the applicability of different methods under different conditions. Lastly, we figure out that soft arm can greatly simplify the perception, planning, and control of interaction tasks through its compliance, which is its main advantage over the rigid arm. Through plentiful experiments in three interaction application scenarios, human-robot interaction, free space interaction task, and confined space interaction task, we demonstrate the potential application prospect of the soft arm.

cs.RO↗

arXiv2020

LIT: Light-field Inference of Transparency for Refractive Object Localization

Translucency is prevalent in everyday scenes. As such, perception of transparent objects is essential for robots to perform manipulation. Compared with texture-rich or texture-less Lambertian objects, transparency induces significant uncertainty on object appearances. Ambiguity can be due to changes in lighting, viewpoint, and backgrounds, each of which brings challenges to existing object pose estimation algorithms. In this work, we propose LIT, a two-stage method for transparent object pose estimation using light-field sensing and photorealistic rendering. LIT employs multiple filters specific to light-field imagery in deep networks to capture transparent material properties, with robust depth and pose estimators based on generative sampling. Along with the LIT algorithm, we introduce the light-field transparent object dataset ProLIT for the tasks of recognition, localization and pose estimation. With respect to this ProLIT dataset, we demonstrate that LIT can outperform both state-of-the-art end-to-end pose estimation methods and a generative pose estimator on transparent objects.

cs.RO↗

arXiv2019

GRIP: Generative Robust Inference and Perception for Semantic Robot Manipulation in Adversarial Environments

Recent advancements have led to a proliferation of machine learning systems used to assist humans in a wide range of tasks. However, we are still far from accurate, reliable, and resource-efficient operations of these systems. For robot perception, convolutional neural networks (CNNs) for object detection and pose estimation are recently coming into widespread use. However, neural networks are known to suffer overfitting during training process and are less robust within unseen conditions, which are especially vulnerable to adversarial scenarios. In this work, we propose Generative Robust Inference and Perception (GRIP) as a two-stage object detection and pose estimation system that aims to combine relative strengths of discriminative CNNs and generative inference methods to achieve robust estimation. Our results show that a second stage of sample-based generative inference is able to recover from false object detection by CNNs, and produce robust estimations in adversarial conditions. We demonstrate the efficacy of GRIP robustness through comparison with state-of-the-art learning-based pose estimators and pick-and-place manipulation in dark and cluttered environments.

cs.RO↗

arXiv2019

A multi-level ADMM algorithm for elliptic PDE-constrained optimization problems

In this paper, the elliptic PDE-constrained optimization problem with box constraints on the control is studied. To numerically solve the problem, we apply the 'optimize-discretize-optimize' strategy. Specifically, the alternating direction method of multipliers (ADMM) algorithm is applied in function space first, then the standard piecewise linear finite element approach is employed to discretize the subproblems in each iteration. Finally, some efficient numerical methods are applied to solve the discretized subproblems based on their structures. Motivated by the idea of the multi-level strategy, instead of fixing the mesh size before the computation process, we propose the strategy of gradually refining the grid. Moreover, the subproblems in each iteration are solved inexactly. Based on the strategies above, an efficient convergent multi-level ADMM (mADMM) algorithm is proposed. We present the convergence analysis and the iteration complexity results o(1/k) of the proposed algorithm for the PDE-constrained optimization problems. Numerical results show the high efficiency of the mADMM algorithm.

math.OC↗

arXiv2019

DigiVoice: Voice Biomarker Featurization and Analysis Pipeline

In recent years, data-driven models have enabled significant advances in medicine. Simultaneously, voice has shown potential for analysis in precision medicine as a biomarker for screening illnesses. There has been a growing trend to pursue voice data to understand neuropsychiatric diseases. In this paper, we present DigiVoice, a comprehensive feature extraction and analysis pipeline for voice data. DigiVoice supports raw .WAV files and text transcriptions in order to analyze the entire content of voice. DigiVoice supports feature extraction including acoustic, natural language, linguistic complexity, and semantic coherence features. DigiVoice also supports machine learning capabilities including data visualization, feature selection, feature transformation, and modeling. To our knowledge, DigiVoice provides the most comprehensive voice feature set for data analysis to date. With DigiVoice, we plan to accelerate research to correlate voice biomarkers with illness to enable data-driven treatment. We have worked closely with our industry partner, NeuroLex Laboratories, to make voice computing open source and accessible. DigiVoice enables researchers to leverage our technology across the domains of voice computing and precision medicine without domain-specific expertise. Our work allows any researchers to use voice as a biomarker in their past, current, or future studies.

eess.AS↗

arXiv2018

An FE-dABCD algorithm for elliptic optimal control problems with constraints on the gradient of the state and control

In this paper, elliptic control problems with integral constraint on the gradient of the state and box constraints on the control are considered. The optimal conditions of the problem are proved. To numerically solve the problem, we use the 'First discretize, then optimize' approach. Specifically, we discretize both the state and the control by piecewise linear functions. To solve the discretized problem efficiently, we first transform it into a multi-block unconstrained convex optimization problem via its dual, then we extend the inexact majorized accelerating block coordinate descent (imABCD) algorithm to solve it. The entire algorithm framework is called finite element duality-based inexact majorized accelerating block coordinate descent (FE-dABCD) algorithm. Thanks to the inexactness of the FE-dABCD algorithm, each subproblems are allowed to be solved inexactly. For the smooth subproblem, we use the generalized minimal residual (GMRES) method with preconditioner to slove it. For the nonsmooth subproblems, one of them has a closed form solution through introducing appropriate proximal term, another is solved combining semi-smooth Newton (SSN) method. Based on these efficient strategies, we prove that our proposed FE-dABCD algorithm enjoys $O(\frac{1}{k^2})$ iteration complexity. Some numerical experiments are done and the numerical results show the efficiency of the FE-dABCD algorithm.

math.OC↗

arXiv2018

Optical control of a single spin-valley in charged WSe$_2$ quantum dots

Control and manipulation of single charges and their internal degrees of freedom, such as spins, is a fundamental goal of nanoscience with promising technological applications. Recently, atomically thin semiconductors such as WSe$_2$ have emerged as a platform for valleytronics, offering rich possibilities for optical, magnetic and electrical control of the valley index. While progress has been made in controlling valley index of ensemble of charge carriers, valley control of individual charges, crucial for valleytronics, remains unexplored. Here, we provide unambiguous evidence for localized holes with net spin in optically active WSe$_2$ quantum dots (QDs) and control their spin-valley state with the helicity of the excitation laser under small magnetic field. We estimate a lower bound on the valley lifetime of a single charge in QD from recombination time to be $\sim$ nanoseconds. Remarkably, neutral QDs do not exhibit such a control, demonstrating the role of excess charge in prolonging the valley lifetime. Our work extends the field of 2D valleytronics to the level of single spin-valley, relevant for quantum information and sensing applications

cond-mat.mes-hall↗

arXiv2018

Entanglement of single-photons and chiral phonons in atomically thin WSe$_2$

Quantum entanglement is a fundamental phenomenon which, on the one hand, reveals deep connections between quantum mechanics, gravity and the space-time; on the other hand, has practical applications as a key resource in quantum information processing. While it is routinely achieved in photon-atom ensembles, entanglement involving the solid-state or macroscopic objects remains challenging albeit promising for both fundamental physics and technological applications. Here, we report entanglement between collective, chiral vibrations in two-dimensional (2D) WSe$_2$ host --- chiral phonons (CPs) --- and single-photons emitted from quantum dots (QDs) present in it. CPs which carry angular momentum were recently observed in WSe$_2$ and are a distinguishing feature of the underlying honeycomb lattice. The entanglement results from a "which-way" scattering process, involving an optical excitation in a QD and doubly-degenerate CPs, which takes place via two indistinguishable paths. Our unveiling of entanglement involving a macroscopic, collective excitation together with strong interaction between CPs and QDs in 2D materials opens up ways for phonon-driven entanglement of QDs and engineering chiral or non-reciprocal interactions at the single-photon level.

cond-mat.mes-hall↗