arXiv subjects

Meiling Wang

Publications and source records attributed to Meiling Wang.

28 records · Page 2Linked to original sources

arXiv2024

VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions

Visual imitation learning (VIL) provides an efficient and intuitive strategy for robotic systems to acquire novel skills. Recent advancements in Vision Language Models (VLMs) have demonstrated remarkable performance in vision and language reasoning capabilities for VIL tasks. Despite the progress, current VIL methods naively employ VLMs to learn high-level plans from human videos, relying on pre-defined motion primitives for executing physical interactions, which remains a major bottleneck. In this work, we present VLMimic, a novel paradigm that harnesses VLMs to directly learn even fine-grained action levels, only given a limited number of human videos. Specifically, VLMimic first grounds object-centric movements from human videos, and learns skills using hierarchical constraint representations, facilitating the derivation of skills with fine-grained action levels from limited human videos. These skills are refined and updated through an iterative comparison strategy, enabling efficient adaptation to unseen environments. Our extensive experiments exhibit that our VLMimic, using only 5 human videos, yields significant improvements of over 27% and 21% in RLBench and real-world manipulation tasks, and surpasses baselines by over 37% in long-horizon tasks.

cs.RO↗

arXiv2024

OpenObject-NAV: Open-Vocabulary Object-Oriented Navigation Based on Dynamic Carrier-Relationship Scene Graph

In everyday life, frequently used objects like cups often have unfixed positions and multiple instances within the same category, and their carriers frequently change as well. As a result, it becomes challenging for a robot to efficiently navigate to a specific instance. To tackle this challenge, the robot must capture and update scene changes and plans continuously. However, current object navigation approaches primarily focus on semantic-level and lack the ability to dynamically update scene representation. This paper captures the relationships between frequently used objects and their static carriers. It constructs an open-vocabulary Carrier-Relationship Scene Graph (CRSG) and updates the carrying status during robot navigation to reflect the dynamic changes of the scene. Based on the CRSG, we further propose an instance navigation strategy that models the navigation process as a Markov Decision Process. At each step, decisions are informed by Large Language Model's commonsense knowledge and visual-language feature similarity. We designed a series of long-sequence navigation tasks for frequently used everyday items in the Habitat simulator. The results demonstrate that by updating the CRSG, the robot can efficiently navigate to moved targets. Additionally, we deployed our algorithm on a real robot and validated its practical effectiveness.

cs.RO↗

arXiv2024

Point Tree Transformer for Point Cloud Registration

Point cloud registration is a fundamental task in the fields of computer vision and robotics. Recent developments in transformer-based methods have demonstrated enhanced performance in this domain. However, the standard attention mechanism utilized in these methods often integrates many low-relevance points, thereby struggling to prioritize its attention weights on sparse yet meaningful points. This inefficiency leads to limited local structure modeling capabilities and quadratic computational complexity. To overcome these limitations, we propose the Point Tree Transformer (PTT), a novel transformer-based approach for point cloud registration that efficiently extracts comprehensive local and global features while maintaining linear computational complexity. The PTT constructs hierarchical feature trees from point clouds in a coarse-to-dense manner, and introduces a novel Point Tree Attention (PTA) mechanism, which follows the tree structure to facilitate the progressive convergence of attended regions towards salient points. Specifically, each tree layer selectively identifies a subset of key points with the highest attention scores. Subsequent layers focus attention on areas of significant relevance, derived from the child points of the selected point set. The feature extraction process additionally incorporates coarse point features that capture high-level semantic information, thus facilitating local structure modeling and the progressive integration of multiscale information. Consequently, PTA empowers the model to concentrate on crucial local structures and derive detailed local information while maintaining linear computational complexity. Extensive experiments conducted on the 3DMatch, ModelNet40, and KITTI datasets demonstrate that our method achieves superior performance over the state-of-the-art methods.

cs.CV↗

arXiv2023

PointGPT: Auto-regressively Generative Pre-training from Point Clouds

Large language models (LLMs) based on the generative pre-training transformer (GPT) have demonstrated remarkable effectiveness across a diverse range of downstream tasks. Inspired by the advancements of the GPT, we present PointGPT, a novel approach that extends the concept of GPT to point clouds, addressing the challenges associated with disorder properties, low information density, and task gaps. Specifically, a point cloud auto-regressive generation task is proposed to pre-train transformer models. Our method partitions the input point cloud into multiple point patches and arranges them in an ordered sequence based on their spatial proximity. Then, an extractor-generator based transformer decoder, with a dual masking strategy, learns latent representations conditioned on the preceding point patches, aiming to predict the next one in an auto-regressive manner. Our scalable approach allows for learning high-capacity models that generalize well, achieving state-of-the-art performance on various downstream tasks. In particular, our approach achieves classification accuracies of 94.9% on the ModelNet40 dataset and 93.4% on the ScanObjectNN dataset, outperforming all other transformer models. Furthermore, our method also attains new state-of-the-art accuracies on all four few-shot learning benchmarks.

cs.CV↗

arXiv2022

AdaCM: Adaptive ColorMLP for Real-Time Universal Photo-realistic Style Transfer

Photo-realistic style transfer aims at migrating the artistic style from an exemplar style image to a content image, producing a result image without spatial distortions or unrealistic artifacts. Impressive results have been achieved by recent deep models. However, deep neural network based methods are too expensive to run in real-time. Meanwhile, bilateral grid based methods are much faster but still contain artifacts like overexposure. In this work, we propose the \textbf{Adaptive ColorMLP (AdaCM)}, an effective and efficient framework for universal photo-realistic style transfer. First, we find the complex non-linear color mapping between input and target domain can be efficiently modeled by a small multi-layer perceptron (ColorMLP) model. Then, in \textbf{AdaCM}, we adopt a CNN encoder to adaptively predict all parameters for the ColorMLP conditioned on each input content and style image pair. Experimental results demonstrate that AdaCM can generate vivid and high-quality stylization results. Meanwhile, our AdaCM is ultrafast and can process a 4K resolution image in 6ms on one V100 GPU.

cs.CV↗

arXiv2021

Full Transformer Framework for Robust Point Cloud Registration with Deep Information Interaction

Recent Transformer-based methods have achieved advanced performance in point cloud registration by utilizing advantages of the Transformer in order-invariance and modeling dependency to aggregate information. However, they still suffer from indistinct feature extraction, sensitivity to noise, and outliers. The reasons are: (1) the adoption of CNNs fails to model global relations due to their local receptive fields, resulting in extracted features susceptible to noise; (2) the shallow-wide architecture of Transformers and lack of positional encoding lead to indistinct feature extraction due to inefficient information interaction; (3) the omission of geometrical compatibility leads to inaccurate classification between inliers and outliers. To address above limitations, a novel full Transformer network for point cloud registration is proposed, named the Deep Interaction Transformer (DIT), which incorporates: (1) a Point Cloud Structure Extractor (PSE) to model global relations and retrieve structural information with Transformer encoders; (2) a deep-narrow Point Feature Transformer (PFT) to facilitate deep information interaction across two point clouds with positional encoding, such that Transformers can establish comprehensive associations and directly learn relative position between points; (3) a Geometric Matching-based Correspondence Confidence Evaluation (GMCCE) method to measure spatial consistency and estimate inlier confidence by designing the triangulated descriptor. Extensive experiments on clean, noisy, partially overlapping point cloud registration demonstrate that our method outperforms state-of-the-art methods.

cs.CV↗

arXiv2021

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer

Fast arbitrary neural style transfer has attracted widespread attention from academic, industrial and art communities due to its flexibility in enabling various applications. Existing solutions either attentively fuse deep style feature into deep content feature without considering feature distributions, or adaptively normalize deep content feature according to the style such that their global statistics are matched. Although effective, leaving shallow feature unexplored and without locally considering feature statistics, they are prone to unnatural output with unpleasing local distortions. To alleviate this problem, in this paper, we propose a novel attention and normalization module, named Adaptive Attention Normalization (AdaAttN), to adaptively perform attentive normalization on per-point basis. Specifically, spatial attention score is learnt from both shallow and deep features of content and style images. Then per-point weighted statistics are calculated by regarding a style feature point as a distribution of attention-weighted output of all style feature points. Finally, the content feature is normalized so that they demonstrate the same local feature statistics as the calculated per-point weighted style feature statistics. Besides, a novel local feature loss is derived based on AdaAttN to enhance local visual quality. We also extend AdaAttN to be ready for video style transfer with slight modifications. Experiments demonstrate that our method achieves state-of-the-art arbitrary image/video style transfer. Codes and models are available.

cs.CV↗

arXiv2020

Group Connectivity under $3$-Edge-Connectivity

Let $S,T$ be two distinct finite Abelian groups with $|S|=|T|$. A fundamental theorem of Tutte shows that a graph admits a nowhere-zero $S$-flow if and only if it admits a nowhere-zero $T$-flow. Jaeger, Linial, Payan and Tarsi in 1992 introduced group connectivity as an extension of flow theory, and they asked whether such a relation holds for group connectivity analogy. It was negatively answered by Hušek, Mohelníková and Šámal in 2017 for graphs with edge-connectivity 2 for the groups $S=\mathbb{Z}_4$ and $T=\mathbb{Z}_2^2$. In this paper, we extend their results to $3$-edge-connected graphs (including both cubic and general graphs), which answers open problems proposed by Hušek, Mohelníková and Šámal(2017) and Lai, Li, Shao and Zhan(2011). Combining some previous results, this characterizes all the equivalence of group connectivity under $3$-edge-connectivity, showing that every $3$-edge-connected $S$-connected graph is $T$-connected if and only if $\{S,T\}\neq \{\mathbb{Z}_4,\mathbb{Z}_2^2\}$.

math.CO↗

arXiv2019

Spanning Triangle-trees and Flows of Graphs

In this paper we study the flow-property of graphs containing a spanning triangle-tree. Our main results provide a structure characterization of graphs with a spanning triangle-tree admitting a nowhere-zero $3$-flow. All these graphs without nowhere-zero $3$-flows are constructed from $K_4$ by a so-called bull-growing operation. This generalizes a result of Fan et al. in 2008 on triangularly-connected graphs and particularly shows that every $4$-edge-connected graph with a spanning triangle-tree has a nowhere-zero $3$-flow. A well-known classical theorem of Jaeger in 1979 shows that every graph with two edge-disjoint spanning trees admits a nowhere-zero $4$-flow. We prove that every graph with two edge-disjoint spanning triangle-trees has a flow strictly less than $3$.

math.CO↗

arXiv2019

Complementary Graphs with Flows Less Than Three

X. Hou, H.-J. Lai, P. Li and C.-Q. Zhang [J. Graph Theory 69 (2012) 464-470] showed that for a simple graph $G$ with $|V(G)|\ge 44$, if $\min\{δ(G),δ(G^c)\}\ge 4$, then either $G$ or its complementary graph $G^c$ has a nowhere-zero $3$-flow. In this paper, we improve this result by showing that if $|V(G)|\ge 32$ and $\min\{δ(G),δ(G^c)\}\ge 4$, then either $G$ or $G^c$ has flow index strictly less than $3$. Our result is proved by a newly developed closure operation and contraction method.

math.CO↗