arXiv subjects

Xiaobao Wei

Publications and source records attributed to Xiaobao Wei.

47 records · Page 3Linked to original sources

arXiv2024

WM-MoE: Weather-aware Multi-scale Mixture-of-Experts for Blind Adverse Weather Removal

Adverse weather removal tasks like deraining, desnowing, and dehazing are usually treated as separate tasks. However, in practical autonomous driving scenarios, the type, intensity,and mixing degree of weather are unknown, so handling each task separately cannot deal with the complex practical scenarios. In this paper, we study the blind adverse weather removal problem. Mixture-of-Experts (MoE) is a popular model that adopts a learnable gate to route the input to different expert networks. The principle of MoE involves using adaptive networks to process different types of unknown inputs. Therefore, MoE has great potential for blind adverse weather removal. However, the original MoE module is inadequate for coupled multiple weather types and fails to utilize multi-scale features for better performance. To this end, we propose a method called Weather-aware Multi-scale MoE (WM-MoE) based on Transformer for blind weather removal. WM-MoE includes two key designs: WEather-Aware Router (WEAR) and Multi-Scale Experts (MSE). WEAR assigns experts for each image token based on decoupled content and weather features, which enhances the model's capability to process multiple adverse weathers. To obtain discriminative weather features from images, we propose Weather Guidance Fine-grained Contrastive Learning (WGF-CL), which utilizes weather cluster information to guide the assignment of positive and negative samples for each image token. Since processing different weather types requires different receptive fields, MSE leverages multi-scale features to enhance the spatial relationship modeling capability, facilitating the high-quality restoration of diverse weather types and intensities. Our method achieves state-of-the-art performance in blind adverse weather removal on two public datasets and our dataset. We also demonstrate the advantage of our method on downstream segmentation tasks.

cs.CV↗

arXiv2024

NTO3D: Neural Target Object 3D Reconstruction with Segment Anything

Neural 3D reconstruction from multi-view images has recently attracted increasing attention from the community. Existing methods normally learn a neural field for the whole scene, while it is still under-explored how to reconstruct a target object indicated by users. Considering the Segment Anything Model (SAM) has shown effectiveness in segmenting any 2D images, in this paper, we propose NTO3D, a novel high-quality Neural Target Object 3D (NTO3D) reconstruction method, which leverages the benefits of both neural field and SAM. We first propose a novel strategy to lift the multi-view 2D segmentation masks of SAM into a unified 3D occupancy field. The 3D occupancy field is then projected into 2D space and generates the new prompts for SAM. This process is iterative until convergence to separate the target object from the scene. After this, we then lift the 2D features of the SAM encoder into a 3D feature field in order to improve the reconstruction quality of the target object. NTO3D lifts the 2D masks and features of SAM into the 3D neural field for high-quality neural target object 3D reconstruction. We conduct detailed experiments on several benchmark datasets to demonstrate the advantages of our method. The code will be available at: https://github.com/ucwxb/NTO3D.

cs.CV↗

arXiv2023

MoEC: Mixture of Experts Implicit Neural Compression

Emerging Implicit Neural Representation (INR) is a promising data compression technique, which represents the data using the parameters of a Deep Neural Network (DNN). Existing methods manually partition a complex scene into local regions and overfit the INRs into those regions. However, manually designing the partition scheme for a complex scene is very challenging and fails to jointly learn the partition and INRs. To solve the problem, we propose MoEC, a novel implicit neural compression method based on the theory of mixture of experts. Specifically, we use a gating network to automatically assign a specific INR to a 3D point in the scene. The gating network is trained jointly with the INRs of different local regions. Compared with block-wise and tree-structured partitions, our learnable partition can adaptively find the optimal partition in an end-to-end manner. We conduct detailed experiments on massive and diverse biomedical data to demonstrate the advantages of MoEC against existing approaches. In most of experiment settings, we have achieved state-of-the-art results. Especially in cases of extreme compression ratios, such as 6000x, we are able to uphold the PSNR of 48.16.

cs.CV↗

arXiv2023

DiffusionTalker: Personalization and Acceleration for Speech-Driven 3D Face Diffuser

Speech-driven 3D facial animation has been an attractive task in both academia and industry. Traditional methods mostly focus on learning a deterministic mapping from speech to animation. Recent approaches start to consider the non-deterministic fact of speech-driven 3D face animation and employ the diffusion model for the task. However, personalizing facial animation and accelerating animation generation are still two major limitations of existing diffusion-based methods. To address the above limitations, we propose DiffusionTalker, a diffusion-based method that utilizes contrastive learning to personalize 3D facial animation and knowledge distillation to accelerate 3D animation generation. Specifically, to enable personalization, we introduce a learnable talking identity to aggregate knowledge in audio sequences. The proposed identity embeddings extract customized facial cues across different people in a contrastive learning manner. During inference, users can obtain personalized facial animation based on input audio, reflecting a specific talking style. With a trained diffusion model with hundreds of steps, we distill it into a lightweight model with 8 steps for acceleration. Extensive experiments are conducted to demonstrate that our method outperforms state-of-the-art methods. The code will be released.

cs.CV↗

arXiv2023

Revisit to the yield ratio of triton and $^3$He as an indicator of neutron-rich neck emission

The neutron rich neck zone created in heavy ion reaction is experimentally probed by the production of the $A=3$ isobars. The energy spectra and angular distributions of triton and $^3$He are measured with the CSHINE detector in $^{86}$Kr +$^{208}$Pb reactions at 25 MeV/u. While the energy spectrum of $^{3}$He is harder than that of triton, known as "$^{3}$He-puzzle", the yield ratio $R({\rm t/^3He})$ presents a robust rising trend with the polar angle in laboratory. Using the fission fragments to reconstruct the fission plane, the enhancement of out-plane $R({\rm t/^3He})$ is confirmed in comparison to the in-plane ratios. Transport model simulations reproduce qualitatively the experimental trends, but the quantitative agreement is not achieved. The results demonstrate that a neutron rich neck zone is formed in the reactions. Further studies are called for to understand the clustering and the isospin dynamics related to neck formation.

nucl-ex↗

arXiv2023

Open-Vocabulary Point-Cloud Object Detection without 3D Annotation

The goal of open-vocabulary detection is to identify novel objects based on arbitrary textual descriptions. In this paper, we address open-vocabulary 3D point-cloud detection by a dividing-and-conquering strategy, which involves: 1) developing a point-cloud detector that can learn a general representation for localizing various objects, and 2) connecting textual and point-cloud representations to enable the detector to classify novel object categories based on text prompting. Specifically, we resort to rich image pre-trained models, by which the point-cloud detector learns localizing objects under the supervision of predicted 2D bounding boxes from 2D pre-trained detectors. Moreover, we propose a novel de-biased triplet cross-modal contrastive learning to connect the modalities of image, point-cloud and text, thereby enabling the point-cloud detector to benefit from vision-language pre-trained models,i.e.,CLIP. The novel use of image and vision-language pre-trained models for point-cloud detectors allows for open-vocabulary 3D object detection without the need for 3D annotations. Experiments demonstrate that the proposed method improves at least 3.03 points and 7.47 points over a wide range of baselines on the ScanNet and SUN RGB-D datasets, respectively. Furthermore, we provide a comprehensive analysis to explain why our approach works.

cs.CV↗

arXiv2022

Observing the Ping-pong Modality of Isospin Degree of Freedom in Cluster Emission from Heavy Ion Reactions

Two-body correlations of the isotope-resolved light and heavy clusters are measured in $^{86}$Kr+$^{\rm 208}$Pb reactions at 25 MeV/u. The yield and kinetic variables of the $A=3$ isobars, triton and $^3$He, are analyzed in coincidence with the heavy clusters of $7\le A \le 14$ emitted at the earlier chance. While the velocity spectra of both triton and $^3$He exhibit scaling behavior over the type of the heavy clusters, the yield ratios of ${\rm t/^3He}$ correlate reversely to the neutron-to-proton ratio $N/Z$ of the latter, showing the ping-pong modality of the $N/Z$ of emitted clusters. The commonality that the $N/Z$ of the residues keeps the initial system value is extended to the cluster emission in heavy ion reactions. The comparison of transport model calculations to the data is discussed.

nucl-ex↗

arXiv2022

Uncertainty Guided Depth Fusion for Spike Camera

Depth estimation is essential for various important real-world applications such as autonomous driving. However, it suffers from severe performance degradation in high-velocity scenario since traditional cameras can only capture blurred images. To deal with this problem, the spike camera is designed to capture the pixel-wise luminance intensity at high frame rate. However, depth estimation with spike camera remains very challenging using traditional monocular or stereo depth estimation algorithms, which are based on the photometric consistency. In this paper, we propose a novel Uncertainty-Guided Depth Fusion (UGDF) framework to fuse the predictions of monocular and stereo depth estimation networks for spike camera. Our framework is motivated by the fact that stereo spike depth estimation achieves better results at close range while monocular spike depth estimation obtains better results at long range. Therefore, we introduce a dual-task depth estimation architecture with a joint training strategy and estimate the distributed uncertainty to fuse the monocular and stereo results. In order to demonstrate the advantage of spike depth estimation over traditional camera depth estimation, we contribute a spike-depth dataset named CitySpike20K, which contains 20K paired samples, for spike depth estimation. UGDF achieves state-of-the-art results on CitySpike20K, surpassing all monocular or stereo spike depth estimation baselines. We conduct extensive experiments to evaluate the effectiveness and generalization of our method on CitySpike20K. To the best of our knowledge, our framework is the first dual-task fusion framework for spike camera depth estimation. Code and dataset will be released.

cs.CV↗

arXiv2022

MTTrans: Cross-Domain Object Detection with Mean-Teacher Transformer

Recently, DEtection TRansformer (DETR), an end-to-end object detection pipeline, has achieved promising performance. However, it requires large-scale labeled data and suffers from domain shift, especially when no labeled data is available in the target domain. To solve this problem, we propose an end-to-end cross-domain detection Transformer based on the mean teacher framework, MTTrans, which can fully exploit unlabeled target domain data in object detection training and transfer knowledge between domains via pseudo labels. We further propose the comprehensive multi-level feature alignment to improve the pseudo labels generated by the mean teacher framework taking advantage of the cross-scale self-attention mechanism in Deformable DETR. Image and object features are aligned at the local, global, and instance levels with domain query-based feature alignment (DQFA), bi-level graph-based prototype alignment (BGPA), and token-wise image feature alignment (TIFA). On the other hand, the unlabeled target domain data pseudo-labeled and available for the object detection training by the mean teacher framework can lead to better feature extraction and alignment. Thus, the mean teacher framework and the comprehensive multi-level feature alignment can be optimized iteratively and mutually based on the architecture of Transformers. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in three domain adaptation scenarios, especially the result of Sim10k to Cityscapes scenario is remarkably improved from 52.6 mAP to 57.9 mAP. Code will be released.

cs.CV↗

arXiv2022

Open-Vocabulary 3D Detection via Image-level Class and Debiased Cross-modal Contrastive Learning

Current point-cloud detection methods have difficulty detecting the open-vocabulary objects in the real world, due to their limited generalization capability. Moreover, it is extremely laborious and expensive to collect and fully annotate a point-cloud detection dataset with numerous classes of objects, leading to the limited classes of existing point-cloud datasets and hindering the model to learn general representations to achieve open-vocabulary point-cloud detection. As far as we know, we are the first to study the problem of open-vocabulary 3D point-cloud detection. Instead of seeking a point-cloud dataset with full labels, we resort to ImageNet1K to broaden the vocabulary of the point-cloud detector. We propose OV-3DETIC, an Open-Vocabulary 3D DETector using Image-level Class supervision. Specifically, we take advantage of two modalities, the image modality for recognition and the point-cloud modality for localization, to generate pseudo labels for unseen classes. Then we propose a novel debiased cross-modal contrastive learning method to transfer the knowledge from image modality to point-cloud modality during training. Without hurting the latency during inference, OV-3DETIC makes the point-cloud detector capable of achieving open-vocabulary detection. Extensive experiments demonstrate that the proposed OV-3DETIC achieves at least 10.77 % mAP improvement (absolute value) and 9.56 % mAP improvement (absolute value) by a wide range of baselines on the SUN-RGBD dataset and ScanNet dataset, respectively. Besides, we conduct sufficient experiments to shed light on why the proposed OV-3DETIC works.

cs.CV↗

arXiv2022

Center-of-Mass-based Robust Grasp Pose Adaptation Using RGBD Camera and Force/Torque Sensing

Object dropping may occur when the robotic arm grasps objects with uneven mass distribution due to additional moments generated by objects' gravity. To solve this problem, we present a novel work that does not require extra wrist and tactile sensors and large amounts of experiments for learning. First, we obtain the center-of-mass position of the rod object using the widely fixed joint torque sensors on the robot arm and RGBD camera. Further, we give the strategy of grasping to improve grasp stability. Simulation experiments are performed in "Mujoco". Results demonstrate that our work is effective in enhancing grasping robustness.

cs.RO↗