arXiv subjects

Wei Ji

Publications and source records attributed to Wei Ji.

At least 163 records · Page 9Linked to original sources

arXiv2024

Observation of unconventional van der Waals multiferroics near room temperature

The search for two-dimensional (2D) van der Waals (vdW) multiferroics is an exciting yet challenging endeavor. Room-temperature 2D vdW few-layer multiferroic is a much bigger insurmountable obstacle. Here we report the discovery of an unconventional 2D vdW multiferroic with out-of-plane ferroelectric polarization and long-range magnetic orders in trilayer NiI2 device from 10 K to 295 K. The evolutions of magnetic domains with magnetic field, and the evolutions between ferroelectric and antiferroelectric phase have been unambiguously observed. More significantly, we realize a robust mutual control of magnetism and ferroelectricity at room temperature. The magnetic domains are manipulated by a small voltage ranging from 1 V to 6 V at 0 T and 295 K. This work opens opportunities for exploring multiferroic physics at the limit of few atomic layers.

cond-mat.mtrl-sci↗

arXiv2024

Composed Image Retrieval with Text Feedback via Multi-grained Uncertainty Regularization

We investigate composed image retrieval with text feedback. Users gradually look for the target of interest by moving from coarse to fine-grained feedback. However, existing methods merely focus on the latter, i.e., fine-grained search, by harnessing positive and negative pairs during training. This pair-based paradigm only considers the one-to-one distance between a pair of specific points, which is not aligned with the one-to-many coarse-grained retrieval process and compromises the recall rate. In an attempt to fill this gap, we introduce a unified learning approach to simultaneously modeling the coarse- and fine-grained retrieval by considering the multi-grained uncertainty. The key idea underpinning the proposed method is to integrate fine- and coarse-grained retrieval as matching data points with small and large fluctuations, respectively. Specifically, our method contains two modules: uncertainty modeling and uncertainty regularization. (1) The uncertainty modeling simulates the multi-grained queries by introducing identically distributed fluctuations in the feature space. (2) Based on the uncertainty modeling, we further introduce uncertainty regularization to adapt the matching objective according to the fluctuation range. Compared with existing methods, the proposed strategy explicitly prevents the model from pushing away potential candidates in the early stage, and thus improves the recall rate. On the three public datasets, i.e., FashionIQ, Fashion200k, and Shoes, the proposed method has achieved +4.03%, +3.38%, and +2.40% Recall@50 accuracy over a strong baseline, respectively.

cs.CV↗

arXiv2024

Panoptic Scene Graph Generation with Semantics-Prototype Learning

Panoptic Scene Graph Generation (PSG) parses objects and predicts their relationships (predicate) to connect human language and visual scenes. However, different language preferences of annotators and semantic overlaps between predicates lead to biased predicate annotations in the dataset, i.e. different predicates for same object pairs. Biased predicate annotations make PSG models struggle in constructing a clear decision plane among predicates, which greatly hinders the real application of PSG models. To address the intrinsic bias above, we propose a novel framework named ADTrans to adaptively transfer biased predicate annotations to informative and unified ones. To promise consistency and accuracy during the transfer process, we propose to measure the invariance of representations in each predicate class, and learn unbiased prototypes of predicates with different intensities. Meanwhile, we continuously measure the distribution changes between each presentation and its prototype, and constantly screen potential biased data. Finally, with the unbiased predicate-prototype representation embedding space, biased annotations are easily identified. Experiments show that ADTrans significantly improves the performance of benchmark models, achieving a new state-of-the-art performance, and shows great generalization and effectiveness on multiple datasets.

cs.CV↗

arXiv2024

Animal3D: A Comprehensive Dataset of 3D Animal Pose and Shape

Accurately estimating the 3D pose and shape is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. However, research in this area is held back by the lack of a comprehensive and diverse dataset with high-quality 3D pose and shape annotations. In this paper, we propose Animal3D, the first comprehensive dataset for mammal animal 3D pose and shape estimation. Animal3D consists of 3379 images collected from 40 mammal species, high-quality annotations of 26 keypoints, and importantly the pose and shape parameters of the SMAL model. All annotations were labeled and checked manually in a multi-stage process to ensure highest quality results. Based on the Animal3D dataset, we benchmark representative shape and pose estimation models at: (1) supervised learning from only the Animal3D data, (2) synthetic to real transfer from synthetically generated images, and (3) fine-tuning human pose and shape estimation models. Our experimental results demonstrate that predicting the 3D shape and pose of animals across species remains a very challenging task, despite significant advances in human pose estimation. Our results further demonstrate that synthetic pre-training is a viable strategy to boost the model performance. Overall, Animal3D opens new directions for facilitating future research in animal 3D pose and shape estimation, and is publicly available.

cs.CV↗

arXiv2024

Levitated ferromagnetic magnetometer with energy resolution well below $\hbar$

A quantum limit on the measurement of magnetic field has been recently pointed out, stating that the so-called Energy Resolution $E_\mathrm{R}$ is bounded to $E_\mathrm{R} \gtrsim \hbar$. This limit holds indeed true for the vast majority of existing quantum magnetometers, including SQUIDs, solid state spins and optically pumped atomic magnetometers. However, it can be surpassed by highly correlated spin systems, as recently demonstrated with a single-domain spinor Bose-Einstein Condensate. Here we show that similar and potentially much better resolution can be achieved with a hard ferromagnet levitated above a superconductor at cryogenic temperature. We demonstrate $E_\mathrm{R}=\left( 0.064 \pm 0.010 \right) \, \hbar$ and anticipate that $E_\mathrm{R}<10^{-3} \, \hbar$ is within reach with near-future improvements. This finding opens the way to new applications in condensed matter, biophysics and fundamental science. In particular, we propose an experiment to search for axionlike dark matter and project a sensitivity orders of magnitude better than in previous searches.

quant-ph↗

arXiv2023

Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation

The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation due to its impressive capabilities in various segmentation tasks and its prompt-based interface. However, recent studies and individual experiments have shown that SAM underperforms in medical image segmentation, since the lack of the medical specific knowledge. This raises the question of how to enhance SAM's segmentation capability for medical images. In this paper, instead of fine-tuning the SAM model, we propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model using a light yet effective adaptation technique. In Med-SA, we propose Space-Depth Transpose (SD-Trans) to adapt 2D SAM to 3D medical images and Hyper-Prompting Adapter (HyP-Adpt) to achieve prompt-conditioned adaptation. We conduct comprehensive evaluation experiments on 17 medical image segmentation tasks across various image modalities. Med-SA outperforms several state-of-the-art (SOTA) medical image segmentation methods, while updating only 2\% of the parameters. Our code is released at https://github.com/KidsWithTokens/Medical-SAM-Adapter.

cs.CV↗

arXiv2023

MedSegDiff-V2: Diffusion based Medical Image Segmentation with Transformer

The Diffusion Probabilistic Model (DPM) has recently gained popularity in the field of computer vision, thanks to its image generation applications, such as Imagen, Latent Diffusion Models, and Stable Diffusion, which have demonstrated impressive capabilities and sparked much discussion within the community. Recent investigations have further unveiled the utility of DPM in the domain of medical image analysis, as underscored by the commendable performance exhibited by the medical image segmentation model across various tasks. Although these models were originally underpinned by a UNet architecture, there exists a potential avenue for enhancing their performance through the integration of vision transformer mechanisms. However, we discovered that simply combining these two models resulted in subpar performance. To effectively integrate these two cutting-edge techniques for the Medical image segmentation, we propose a novel Transformer-based Diffusion framework, called MedSegDiff-V2. We verify its effectiveness on 20 medical image segmentation tasks with different image modalities. Through comprehensive evaluation, our approach demonstrates superiority over prior state-of-the-art (SOTA) methodologies. Code is released at https://github.com/KidsWithTokens/MedSegDiff

eess.IV↗

arXiv2023

NExT-Chat: An LMM for Chat, Detection and Segmentation

The development of large language models (LLMs) has greatly advanced the field of multimodal understanding, leading to the emergence of large multimodal models (LMMs). In order to enhance the level of visual comprehension, recent studies have equipped LMMs with region-level understanding capabilities by representing object bounding box coordinates as a series of text sequences (pix2seq). In this paper, we introduce a novel paradigm for object location modeling called pix2emb method, where we ask the LMM to output the location embeddings and then decode them with different decoders. This paradigm allows us to use different location formats (such as bounding boxes and masks) in multimodal conversations. Leveraging the proposed pix2emb method, we train an LMM named NExT-Chat and demonstrate its capability of handling multiple tasks like visual grounding, region captioning, and grounded reasoning. Comprehensive experiments show the effectiveness of our NExT-Chat on various tasks, e.g., NExT-Chat (87.7) vs. Shikra (86.9) on POPE-Random, NExT-Chat (68.9) vs. LISA (67.9) on referring expression segmentation task, and NExT-Chat (79.6) vs. Kosmos-2 (62.3) on region caption task. The code and model are released at https://github.com/NExT-ChatV/NExT-Chat.

cs.CV↗

arXiv2023

Identification of Carbon Stars from LAMOST DR7

Carbon stars are excellent kinematic tracers of galaxies and play important roles in understanding the evolution of the Galaxy. Therefore, it is worthwhile to search for them in a large amount of spectra. In this work, we build a new carbon star catalog based on the LAMOST DR7 spectra. The catalog contains 4542 spectra of 3546 carbon stars, identified through line index and near-infrared color-color diagrams. Through visual inspection of the spectra, we further subclassify them into 925 C--H, 384 C--R, 608 C--N, and 1292 Ba stars. However, 437 stars could not be sub-classified due to their low signal-to-noise. Moreover, by comparing with LAMOST DR7 pipeline we find 567 more carbon stars and visually sub-classify them. We find that on the $J-H$ vs. $H-K_{\rm s}$ two-color diagram, C--N stars can be reliably distinguished from the other three sub-types. Additionally, by utilizing the Gaia distance, we study the distribution of carbon stars in the H-R diagram and identify 258 dwarf carbon stars by the criterion $M_{\rm G}>$5.0\,mag. Finally, we present the spatial distribution in Galactic coordinates of the 3546 carbon stars. The majority of C-N, C-R, and Ba stars are distributed at low Galactic latitudes, while most C--H and dC stars distribute at high Galactic latitudes.

astro-ph.GA↗

arXiv2023

Domain-wise Invariant Learning for Panoptic Scene Graph Generation

Panoptic Scene Graph Generation (PSG) involves the detection of objects and the prediction of their corresponding relationships (predicates). However, the presence of biased predicate annotations poses a significant challenge for PSG models, as it hinders their ability to establish a clear decision boundary among different predicates. This issue substantially impedes the practical utility and real-world applicability of PSG models. To address the intrinsic bias above, we propose a novel framework to infer potentially biased annotations by measuring the predicate prediction risks within each subject-object pair (domain), and adaptively transfer the biased annotations to consistent ones by learning invariant predicate representation embeddings. Experiments show that our method significantly improves the performance of benchmark models, achieving a new state-of-the-art performance, and shows great generalization and effectiveness on PSG dataset.

cs.CV↗

arXiv2023

VPGTrans: Transfer Visual Prompt Generator across LLMs

While developing a new multimodal LLM (MLLM) by pre-training on tremendous image-text pairs from scratch can be exceedingly resource-consuming, connecting an existing LLM with a comparatively lightweight visual prompt generator (VPG) becomes a feasible paradigm. However, further tuning the VPG part of the MLLM still suffers from indispensable computational costs, i.e., requiring thousands of GPU hours and millions of training data. One alternative solution is to transfer an existing VPG from any existing MLLMs for the target MLLM. In this work, we for the first time investigate the VPG transferability across LLMs, and explore a solution to reduce the cost of VPG transfer. We first study the VPG transfer across different LLM sizes (e.g., small-to-large), and across different LLM types, through which we diagnose the key factors to maximize the transfer efficiency. Based on our observation, we design a two-stage transfer framework named VPGTrans, which is simple yet highly effective. Through extensive experiments, we demonstrate that VPGTrans helps significantly speed up the transfer learning process without compromising performance. Remarkably, it helps achieve the VPG transfer from BLIP-2 OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and 10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from scratch. Further, a series of intriguing findings and potential rationales behind them are provided and discussed. Finally, we showcase the practical value of our VPGTrans approach, by customizing two novel MLLMs, including VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

cs.CV↗

arXiv2023

A Brief Review of Single Event Burnout Failure Mechanisms and Design Tolerances of Silicon Carbide MOSFETs

Radiation hardening of the MOSFET is of the highest priority for sustaining high-power systems in the space radiation environment. SiC-based power electronics are being looked at as a strong alternative for high power spaceborne power electronic systems. The SiC MOSFET has been shown to be most prone to SEB of the radiation effects. The knowledge of SiC MOSFET device degradation and failure mechanisms are reviewed. Additionally, the viability of rad-tolerant SiC MOSFET designs and the methods of SEB simulation are evaluated. A merit system is proposed to consider the performance of radiation tolerance and nominal electrical performance. Criteria needed for high-fidelity SEB simulation are also reviewed. This paper stands as a necessary analytical review to intercede the development of rad-hard power devices for space and extreme environment applications.

physics.ins-det↗

arXiv2023

Towards Complex-query Referring Image Segmentation: A Novel Benchmark

Referring Image Understanding (RIS) has been extensively studied over the past decade, leading to the development of advanced algorithms. However, there has been a lack of research investigating how existing algorithms should be benchmarked with complex language queries, which include more informative descriptions of surrounding objects and backgrounds (\eg \textit{"the black car."} vs. \textit{"the black car is parking on the road and beside the bus."}). Given the significant improvement in the semantic understanding capability of large pre-trained models, it is crucial to take a step further in RIS by incorporating complex language that resembles real-world applications. To close this gap, building upon the existing RefCOCO and Visual Genome datasets, we propose a new RIS benchmark with complex queries, namely \textbf{RIS-CQ}. The RIS-CQ dataset is of high quality and large scale, which challenges the existing RIS with enriched, specific and informative queries, and enables a more realistic scenario of RIS research. Besides, we present a nichetargeting method to better task the RIS-CQ, called dual-modality graph alignment model (\textbf{\textsc{DuMoGa}}), which outperforms a series of RIS methods.

cs.CV↗

arXiv2023

Spin-resolved imaging of atomic-scale helimagnetism in monolayer NiI2

Identifying intrinsic noncollinear magnetic order in monolayer van der Waals (vdW) crystals is highly desirable for understanding the delicate magnetic interactions at reduced spatial constraints and miniaturized spintronic applications, but remains elusive in experiments. Here, we achieved spin-resolved imaging of helimagnetism at atomic scale in monolayer NiI2 crystals, that were grown on graphene-covered SiC(0001) substrate, using spin-polarized scanning tunneling microscopy. Our experiments identify the existence of a spin spiral state with canted plane in monolayer NiI2. The spin modulation Q vector of the spin spiral is determined as (0.2203, 0, 0), which is different from its bulk value or its in-plane projection, but agrees well with our first principles calculations. The spin spiral surprisingly indicates collective spin switching behavior under magnetic field, whose origin is ascribed to the incommensurability between the spin spiral and the crystal lattice. Our work unambiguously identifies the helimagnetic state in monolayer NiI2, paving the way for illuminating its expected type-II multiferroic order and developing spintronic devices based on vdW magnets.

cond-mat.mes-hall↗

arXiv2023

Intralayer Negative Poisson's Ratio in Two-Dimensional Black Arsenic by Strain Engineering

Negative Poisson's ratio as the anomalous characteristic generally exists in artificial architectures, such as re-entrant and honeycomb structures. The structures with negative Poisson's ratio have attracted intensive attention due to their unique auxetic effect and many promising applications in shear resistant and energy absorption fields. However, experimental observation of negative Poisson's ratio in natural materials barely happened, although various two-dimensional layered materials are predicted in theory. Herein, we report the anisotropic Raman response and the intrinsic intralayer negative Poisson's ratio of two-dimensional natural black arsenic (b-As) via strain engineering strategy. The results were evident by the detailed Raman spectrum of b-As under uniaxial strain together with density functional theory calculations. It is found that b-As was softer along the armchair than zigzag direction. The anisotropic mechanical features and van der Waals interactions play essential roles in strain-dependent Raman shifts and negative Poisson's ratio in the natural b-As along zigzag direction. This work may shed a light on the mechanical properties and potential applications of two-dimensional puckered materials.

cond-mat.mtrl-sci↗

arXiv2023

Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World

Scene Graph Generation (SGG) aims to extract relationships in images for vision understanding. Although recent works have made steady progress on SGG, they still suffer long-tail distribution issues that tail-predicates are more costly to train and hard to distinguish due to a small amount of annotated data compared to frequent predicates. Existing re-balancing strategies try to handle it via prior rules but are still confined to pre-defined conditions, which are not scalable for various models and datasets. In this paper, we propose a Cross-modal prediCate boosting (CaCao) framework, where a visually-prompted language model is learned to generate diverse fine-grained predicates in a low-resource way. The proposed CaCao can be applied in a plug-and-play fashion and automatically strengthen existing SGG to tackle the long-tailed problem. Based on that, we further introduce a novel Entangled cross-modal prompt approach for open-world predicate scene graph generation (Epic), where models can generalize to unseen predicates in a zero-shot manner. Comprehensive experiments on three benchmark datasets show that CaCao consistently boosts the performance of multiple scene graph generation models in a model-agnostic way. Moreover, our Epic achieves competitive performance on open-world predicate prediction. The data and code for this paper are publicly available.

cs.CV↗

arXiv2023

Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Models

Prompt tuning, a recently emerging paradigm, enables the powerful vision-language pre-training models to adapt to downstream tasks in a parameter -- and data -- efficient way, by learning the ``soft prompts'' to condition frozen pre-training models. Though effective, it is particularly problematic in the few-shot scenario, where prompt tuning performance is sensitive to the initialization and requires a time-consuming process to find a good initialization, thus restricting the fast adaptation ability of the pre-training models. In addition, prompt tuning could undermine the generalizability of the pre-training models, because the learnable prompt tokens are easy to overfit to the limited training samples. To address these issues, we introduce a novel Gradient-RegulAted Meta-prompt learning (GRAM) framework that jointly meta-learns an efficient soft prompt initialization for better adaptation and a lightweight gradient regulating function for strong cross-domain generalizability in a meta-learning paradigm using only the unlabeled image-text pre-training data. Rather than designing a specific prompt tuning method, our GRAM can be easily incorporated into various prompt tuning methods in a model-agnostic way, and comprehensive experiments show that GRAM brings about consistent improvement for them in several settings (i.e., few-shot learning, cross-domain generalization, cross-dataset generalization, etc.) over 11 datasets. Further, experiments show that GRAM enables the orthogonal methods of textual and visual prompt tuning to work in a mutually-enhanced way, offering better generalizability beyond the uni-modal prompt tuning methods.

cs.CV↗

arXiv2023

Online Distillation-enhanced Multi-modal Transformer for Sequential Recommendation

Multi-modal recommendation systems, which integrate diverse types of information, have gained widespread attention in recent years. However, compared to traditional collaborative filtering-based multi-modal recommendation systems, research on multi-modal sequential recommendation is still in its nascent stages. Unlike traditional sequential recommendation models that solely rely on item identifier (ID) information and focus on network structure design, multi-modal recommendation models need to emphasize item representation learning and the fusion of heterogeneous data sources. This paper investigates the impact of item representation learning on downstream recommendation tasks and examines the disparities in information fusion at different stages. Empirical experiments are conducted to demonstrate the need to design a framework suitable for collaborative learning and fusion of diverse information. Based on this, we propose a new model-agnostic framework for multi-modal sequential recommendation tasks, called Online Distillation-enhanced Multi-modal Transformer (ODMT), to enhance feature interaction and mutual learning among multi-source input (ID, text, and image), while avoiding conflicts among different features during training, thereby improving recommendation accuracy. To be specific, we first introduce an ID-aware Multi-modal Transformer module in the item representation learning stage to facilitate information interaction among different features. Secondly, we employ an online distillation training strategy in the prediction optimization stage to make multi-source data learn from each other and improve prediction robustness. Experimental results on a stream media recommendation dataset and three e-commerce recommendation datasets demonstrate the effectiveness of the proposed two modules, which is approximately 10% improvement in performance compared to baseline models.

cs.IR↗