arXiv subjects

Haiping Wu

Publications and source records attributed to Haiping Wu.

16 recordsLinked to original sources

arXiv2024

Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

We present Florence-VL, a new family of multimodal large language models (MLLMs) with enriched visual representations produced by Florence-2, a generative vision foundation model. Unlike the widely used CLIP-style vision transformer trained by contrastive learning, Florence-2 can capture different levels and aspects of visual features, which are more versatile to be adapted to diverse downstream tasks. We propose a novel feature-fusion architecture and an innovative training recipe that effectively integrates Florence-2's visual features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted from different depths and under multiple prompts. Our model training is composed of end-to-end pretraining of the whole model followed by finetuning of the projection layer and the LLM, on a carefully designed recipe of diverse open-source datasets that include high-quality image captions and instruction-tuning pairs. Our quantitative analysis and visualization of Florence-VL's visual features show its advantages over popular vision encoders on vision-language alignment, where the enriched depth and breath play important roles. Florence-VL achieves significant improvements over existing state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks covering general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, etc. To facilitate future research, our models and the complete training recipe are open-sourced. https://github.com/JiuhaiChen/Florence-VL

cs.CV↗

arXiv2023

Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks

We introduce Florence-2, a novel vision foundation model with a unified, prompt-based representation for a variety of computer vision and vision-language tasks. While existing large vision models excel in transfer learning, they struggle to perform a diversity of tasks with simple instructions, a capability that implies handling the complexity of various spatial hierarchy and semantic granularity. Florence-2 was designed to take text-prompt as task instructions and generate desirable results in text forms, whether it be captioning, object detection, grounding or segmentation. This multi-task learning setup demands large-scale, high-quality annotated data. To this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive visual annotations on 126 million images, using an iterative strategy of automated image annotation and model refinement. We adopted a sequence-to-sequence structure to train Florence-2 to perform versatile and comprehensive vision tasks. Extensive evaluations on numerous tasks demonstrated Florence-2 to be a strong vision foundation model contender with unprecedented zero-shot and fine-tuning capabilities.

cs.CV↗

arXiv2021

Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency

Recent works have advanced the performance of self-supervised representation learning by a large margin. The core among these methods is intra-image invariance learning. Two different transformations of one image instance are considered as a positive sample pair, where various tasks are designed to learn invariant representations by comparing the pair. Analogically, for video data, representations of frames from the same video are trained to be closer than frames from other videos, i.e. intra-video invariance. However, cross-video relation has barely been explored for visual representation learning. Unlike intra-video invariance, ground-truth labels of cross-video relation is usually unavailable without human labors. In this paper, we propose a novel contrastive learning method which explores the cross-video relation by using cycle-consistency for general image representation learning. This allows to collect positive sample pairs across different video instances, which we hypothesize will lead to higher-level semantics. We validate our method by transferring our image representation to multiple downstream tasks including visual object tracking, image classification, and action recognition. We show significant improvement over state-of-the-art contrastive learning methods. Project page is available at https://happywu.github.io/cycle_contrast_video.

cs.CV↗

arXiv2021

CvT: Introducing Convolutions to Vision Transformers

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{https://github.com/leoxiaobin/CvT}.

cs.CV↗

arXiv2020

Transition between half-metal and ferromagnetic semiconductor induced by silicon vacancy in epitaxial silicene

Since the inevitability in experimental synthesis, defects show great importance to many materials. They will deeply regulate the properties of the materials, and then affect the further applications. Thus, exploring the effects of defects on the properties of materials is desired. Here, by using first-principles calculations, we systematically studied the effect of silicon vacancy defects on the properties of silicene generated on Nterminated cubic boron nitride (111) surface. It is found that the introduction of silicon vacancy would trigger transition between half-metal and ferromagnetic semiconductor. With small vacancy ratios of 1/36 and 1/24, the ground-state of the samples would behave as ferromagnetic semiconductors, and the band gaps are about 1.25 and 0.95 eV, respectively. When the vacancy ratio is increased up to 1/6, the sample would turn into a ferromagnetic half-metal with a half-metallic gap of around 0.15 eV. The change of the electronic structure of the samples is driven by the different electron transfer between silicon layer and substrate, i.e., there will be different amount of electrons transferred from the silicon layer to the substrate when the vacancy ratio is altered. This work would open a new way to regulate the properties of materials and extend applications in nanoelectronic field.

cond-mat.mtrl-sci↗

arXiv2020

Transient Dynamics of a Miura-Origami Tube during Free Deployment

With excellent folding-induced deformability and shape reconfigurability, origami-based designs have shown great potentials in developing deployable structures. Noting that origami deployment is essentially a dynamic process, while its dynamical behaviors remain largely unexplored owing to the challenges in modeling. This research aims at advancing the state of the art of origami deployable structures by exploring the transient dynamics under free deployment, with the Miura-origami tube being selected as the object of study because it possesses relatively simple geometry, exceptional kinematic properties, and wide applications. In detail, a preliminary free deployment test is performed, which indicates that the transient oscillation in the transverse direction is nonnegligible and the tube deployment is no longer a single-degree-of-freedom (SDOF) mechanism. Based on experimental observations, four assumptions are made for modeling purposes, and a 2N-DOF dynamic model is established for an N-cell Miura-origami tube to predict the transient oscillations in both the deploying and the transverse directions. Employing the settling times and the overshoot values as the transient dynamic indexes, a comprehensive parameter study is then carried out. It reveals that both the physical and geometrical parameters will significantly affect the transient deploying dynamics, with some of the parameter dependence relationships being counter-intuitive. The results show that the relationships between the transient dynamic behaviors and the examined parameters are sometimes contradictory in the deploying and the transverse directions, suggesting the necessity of a compromise in design.

physics.app-ph↗

arXiv2020

Moderate bandgap and high carrier mobility simultaneously realized in bilayer silicene by oxidation

Semiconductors simultaneously possessing high carrier mobility, moderate bandgap, and ambient environment stability are so important for the modern industry, and Si-based semiconducting materials can match well with the previous silicon based electronic components. Thus, searching for such Si-based semiconductors has been one hot project due to the lack of them nowadays. Here, with the help of density functional theory, we found that the oxidized bilayer silicene exhibits high carrier mobility with a moderate direct bandgap of 1.02 eV. The high carrier mobility is caused by the remaining of big pi bond, and the moderate bandgap is opened by the saturation of dangling Si 3p bonds. Originated from the formation of strong Si-O and Si-Si bonds, the sample exhibits strong thermodynamic and dynamical stabilities. Our work indicates that the oxidized bilayer silicene has many potential applications in modern electronic fields.

cond-mat.mes-hall↗

arXiv2019

Sequence Level Semantics Aggregation for Video Object Detection

Video objection detection (VID) has been a rising research direction in recent years. A central issue of VID is the appearance degradation of video frames caused by fast motion. This problem is essentially ill-posed for a single frame. Therefore, aggregating features from other frames becomes a natural choice. Existing methods rely heavily on optical flow or recurrent neural networks for feature aggregation. However, these methods emphasize more on the temporally nearby frames. In this work, we argue that aggregating features in the full-sequence level will lead to more discriminative and robust features for video object detection. To achieve this goal, we devise a novel Sequence Level Semantics Aggregation (SELSA) module. We further demonstrate the close relationship between the proposed method and the classic spectral clustering method, providing a novel view for understanding the VID problem. We test the proposed method on the ImageNet VID and the EPIC KITCHENS dataset and achieve new state-of-the-art results. Our method does not need complicated postprocessing methods such as Seq-NMS or Tubelet rescoring, which keeps the pipeline simple and clean.

cs.CV↗

arXiv2019

Substrate-induced half-metallic property in epitaxial silicene

For most practical applications in electronic devices, two-dimensional materials should be transferred onto semiconducting or insulating substrates, since they are usually generated on metallic substrates. However, the transfer often leads to wrinkles, damages, contaminations and so on which would destroy the intrinsic properties of samples. Thus, generating two-dimensional materials directly on nonmetallic substrates has been a desirable goal for a long time. Here, via a swarm structure search method and density functional theory, we employed an insulating N-terminated cubic boron nitride(111) surface as a substrate for the generation of silicene. The result shows that the silicene behaves as a ferromagnetic half-metal because of the strong interaction between silicon and surface nitrogen atoms. The magnetic moments are mainly located on surface nitrogen sites without bonding silicon atoms and the value is about 0.12 uB. In spin-up channel, it behaves as a direct band gap semiconductor with a gap of around 1.35 eV, while it exhibits metallic characteristic in spin-down channel, and the half-metallic band gap is about 0.11 eV. Besides, both the magnetic and electronic properties are not sensitive to the external compressive strain. This work maybe open a way for the utility of silicene in spintronic field.

cond-mat.mtrl-sci↗

arXiv2018

Simple Baselines for Human Pose Estimation and Tracking

There has been significant progress on pose estimation and increasing interests on pose tracking in recent years. At the same time, the overall algorithm and system complexity increases as well, making the algorithm analysis and comparison more difficult. This work provides simple and effective baseline methods. They are helpful for inspiring and evaluating new ideas for the field. State-of-the-art results are achieved on challenging benchmarks. The code will be available at https://github.com/leoxiaobin/pose.pytorch.

cs.CV↗

arXiv2018

Discovery of Intrinsic Ferromagnetic Ferroelectricity in Transition Metal Halides Monolayer

The realization of multiferroics in nanostructures, combined with a large electric dipole and ferromagnetic ordering, could lead to new applications, such as high-density multi-state data storage. Although multiferroics have been broadly studied for decades, ferromagnetic ferroelectricity is rarely explored, especially in two-dimensional (2D) systems. Here we report the discovery of 2D ferromagnetic ferroelectricity in layered transition metal halide systems. On the basis of first-principles calculations, we reveal that charged CrBr3 monolayer exhibits in-plane multiferroicity, which is ensured by the combination of orbital and charge ordering as realized by the asymmetric Jahn-Teller distortions of octahedral Cr-Br6 units. As an example, we further show that (CrBr3)2Li is a ferromagnetic ferroelectric multiferroic. The explored phenomena and mechanism of multiferroics in this 2D system are not only useful for fundamental research in multiferroics but also enable a wide range of applications in nano-devices.

cond-mat.mtrl-sci↗

arXiv2017

Semimetallic single atomic layer silicon allotrope with Dirac fermions

Materials with Dirac point are so amazing since the charge carriers are massless and have an effective speed of light. Among the reported two-dimensional silicon allotropes, no one showing such exciting nature was proved experimentally. This fact motivates us to search for other such two-dimensional silicon allotropes. As a result, a new single atomic layer thin silicon allotrope was predicted by employing CALYPSO code in this work. This silicon allotrope is composed of eight- membered rings linked by Si-Si bonds and presents buckling formation. Expectedly, the electronic calculation reveals that there exists Dirac point at Fermi energy level. Furthermore, the ab initio molecular dynamics simulations displays that the original atomic configuration can be remained even at an extremely high temperature of 1000 K. We hope this work can expand the family of single atomic layer thin silicon allotropes with Dirac fermions.

cond-mat.mtrl-sci↗

arXiv2017

Atomically thin mononitrides SiN and GeN: new two-dimensional semiconducting materials

Low-dimensional Si-based semiconductors are unique materials that can both match well with the Si-based electronics and satisfy the demand of miniaturization in modern industry. Owing to the lack of such materials, many researchers put their efforts into this field. In this work, employing a swarm structure search method and density functional theory, we theoretically predict two-dimensional atomically thin mononitrides SiN and GeN, both of which present semiconducting nature. Furthermore study shows that SiN and GeN behave as indirect band gap semiconductors with the gap of 1.75 and 1.20 eV, respectively. The ab initio molecular dynamics calculation tells that both two mononitrides can exist stably even at extremely high temperature of 2000 K. Notably, electron mobilities are evaluated as 0.888x$10^3$ $cm^2V^{-1}s^{-1}$ and 0.413x$10^3$ $cm^2V^{-1}s^{-1}$ for SiN and GeN, respectively. The present work expands the family of low-dimensional Si-based semiconductors.

cond-mat.mtrl-sci↗

arXiv2017

Graphene-like quaternary compound SiBCN: a new wide direct band gap semiconductor predicted by a first-principles study

Due to the lack of two-dimensional silicon-based semiconductors and the fact that most of the components and devices are generated on single-crystal silicon or silicon-based substrates in modern industry, designing two-dimensional silicon-based semiconductors is highly desired. With the combination of a swarm structure search method and density functional theory in this work, a quaternary compound SiBCN with graphene-like structure is found and displays a wide direct band gap as expected. The band gap is of ~2.63 eV which is just between ~2.20 and ~3.39 eV of the highlighted semiconductors SiC and GaN. Notably, the further calculation reveals that SiBCN possesses high carrier mobility with ~5.14x10^3 and ~13.07x10^3 cm^2V^-1s^-1 for electron and hole, respectively. Furthermore, the ab initio molecular dynamics simulations also show that the graphene-like structure of SiBCN can be well kept even at an extremely high temperature of 2000 K. The present work tells that designing ulticomponent silicides may be a practicable way to search for new silicon-based low-dimensional semiconductors which can match well with the previous Si-based substrates.

cond-mat.mes-hall↗

arXiv2016

Quantum anomalous Hall effect in ferromagnetic transition metal halides

The quantum anomalous Hall (QAH) effect is a novel topological spintronic phenomenon arising from inherent magnetization and spin-orbit coupling. Various theoretical and experimental efforts have been devoted in search of robust intrinsic QAH insulators. However, up to now, it has only been observed in Cr or V doped (Bi,Sb)2Te3 film in experiments with very low working temperature. Based on the successful synthesis of transition metal halides, we use first-principles calculations to predict that RuI3 monolayer is an intrinsic ferromagnetic QAH insulator with a topologically nontrivial global band gap of 11 meV. This topologically nontrivial band gap at the Fermi level is due to its crystal symmetry, thus the QAH effect is robust. Its Curie temperature, estimated to be ~360 K using Monte-Carlo simulation, is above room temperature and higher than most of two-dimensional ferromagnetic thin films. We also discuss the manipulation of its exchange energy and nontrivial band gap by applying in-plane strain. Our work adds a new experimentally feasible member to the QAH insulator family, which is expected to have broad applications in nanoelectronics and spintronics.

physics.comp-ph↗

arXiv2012

Ring-like spin segregation of binary mixtures in a high-velocity rotating drum

We present molecular dynamics simulations on the segregation of binary mixtures in a high-velocity rotating drum. Depending on the ratio of particle's radius and density, the ring-like spin segregation patterns in radial direction show similarities to the Brazil-nut effect and its reverse form. The smaller and heavier particles are easier to accumulate towards the drum wall and the bigger and lighter ones towards the drum center. Furthermore, we quantify the effects of particle's radius and density on the segregation states and plot the phase diagram of segregation in the ${ρ_b}/{ρ_s}$ - ${r_b}/{r_s}$ space. The observed phenomena can also be well explained by the combined actions, i.e., the percolation effect and the buoyancy effect.

cond-mat.soft↗