arXiv subjects

Biao Wang

Publications and source records attributed to Biao Wang.

At least 73 records · Page 4Linked to original sources

arXiv2022

Motion Transformer for Unsupervised Image Animation

Image animation aims to animate a source image by using motion learned from a driving video. Current state-of-the-art methods typically use convolutional neural networks (CNNs) to predict motion information, such as motion keypoints and corresponding local transformations. However, these CNN based methods do not explicitly model the interactions between motions; as a result, the important underlying motion relationship may be neglected, which can potentially lead to noticeable artifacts being produced in the generated animation video. To this end, we propose a new method, the motion transformer, which is the first attempt to build a motion estimator based on a vision transformer. More specifically, we introduce two types of tokens in our proposed method: i) image tokens formed from patch features and corresponding position encoding; and ii) motion tokens encoded with motion information. Both types of tokens are sent into vision transformers to promote underlying interactions between them through multi-head self attention blocks. By adopting this process, the motion information can be better learned to boost the model performance. The final embedded motion tokens are then used to predict the corresponding motion keypoints and local transformations. Extensive experiments on benchmark datasets show that our proposed method achieves promising results to the state-of-the-art baselines. Our source code will be public available.

cs.CV↗

arXiv2022

Spatio-Temporal Relation Learning for Video Anomaly Detection

Anomaly identification is highly dependent on the relationship between the object and the scene, as different/same object actions in same/different scenes may lead to various degrees of normality and anomaly. Therefore, object-scene relation actually plays a crucial role in anomaly detection but is inadequately explored in previous works. In this paper, we propose a Spatial-Temporal Relation Learning (STRL) framework to tackle the video anomaly detection task. First, considering dynamic characteristics of the objects as well as scene areas, we construct a Spatio-Temporal Auto-Encoder (STAE) to jointly exploit spatial and temporal evolution patterns for representation learning. For better pattern extraction, two decoding branches are designed in the STAE module, i.e. an appearance branch capturing spatial cues by directly predicting the next frame, and a motion branch focusing on modeling the dynamics via optical flow prediction. Then, to well concretize the object-scene relation, a Relation Learning (RL) module is devised to analyze and summarize the normal relations by introducing the Knowledge Graph Embedding methodology. Specifically in this process, the plausibility of object-scene relation is measured by jointly modeling object/scene features and optimizable object-scene relation maps. Extensive experiments are conducted on three public datasets, and the superior performance over the state-of-the-art methods demonstrates the effectiveness of our method.

cs.CV↗

arXiv2022

Bond-Selective Intensity Diffraction Tomography

Recovering molecular information remains a grand challenge in the widely used holographic and computational imaging technologies. To address this challenge, we developed a computational mid-infrared photothermal microscope, termed Bond-selective Intensity Diffraction Tomography (BS-IDT). Based on a low-cost brightfield microscope with an add-on pulsed light source, BS-IDT recovers both infrared spectra and bond-selective 3D refractive index maps from intensity-only measurements. High-fidelity infrared fingerprint spectra extraction is validated. Volumetric chemical imaging of biological cells is demonstrated at a speed of ~20 seconds per volume, with a lateral and axial resolution of ~350 nm and ~1.1 micron, respectively. BS-IDT's application potential is investigated by chemically quantifying lipids stored in cancer cells and volumetric chemical imaging on Caenorhabditis elegans with a large field of view (~100 micron X 100 micron).

physics.optics↗

arXiv2022

Spatiotemporal Self-attention Modeling with Temporal Patch Shift for Action Recognition

Transformer-based methods have recently achieved great advancement on 2D image-based vision tasks. For 3D video-based tasks such as action recognition, however, directly applying spatiotemporal transformers on video data will bring heavy computation and memory burdens due to the largely increased number of patches and the quadratic complexity of self-attention computation. How to efficiently and effectively model the 3D self-attention of video data has been a great challenge for transformers. In this paper, we propose a Temporal Patch Shift (TPS) method for efficient 3D self-attention modeling in transformers for video-based action recognition. TPS shifts part of patches with a specific mosaic pattern in the temporal dimension, thus converting a vanilla spatial self-attention operation to a spatiotemporal one with little additional cost. As a result, we can compute 3D self-attention using nearly the same computation and memory cost as 2D self-attention. TPS is a plug-and-play module and can be inserted into existing 2D transformer models to enhance spatiotemporal feature learning. The proposed method achieves competitive performance with state-of-the-arts on Something-something V1 & V2, Diving-48, and Kinetics400 while being much more efficient on computation and memory cost. The source code of TPS can be found at https://github.com/MartinXM/TPS.

cs.CV↗

arXiv2022

High-throughput screening of piezo-photocatalytic materials for hydrogen production

Finding cost-effective and efficient photocatalytic materials able to catalyse the water splitting reaction under visible light is one of the greatest challenges in current environmental material science. Despite that many photocatalysts are already known in the context of green hydrogen production, strategies to systematically and rationally modify their optoelectronic properties to achieve desired photocatalytic performance are yet to be established. Piezoelectric materials react to mechanical stimuli by adjusting their band gaps and band alignments, thus offering a possible route to precise photocatalyst design. However, piezo-photocatalysts are relatively scarce and have been seldom investigated to date. Here, we present a high-throughput screening of piezo-photocatalytic materials performed over $\sim 1,000$ bulk piezoelectrics that relies on a simple electrostatic model and first-principles calculations. A total of $\sim 10$ previously overlooked binary and tertiary bulk compounds are theoretically identified as highly promising piezo-photocatalysts due to their appropriate optoelectronic properties and superb band alignment tunability driven by uniaxial strain.

cond-mat.mtrl-sci↗

arXiv2022

Approximate synchronization of coupled multi-valued logical networks

This article deals with the approximate synchronization of two coupled multi-valued logical networks. According to the initial state set from which both systems start, two kinds of approximate synchronization problem, local approximate synchronization and global approximate synchronization, are proposed for the first time. Three new notions: approximate synchronization state set (ASSS), the maximum approximate synchronization basin (MASB) and the shortest approximate synchronization time (SAST) are introduced and analyzed. Based on ASSS, several necessary and sufficient conditions are obtained for approximate synchronization. MASB, the set of all possible initial states, from which the systems are approximately synchronous, is investigated combining with the maximum invariant subset. And the calculation method of the SAST, associated with transient period, is presented. By virtue of MASB, pinning control scheme is investigated to make two coupled systems achieve global approximate synchronization. Furthermore, the related theories are also applied to the complete synchronization problem of $k$-valued ($k\geq2$) logical networks. Finally, four examples are given to illustrate the obtained results.

eess.SY↗

arXiv2022

SP-ViT: Learning 2D Spatial Priors for Vision Transformers

Recently, transformers have shown great potential in image classification and established state-of-the-art results on the ImageNet benchmark. However, compared to CNNs, transformers converge slowly and are prone to overfitting in low-data regimes due to the lack of spatial inductive biases. Such spatial inductive biases can be especially beneficial since the 2D structure of an input image is not well preserved in transformers. In this work, we present Spatial Prior-enhanced Self-Attention (SP-SA), a novel variant of vanilla Self-Attention (SA) tailored for vision transformers. Spatial Priors (SPs) are our proposed family of inductive biases that highlight certain groups of spatial relations. Unlike convolutional inductive biases, which are forced to focus exclusively on hard-coded local regions, our proposed SPs are learned by the model itself and take a variety of spatial relations into account. Specifically, the attention score is calculated with emphasis on certain kinds of spatial relations at each head, and such learned spatial foci can be complementary to each other. Based on SP-SA we propose the SP-ViT family, which consistently outperforms other ViT models with similar GFlops or parameters. Our largest model SP-ViT-L achieves a record-breaking 86.3% Top-1 accuracy with a reduction in the number of parameters by almost 50% compared to previous state-of-the-art model (150M for SP-ViT-L vs 271M for CaiT-M-36) among all ImageNet-1K models trained on 224x224 and fine-tuned on 384x384 resolution w/o extra data.

cs.CV↗

arXiv2022

Nonlinear beam self-maintaining effect in graded-index multimode fiber

Multimode fiber systems are desirable for industrial and scientific applications. As an interesting effect for the laser beam propagation in a multimode fiber, nonlinear Kerr beam cleanup has attracted considerable research interest due to the spatial beam compressing. However, its physical mechanisms, especially the influences of input conditions on its performances, remain unclear. Here, we report a new self-organized regime for the multimode beam propagation in a graded-index multimode fiber: when the input laser has a dominant mode in which most of the laser energy is concentrated, the beam profile can be maintaining in a well-defined structure similar to the input dominant mode in nonlinear regime, while it will evolve to an irregular pattern in linear regime. The existence and universality of this nonlinear beam self-maintaining effect have been verified by the experimental and numerical data. Our results also provide evidence that nonlinear Kerr effects can be the driving mechanism and nonlinear Kerr beam cleanup is a specific case of this effect. Further research into this spatial beam shaping effect may provide a new perspective to understand other multimode fiber nonlinearities.

physics.optics↗

arXiv2022

Biaxial strain engineering on the superconducting properties of MgB2 thin film

The effect of biaxial strain on the superconducting properties of MgB2 thin films was studied by first-principles calculations. The stability analyses by phonon dispersions show that biaxial strain as much as 7% can be applied onto MgB2 without inducing any imaginary frequency. The superconducting property calculations based on the frame of Migdal-Eliashberg theory successfully reproduce the two-gap superconductivity of MgB2. The results show that the tensile biaxial strain can increase the critical temperature of MgB2 while the compressive biaxial strain would decrease the critical temperature. The detailed microscopic mechanism of the biaxial strain effect on the superconducting properties was studied by calculations of electronic structures and phonon dispersions. The increased Tc is a combining result of the increased electron density at the Fermi level and the in-plane boron phonon softening. By means of high-throughput screening of proper substrates, it is found that most of the substrates would result in tensile strain in MgB2 film, which is in agreement with many experimental works. The results in this work provide detailed understanding of the biaxial strain engineering mechanism and demonstrate that biaxial strain engineering can be an effective way of tuning the superconducting properties of MgB2 and other similar materials.

cond-mat.mtrl-sci↗

arXiv2022

Dense Learning based Semi-Supervised Object Detection

Semi-supervised object detection (SSOD) aims to facilitate the training and deployment of object detectors with the help of a large amount of unlabeled data. Though various self-training based and consistency-regularization based SSOD methods have been proposed, most of them are anchor-based detectors, ignoring the fact that in many real-world applications anchor-free detectors are more demanded. In this paper, we intend to bridge this gap and propose a DenSe Learning (DSL) based anchor-free SSOD algorithm. Specifically, we achieve this goal by introducing several novel techniques, including an Adaptive Filtering strategy for assigning multi-level and accurate dense pixel-wise pseudo-labels, an Aggregated Teacher for producing stable and precise pseudo-labels, and an uncertainty-consistency-regularization term among scales and shuffled patches for improving the generalization capability of the detector. Extensive experiments are conducted on MS-COCO and PASCAL-VOC, and the results show that our proposed DSL method records new state-of-the-art SSOD performance, surpassing existing methods by a large margin. Codes can be found at \textcolor{blue}{https://github.com/chenbinghui1/DSL}.

cs.CV↗

arXiv2022

Structure-Aware Motion Transfer with Deformable Anchor Model

Given a source image and a driving video depicting the same object type, the motion transfer task aims to generate a video by learning the motion from the driving video while preserving the appearance from the source image. In this paper, we propose a novel structure-aware motion modeling approach, the deformable anchor model (DAM), which can automatically discover the motion structure of arbitrary objects without leveraging their prior structure information. Specifically, inspired by the known deformable part model (DPM), our DAM introduces two types of anchors or keypoints: i) a number of motion anchors that capture both appearance and motion information from the source image and driving video; ii) a latent root anchor, which is linked to the motion anchors to facilitate better learning of the representations of the object structure information. Moreover, DAM can be further extended to a hierarchical version through the introduction of additional latent anchors to model more complicated structures. By regularizing motion anchors with latent anchor(s), DAM enforces the correspondences between them to ensure the structural information is well captured and preserved. Moreover, DAM can be learned effectively in an unsupervised manner. We validate our proposed DAM for motion transfer on different benchmark datasets. Extensive experiments clearly demonstrate that DAM achieves superior performance relative to existing state-of-the-art methods.

cs.CV↗

arXiv2022

Learning Pixel-Level Distinctions for Video Highlight Detection

The goal of video highlight detection is to select the most attractive segments from a long video to depict the most interesting parts of the video. Existing methods typically focus on modeling relationship between different video segments in order to learning a model that can assign highlight scores to these segments; however, these approaches do not explicitly consider the contextual dependency within individual segments. To this end, we propose to learn pixel-level distinctions to improve the video highlight detection. This pixel-level distinction indicates whether or not each pixel in one video belongs to an interesting section. The advantages of modeling such fine-level distinctions are two-fold. First, it allows us to exploit the temporal and spatial relations of the content in one video, since the distinction of a pixel in one frame is highly dependent on both the content before this frame and the content around this pixel in this frame. Second, learning the pixel-level distinction also gives a good explanation to the video highlight task regarding what contents in a highlight segment will be attractive to people. We design an encoder-decoder network to estimate the pixel-level distinction, in which we leverage the 3D convolutional neural networks to exploit the temporal context information, and further take advantage of the visual saliency to model the spatial distinction. State-of-the-art performance on three public benchmarks clearly validates the effectiveness of our framework for video highlight detection.

cs.CV↗

arXiv2022

Dynamics on the number of prime divisors for additive arithmetic semigroups

In 2020, Bergelson and Richter gave a dynamical generalization of the classical Prime Number Theorem, which has been generalized by Loyd in a disjoint form with the Erdős-Kac Theorem. These generalizations reveal the rich ergodic properties of the number of prime divisors of integers. In this article, we show a new generalization of Bergelson and Richter's Theorem in a disjoint form with the distribution of the largest prime factors of integers. Then following Bergelson and Richter's techniques, we will show the analogues of all of these results for the arithmetic semigroups arising from finite fields as well.

math.NT↗

arXiv2022

Deep Multi-Branch Aggregation Network for Real-Time Semantic Segmentation in Street Scenes

Real-time semantic segmentation, which aims to achieve high segmentation accuracy at real-time inference speed, has received substantial attention over the past few years. However, many state-of-the-art real-time semantic segmentation methods tend to sacrifice some spatial details or contextual information for fast inference, thus leading to degradation in segmentation quality. In this paper, we propose a novel Deep Multi-branch Aggregation Network (called DMA-Net) based on the encoder-decoder structure to perform real-time semantic segmentation in street scenes. Specifically, we first adopt ResNet-18 as the encoder to efficiently generate various levels of feature maps from different stages of convolutions. Then, we develop a Multi-branch Aggregation Network (MAN) as the decoder to effectively aggregate different levels of feature maps and capture the multi-scale information. In MAN, a lattice enhanced residual block is designed to enhance feature representations of the network by taking advantage of the lattice structure. Meanwhile, a feature transformation block is introduced to explicitly transform the feature map from the neighboring branch before feature aggregation. Moreover, a global context block is used to exploit the global contextual information. These key components are tightly combined and jointly optimized in a unified network. Extensive experimental results on the challenging Cityscapes and CamVid datasets demonstrate that our proposed DMA-Net respectively obtains 77.0% and 73.6% mean Intersection over Union (mIoU) at the inference speed of 46.7 FPS and 119.8 FPS by only using a single NVIDIA GTX 1080Ti GPU. This shows that DMA-Net provides a good tradeoff between segmentation quality and speed for semantic segmentation in street scenes.

cs.CV↗

arXiv2022

Valley Piezoelectric Mechanism for Interpreting and Optimizing Piezoelectricity in Quantum Materials via Anomalous Hall Effect

Quantum materials have exhibited attractive electro-mechanical responses, but their piezoelectric coefficients are far from satisfactory due to the lack of fundamental mechanisms to benefit from the quantum effects. We discovered the valley piezoelectric mechanism that is absent in traditional piezoelectric theory yet promising to overcome this challenge. A theoretical model was developed to elucidate the valley piezoelectricity as the Valley Hall effect driven by pseudoelectric field, which can be significant in quantum systems with broken time reversal symmetry. Consistent tight-binding and density-functional-theory (DFT) calculations validate the model and unveil the crucial dependence of valley piezoelectricity on valley splitting, hybridization energy, bandgap, and Poisson ratio. Doping, passivation, and external stress are proposed as rational strategies to optimize piezoelectricity, with a more than 130% increase of piezoelectricity demonstrated by DFT simulations. The general valley piezoelectric model bridges the gap between electro-mechanical response and quantum effects, which opens an opportunity to achieve outstanding piezoelectricity in quantum materials via optimizing spin-valley and spin-orbit couplings.

physics.comp-ph↗

arXiv2021

Move As You Like: Image Animation in E-Commerce Scenario

Creative image animations are attractive in e-commerce applications, where motion transfer is one of the import ways to generate animations from static images. However, existing methods rarely transfer motion to objects other than human body or human face, and even fewer apply motion transfer in practical scenarios. In this work, we apply motion transfer on the Taobao product images in real e-commerce scenario to generate creative animations, which are more attractive than static images and they will bring more benefits. We animate the Taobao products of dolls, copper running horses and toy dinosaurs based on motion transfer method for demonstration.

cs.GR↗

arXiv2021

Absence of proton tunneling during the hydrogen bond symmetrization in $δ$-AlOOH

$δ$-AlOOH is of significant crystallochemical interest due to a subtle structural transition near 10 GPa from a $P2_1nm$ to a $Pnnm$ structure, the nature and origin of hydrogen disorder, the symmetrization of the O-H$\cdots$O hydrogen bond and their interplay. We perform a series of density functional theory based simulations in combination with high-pressure nuclear magnetic resonance experiments on $δ$-AlOOH up to 40 GPa with the goal to better characterize the hydrogen potential and therefore the nature of hydrogen disorder. Simulations predict a phase transition in agreement with our nuclear magnetic resonance experiments at $10-11$ GPa and hydrogen bond symmetrization at $14.7$ GPa. Calculated hydrogen potentials do not show any double-well character and there is no evidence for proton tunneling in our nuclear magnetic resonance data.

cond-mat.mtrl-sci↗

arXiv2021

Variational Attention: Propagating Domain-Specific Knowledge for Multi-Domain Learning in Crowd Counting

In crowd counting, due to the problem of laborious labelling, it is perceived intractability of collecting a new large-scale dataset which has plentiful images with large diversity in density, scene, etc. Thus, for learning a general model, training with data from multiple different datasets might be a remedy and be of great value. In this paper, we resort to the multi-domain joint learning and propose a simple but effective Domain-specific Knowledge Propagating Network (DKPNet)1 for unbiasedly learning the knowledge from multiple diverse data domains at the same time. It is mainly achieved by proposing the novel Variational Attention(VA) technique for explicitly modeling the attention distributions for different domains. And as an extension to VA, Intrinsic Variational Attention(InVA) is proposed to handle the problems of over-lapped domains and sub-domains. Extensive experiments have been conducted to validate the superiority of our DKPNet over several popular datasets, including ShanghaiTech A/B, UCF-QNRF and NWPU.

cs.CV↗