arXiv subjects

Shiwei Zhang

Publications and source records attributed to Shiwei Zhang.

At least 127 records · Page 7Linked to original sources

arXiv2022

Precision Many-Body Study of the Berezinskii-Kosterlitz-Thouless Transition and Temperature-Dependent Properties in the Two-Dimensional Fermi Gas

We perform large-scale, numerically exact calculations on the two-dimensional interacting Fermi gas with a contact attraction. Reaching much larger lattice sizes and lower temperatures than previously possible, we determine systematically the finite-temperature phase diagram of the Berezinskii-Kosterlitz-Thouless (BKT) transitions for interaction strengths ranging from BCS to crossover to BEC regimes. The evolutions of the pairing wavefunctions and the fermion and Cooper pair momentum distributions with temperature are accurately characterized. In the crossover regime, we find that the contact has a non-monotonic temperature dependence, first increasing as temperature is lowered, and then showing a slight decline below the BKT transition temperature to approach the ground-state value from above.

cond-mat.quant-gas↗

arXiv2022

End-to-end Temporal Action Detection with Transformer

Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.

cs.CV↗

arXiv2022

MAR: Masked Autoencoders for Efficient Action Recognition

Standard approaches for video recognition usually operate on the full input videos, which is inefficient due to the widely present spatio-temporal redundancy in videos. Recent progress in masked video modelling, i.e., VideoMAE, has shown the ability of vanilla Vision Transformers (ViT) to complement spatio-temporal contexts given only limited visual contents. Inspired by this, we propose propose Masked Action Recognition (MAR), which reduces the redundant computation by discarding a proportion of patches and operating only on a part of the videos. MAR contains the following two indispensable components: cell running masking and bridging classifier. Specifically, to enable the ViT to perceive the details beyond the visible patches easily, cell running masking is presented to preserve the spatio-temporal correlations in videos, which ensures the patches at the same spatial location can be observed in turn for easy reconstructions. Additionally, we notice that, although the partially observed features can reconstruct semantically explicit invisible patches, they fail to achieve accurate classification. To address this, a bridging classifier is proposed to bridge the semantic gap between the ViT encoded features for reconstruction and the features specialized for classification. Our proposed MAR reduces the computational cost of ViT by 53% and extensive experiments show that MAR consistently outperforms existing ViT models with a notable margin. Especially, we found a ViT-Large trained by MAR outperforms the ViT-Huge trained by a standard training scheme by convincing margins on both Kinetics-400 and Something-Something v2 datasets, while our computation overhead of ViT-Large is only 14.5% of ViT-Huge.

cs.CV↗

arXiv2022

Solving 2D and 3D lattice models of correlated fermions -- combining matrix product states with mean field theory

Correlated electron states are at the root of many important phenomena including unconventional superconductivity (USC), where electron-pairing arises from repulsive interactions. Computing the properties of correlated electrons, such as the critical temperature $T_c$ for the onset of USC, efficiently and reliably from the microscopic physics with quantitative methods remains a major challenge for almost all models and materials. In this theoretical work we combine matrix product states (MPS) with static mean field (MF) to provide a solution to this challenge for quasi-one-dimensional (Q1D) systems: Two- and three-dimensional (2D/3D) materials comprised of weakly coupled correlated 1D fermions. This MPS+MF framework for the ground state and thermal equilibrium properties of Q1D fermions is developed and validated for attractive Hubbard systems first, and further enhanced via analytical field theory. We then deploy it to compute $T_c$ for superconductivity in 3D arrays of weakly coupled, doped and repulsive Hubbard ladders. The MPS+MF framework thus enables the reliable, quantitative and unbiased study of USC and high-$T_c$ superconductivity - and potentially many more correlated phases - in fermionic Q1D systems from microscopic parameters, in ways inaccessible to previous methods. It opens the possibility of designing deliberately optimized Q1D superconductors, from experiments in ultracold gases to synthesizing new materials.

cond-mat.str-el↗

arXiv2022

Open-world Semantic Segmentation for LIDAR Point Clouds

Current methods for LIDAR semantic segmentation are not robust enough for real-world applications, e.g., autonomous driving, since it is closed-set and static. The closed-set assumption makes the network only able to output labels of trained classes, even for objects never seen before, while a static network cannot update its knowledge base according to what it has seen. Therefore, in this work, we propose the open-world semantic segmentation task for LIDAR point clouds, which aims to 1) identify both old and novel classes using open-set semantic segmentation, and 2) gradually incorporate novel objects into the existing knowledge base using incremental learning without forgetting old classes. For this purpose, we propose a REdundAncy cLassifier (REAL) framework to provide a general architecture for both the open-set semantic segmentation and incremental learning problems. The experimental results show that REAL can simultaneously achieves state-of-the-art performance in the open-set semantic segmentation task on the SemanticKITTI and nuScenes datasets, and alleviate the catastrophic forgetting problem with a large margin during incremental learning.

cs.CV↗

arXiv2022

Context-aware Proposal Network for Temporal Action Detection

This technical report presents our first place winning solution for temporal action detection task in CVPR-2022 AcitivityNet Challenge. The task aims to localize temporal boundaries of action instances with specific classes in long untrimmed videos. Recent mainstream attempts are based on dense boundary matchings and enumerate all possible combinations to produce proposals. We argue that the generated proposals contain rich contextual information, which may benefits detection confidence prediction. To this end, our method mainly consists of the following three steps: 1) action classification and feature extraction by Slowfast, CSN, TimeSformer, TSP, I3D-flow, VGGish-audio, TPN and ViViT; 2) proposal generation. Our proposed Context-aware Proposal Network (CPN) builds on top of BMN, GTAD and PRN to aggregate contextual information by randomly masking some proposal features. 3) action detection. The final detection prediction is calculated by assigning the proposals with corresponding video-level classifcation results. Finally, we ensemble the results under different feature combination settings and achieve 45.8% performance on the test set, which improves the champion result in CVPR-2021 ActivityNet Challenge by 1.1% in terms of average mAP.

cs.CV↗

arXiv2022

Discovery-and-Selection: Towards Optimal Multiple Instance Learning for Weakly Supervised Object Detection

Weakly supervised object detection (WSOD) is a challenging task that requires simultaneously learn object classifiers and estimate object locations under the supervision of image category labels. A major line of WSOD methods roots in multiple instance learning which regards images as bags of instances and selects positive instances from each bag to learn the detector. However, a grand challenge emerges when the detector inclines to converge to discriminative parts of objects rather than the whole objects. In this paper, under the hypothesis that optimal solutions are included in local minima, we propose a discovery-and-selection approach fused with multiple instance learning (DS-MIL), which finds rich local minima and select optimal solution from multiple local minima. To implement DS-MIL, an attention module is proposed so that more context information can be captured by feature maps and more valuable proposals can be collected during training. With proposal candidates, a selection module is proposed to select informative instances for object detector. Experimental results on commonly used benchmarks show that our proposed DS-MIL approach can consistently improve the baselines, reporting state-of-the-art performance.

cs.CV↗

arXiv2022

Hybrid Relation Guided Set Matching for Few-shot Action Recognition

Current few-shot action recognition methods reach impressive performance by learning discriminative features for each video via episodic training and designing various temporal alignment strategies. Nevertheless, they are limited in that (a) learning individual features without considering the entire task may lose the most relevant information in the current episode, and (b) these alignment strategies may fail in misaligned instances. To overcome the two limitations, we propose a novel Hybrid Relation guided Set Matching (HyRSM) approach that incorporates two key components: hybrid relation module and set matching metric. The purpose of the hybrid relation module is to learn task-specific embeddings by fully exploiting associated relations within and cross videos in an episode. Built upon the task-specific features, we reformulate distance measure between query and support videos as a set matching problem and further design a bidirectional Mean Hausdorff Metric to improve the resilience to misaligned instances. By this means, the proposed HyRSM can be highly informative and flexible to predict query categories under the few-shot settings. We evaluate HyRSM on six challenging benchmarks, and the experimental results show its superiority over the state-of-the-art methods by a convincing margin. Project page: https://hyrsm-cvpr2022.github.io/.

cs.CV↗

arXiv2022

Learning from Untrimmed Videos: Self-Supervised Video Representation Learning with Hierarchical Consistency

Natural videos provide rich visual contents for self-supervised learning. Yet most existing approaches for learning spatio-temporal representations rely on manually trimmed videos, leading to limited diversity in visual patterns and limited performance gain. In this work, we aim to learn representations by leveraging more abundant information in untrimmed videos. To this end, we propose to learn a hierarchy of consistencies in videos, i.e., visual consistency and topical consistency, corresponding respectively to clip pairs that tend to be visually similar when separated by a short time span and share similar topics when separated by a long time span. Specifically, a hierarchical consistency learning framework HiCo is presented, where the visually consistent pairs are encouraged to have the same representation through contrastive learning, while the topically consistent pairs are coupled through a topical classifier that distinguishes whether they are topic related. Further, we impose a gradual sampling algorithm for proposed hierarchical consistency learning, and demonstrate its theoretical superiority. Empirically, we show that not only HiCo can generate stronger representations on untrimmed videos, it also improves the representation quality when applied to trimmed videos. This is in contrast to standard contrastive learning that fails to learn appropriate representations from untrimmed videos.

cs.CV↗

arXiv2022

Stripes and spin-density waves in the doped two-dimensional Hubbard model: ground state phase diagram

We determine the spin and charge orders in the ground state of the doped two-dimensional (2D) Hubbard model in its simplest form, namely with only nearest-neighbor hopping and on-site repulsion. At half-filling, the ground state is known to be an anti-ferromagnetic Mott insulator. Doping Mott insulators is believed to be relevant to the superconductivity observed in cuprates. A variety of candidates have been proposed for the ground state of the doped 2D Hubbard model. A recent work employing a combination of several state-of-the-art numerical many-body methods, established the stripe order as the ground state near $1/8$ doping at strong interactions. In this work, we apply one of these methods, the cutting-edge constrained-path auxiliary field quantum Monte Carlo method with self-consistently optimized gauge constraints, to systematically study the model as a function of doping and interaction strength. With careful finite size scaling based on large-scale computations, we map out the ground state phase diagram in terms of its spin and charge order. We find that modulated antiferromagnetic order persists from near half-filling to about $1/5$ doping. At lower interaction strengths or larger doping, these ordered states are best described as spin-density waves, with essentially delocalized holes and modest oscillations in charge correlations. When the charge correlations are stronger (large interaction or small doping), they are best described as stripe states, with the holes more localized near the node in the antiferromagnetic spin order. In both cases, we find that the wavelength in the charge correlations is consistent with so-called filled stripes in the pure Hubbard model.

cond-mat.str-el↗

arXiv2022

TCTrack: Temporal Contexts for Aerial Tracking

Temporal contexts among consecutive frames are far from being fully utilized in existing visual trackers. In this work, we present TCTrack, a comprehensive framework to fully exploit temporal contexts for aerial tracking. The temporal contexts are incorporated at \textbf{two levels}: the extraction of \textbf{features} and the refinement of \textbf{similarity maps}. Specifically, for feature extraction, an online temporally adaptive convolution is proposed to enhance the spatial features using temporal information, which is achieved by dynamically calibrating the convolution weights according to the previous frames. For similarity map refinement, we propose an adaptive temporal transformer, which first effectively encodes temporal knowledge in a memory-efficient way, before the temporal knowledge is decoded for accurate adjustment of the similarity map. TCTrack is effective and efficient: evaluation on four aerial tracking benchmarks shows its impressive performance; real-world UAV tests show its high speed of over 27 FPS on NVIDIA Jetson AGX Xavier.

cs.CV↗

arXiv2022

TAda! Temporally-Adaptive Convolutions for Video Understanding

Spatial convolutions are widely used in numerous deep video models. It fundamentally assumes spatio-temporal invariance, i.e., using shared weights for every location in different frames. This work presents Temporally-Adaptive Convolutions (TAdaConv) for video understanding, which shows that adaptive weight calibration along the temporal dimension is an efficient way to facilitate modelling complex temporal dynamics in videos. Specifically, TAdaConv empowers the spatial convolutions with temporal modelling abilities by calibrating the convolution weights for each frame according to its local and global temporal context. Compared to previous temporal modelling operations, TAdaConv is more efficient as it operates over the convolution kernels instead of the features, whose dimension is an order of magnitude smaller than the spatial resolutions. Further, the kernel calibration brings an increased model capacity. We construct TAda2D and TAdaConvNeXt networks by replacing the 2D convolutions in ResNet and ConvNeXt with TAdaConv, which leads to at least on par or better performance compared to state-of-the-art approaches on multiple video action recognition and localization benchmarks. We also demonstrate that as a readily plug-in operation with negligible computation overhead, TAdaConv can effectively improve many existing video models with a convincing margin.

cs.CV↗

arXiv2021

Does QA-based intermediate training help fine-tuning language models for text classification?

Fine-tuning pre-trained language models for downstream tasks has become a norm for NLP. Recently it is found that intermediate training based on high-level inference tasks such as Question Answering (QA) can improve the performance of some language models for target tasks. However it is not clear if intermediate training generally benefits various language models. In this paper, using the SQuAD-2.0 QA task for intermediate training for target text classification tasks, we experimented on eight tasks for single-sequence classification and eight tasks for sequence-pair classification using two base and two compact language models. Our experiments show that QA-based intermediate training generates varying transfer performance across different language models, except for similar QA tasks.

cs.CL↗

arXiv2021

ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning

The central idea of contrastive learning is to discriminate between different instances and force different views from the same instance to share the same representation. To avoid trivial solutions, augmentation plays an important role in generating different views, among which random cropping is shown to be effective for the model to learn a generalized and robust representation. Commonly used random crop operation keeps the distribution of the difference between two views unchanged along the training process. In this work, we show that adaptively controlling the disparity between two augmented views along the training process enhances the quality of the learned representation. Specifically, we present a parametric cubic cropping operation, ParamCrop, for video contrastive learning, which automatically crops a 3D cubic by differentiable 3D affine transformations. ParamCrop is trained simultaneously with the video backbone using an adversarial objective and learns an optimal cropping strategy from the data. The visualizations show that ParamCrop adaptively controls the center distance and the IoU between two augmented views, and the learned change in the disparity along the training process is beneficial to learning a strong representation. Extensive ablation studies demonstrate the effectiveness of the proposed ParamCrop on multiple contrastive learning frameworks and video backbones. Codes and models will be available.

cs.CV↗

arXiv2021

Ab initio Calculations in Atoms, Molecules, and Solids, Treating Spin-Orbit Coupling and Electron Interaction on Equal Footing

We incorporate explicit, non-perturbative treatment of spin-orbit coupling into ab initio auxiliary-field quantum Monte Carlo (AFQMC) calculations. The approach allows a general computational framework for molecular and bulk systems in which materials specificity, electron correlation, and spin-orbit coupling effects can be captured accurately and on equal footing, with favorable computational scaling versus system size. We adopt relativistic effective-core potentials which have been obtained by fitting to fully relativistic data and which have demonstrated a high degree of reliability and transferability in molecular systems. This results in a 2-component spin-coupled Hamiltonian, which is then treated by generalizing the ab initio AFQMC approach. We demonstrate the method by computing the electron affinity in Pb, the bond dissociation energy in Br$_2$ and I$_2$, and solid Bi.

cond-mat.str-el↗

arXiv2021

Exotic Superfluid Phases in Spin Polarized Systems on Optical Lattices

Leveraging cutting-edge numerical methodologies, we study the ground state of the two-dimensional spin-polarized Fermi gas in an optical lattice. We focus on systems at high density and small spin polarization, corresponding to the parameter regime believed to be most favorable to the formation of the elusive Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) superfluid phase. Our systematic study of large lattice sizes, hosting nearly $500$ atoms, provides strong evidence of the stability of the FFLO state in this regime, as well as a high-accuracy characterization of its properties. Our results for the density correlation function reveal the existence of density order in the system, suggesting the possibility of an intricate coexistence of long-range orders in the ground state. The ground-state properties are seen to differ significantly from the standard mean-field description, providing a compelling avenue for future theoretical and experimental explorations of the interplay between interaction and superfluidity in an exotic phase of matter.

cond-mat.quant-gas↗

arXiv2021

Support-Set Based Cross-Supervision for Video Grounding

Current approaches for video grounding propose kinds of complex architectures to capture the video-text relations, and have achieved impressive improvements. However, it is hard to learn the complicated multi-modal relations by only architecture designing in fact. In this paper, we introduce a novel Support-set Based Cross-Supervision (Sscs) module which can improve existing methods during training phase without extra inference cost. The proposed Sscs module contains two main components, i.e., discriminative contrastive objective and generative caption objective. The contrastive objective aims to learn effective representations by contrastive learning, while the caption objective can train a powerful video encoder supervised by texts. Due to the co-existence of some visual entities in both ground-truth and background intervals, i.e., mutual exclusion, naively contrastive learning is unsuitable to video grounding. We address the problem by boosting the cross-supervision with the support-set concept, which collects visual information from the whole video and eliminates the mutual exclusion of entities. Combined with the original objectives, Sscs can enhance the abilities of multi-modal relation modeling for existing approaches. We extensively evaluate Sscs on three challenging datasets, and show that our method can improve current state-of-the-art methods by large margins, especially 6.35% in terms of R1@0.5 on Charades-STA.

cs.CV↗

arXiv2021

Exploring Stronger Feature for Temporal Action Localization

Temporal action localization aims to localize starting and ending time with action category. Limited by GPU memory, mainstream methods pre-extract features for each video. Therefore, feature quality determines the upper bound of detection performance. In this technical report, we explored classic convolution-based backbones and the recent surge of transformer-based backbones. We found that the transformer-based methods can achieve better classification performance than convolution-based, but they cannot generate accuracy action proposals. In addition, extracting features with larger frame resolution to reduce the loss of spatial information can also effectively improve the performance of temporal action localization. Finally, we achieve 42.42% in terms of mAP on validation set with a single SlowFast feature by a simple combination: BMN+TCANet, which is 1.87% higher than the result of 2020's multi-model ensemble. Finally, we achieve Rank 1st on the CVPR2021 HACS supervised Temporal Action Localization Challenge.

cs.CV↗