Searcharxiv⌕ Search

arXiv subjects

Peng Gao

Publications and source records attributed to Peng Gao.

At least 307 records · Page 17Linked to original sources
arXiv2022

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations using large-scale image-text pairs. It shows impressive performance on downstream tasks by zero-shot knowledge transfer. To further enhance CLIP's adaption capability, existing methods proposed to fine-tune additional learnable modules, which significantly improves the few-shot performance but introduces extra training time and computational resources. In this paper, we propose a training-free adaption method for CLIP to conduct few-shot classification, termed as Tip-Adapter, which not only inherits the training-free advantage of zero-shot CLIP but also performs comparably to those training-required approaches. Tip-Adapter constructs the adapter via a key-value cache model from the few-shot training set, and updates the prior knowledge encoded in CLIP by feature retrieval. On top of that, the performance of Tip-Adapter can be further boosted to be state-of-the-art on ImageNet by fine-tuning the cache model for 10$\times$ fewer epochs than existing methods, which is both effective and efficient. We conduct extensive experiments of few-shot classification on 11 datasets to demonstrate the superiority of our proposed methods. Code is released at https://github.com/gaopengcuhk/Tip-Adapter.

cs.CV↗
arXiv2022

Prototypical Contrast Adaptation for Domain Adaptive Semantic Segmentation

Unsupervised Domain Adaptation (UDA) aims to adapt the model trained on the labeled source domain to an unlabeled target domain. In this paper, we present Prototypical Contrast Adaptation (ProCA), a simple and efficient contrastive learning method for unsupervised domain adaptive semantic segmentation. Previous domain adaptation methods merely consider the alignment of the intra-class representational distributions across various domains, while the inter-class structural relationship is insufficiently explored, resulting in the aligned representations on the target domain might not be as easily discriminated as done on the source domain anymore. Instead, ProCA incorporates inter-class information into class-wise prototypes, and adopts the class-centered distribution alignment for adaptation. By considering the same class prototypes as positives and other class prototypes as negatives to achieve class-centered distribution alignment, ProCA achieves state-of-the-art performance on classical domain adaptation tasks, {\em i.e., GTA5 $\to$ Cityscapes \text{and} SYNTHIA $\to$ Cityscapes}. Code is available at \href{https://github.com/jiangzhengkai/ProCA}{ProCA}

cs.CV↗
arXiv2022

SFE-AI at SemEval-2022 Task 11: Low-Resource Named Entity Recognition using Large Pre-trained Language Models

Large scale pre-training models have been widely used in named entity recognition (NER) tasks. However, model ensemble through parameter averaging or voting can not give full play to the differentiation advantages of different models, especially in the open domain. This paper describes our NER system in the SemEval 2022 task11: MultiCoNER. We proposed an effective system to adaptively ensemble pre-trained language models by a Transformer layer. By assigning different weights to each model for different inputs, we adopted the Transformer layer to integrate the advantages of diverse models effectively. Experimental results show that our method achieves superior performances in Farsi and Dutch.

cs.CL↗
arXiv2022

ConvMAE: Masked Convolution Meets Masked Autoencoders

Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained ConvMAE models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at https://github.com/Alpha-VL/ConvMAE.

cs.CV↗
arXiv2022

A gate-programmable van der Waals metal-ferroelectric-semiconductor memory

Ferroelecticity, one of the keys to realize nonvolatile memories owing to the remanent electric polarization, has been an emerging phenomenon in the two-dimensional (2D) limit. Yet the demonstrations of van der Waals (vdW) memories using 2D ferroelectric materials as an ingredient are very limited. Especially, gate-tunable ferroelectric vdW memristive device, which holds promises in future neuromorphic applications, remains challenging. Here, we show a prototype gate-programmable memory by vertically assembling graphite, CuInP2S6, and MoS2 layers into a metal-ferroelectric-semiconductor architecture. The resulted devices exhibit two-terminal switchable electro-resistance with on-off ratios exceeding 105 and long-term retention, akin to a conventional memristor but strongly coupled to the ferroelectric characteristics of the CuInP2S6 layer. By controlling the top gate, Fermi level of MoS2 can be set inside (outside) of its band gap to quench (enable) the memristive behaviour, yielding a three-terminal gate programmable nonvolatile vdW memory. Our findings pave the way for the engineering of ferroelectric-mediated memories in future implementations of nanoelectronics.

cond-mat.mtrl-sci↗
arXiv2022

Bounds for moments of cubic and quartic Dirichlet $L$-functions

We study the $2k$-th moment of central values of the family of primitive cubic and quartic Dirichlet $L$-functions. We establish sharp lower bounds for all real $k \geq 1/2$ unconditionally for the cubic case and under the Lindelöf hypothesis for the quartic case. We also establish sharp lower bounds for all real $0 \leq k<1/2$ and sharp upper bounds for all real $k \geq 0$ for both the cubic and quartic cases under the generalized Riemann hypothesis (GRH). As an application of our results, we establish quantitative non-vanishing results for the corresponding $L$-values.

math.NT↗
arXiv2022

Electron microscopy probing electron-photon interactions in SiC nanowires with ultra-wide energy and momentum match

Nanoscale materials usually can trap light and strongly interact with it leading to many photonic device applications. The light-matter interactions are commonly probed by optical spectroscopy, which, however, have some limitations such as diffraction-limited spatial resolution, tiny momentum transfer and non-continuous excitation/detection. In this work, using scanning transmission electron microscopy-electron energy loss spectroscopy (STEM-EELS) with ultra-wide energy and momentum match and sub-nanometer spatial resolution, we study the optical microcavity resonant spectroscopy in a single SiC nanowire. The longitudinal Fabry-Perot (FP) resonating modes and the transverse whispering-gallery modes (WGMs) are simultaneously excited and detected, which span from near-infrared (~ 1.2 μm) to ultraviolet (~ 0.2 μm) spectral regime and the momentum transfer can be ranging up to 108 cm{^{-1}}. The size effects on the resonant spectra of nanowires are also revealed. Moreover, the nanoscale decay length of resonant EELS is revealed, which is contributed by the strongly localized electron-photon interactions in the SiC nanowire. This work provides a new alternative technique to investigate the optical resonating spectroscopy of a single nanowire structure and to explore the light-matter interactions in dielectric nanostructures, which is also promising for modulating free electrons via photonic structures.

cond-mat.mes-hall↗
arXiv2022

Optimal convergence order for multi-scale stochastic Burgers equation

In this paper, we study the strong and weak convergence rates for multi-scale one-dimensional stochastic Burgers equation. Based on the techniques of Galerkin approximation, Kolmogorov equation and Poisson equation, we obtain the slow component strongly and weakly converges to the solution of the corresponding averaged equation with optimal orders 1/2 and 1 respectively. The highly nonlinear term in system brings us huge difficulties, we develop new technique to overcome these difficulties. To the best of our knowledge, this work seems to be the first result in which the optimal convergence orders in strong and weak sense for multi-scale stochastic partial differential equations with highly nonlinear term.

math.PR↗
arXiv2022

POS-BERT: Point Cloud One-Stage BERT Pre-Training

Recently, the pre-training paradigm combining Transformer and masked language modeling has achieved tremendous success in NLP, images, and point clouds, such as BERT. However, directly extending BERT from NLP to point clouds requires training a fixed discrete Variational AutoEncoder (dVAE) before pre-training, which results in a complex two-stage method called Point-BERT. Inspired by BERT and MoCo, we propose POS-BERT, a one-stage BERT pre-training method for point clouds. Specifically, we use the mask patch modeling (MPM) task to perform point cloud pre-training, which aims to recover masked patches information under the supervision of the corresponding tokenizer output. Unlike Point-BERT, its tokenizer is extra-trained and frozen. We propose to use the dynamically updated momentum encoder as the tokenizer, which is updated and outputs the dynamic supervision signal along with the training process. Further, in order to learn high-level semantic representation, we combine contrastive learning to maximize the class token consistency between different transformation point clouds. Extensive experiments have demonstrated that POS-BERT can extract high-quality pre-training features and promote downstream tasks to improve performance. Using the pre-training model without any fine-tuning to extract features and train linear SVM on ModelNet40, POS-BERT achieves the state-of-the-art classification accuracy, which exceeds Point-BERT by 3.5\%. In addition, our approach has significantly improved many downstream tasks, such as fine-tuned classification, few-shot classification, part segmentation. The code and trained-models will be available at: \url{https://github.com/fukexue/POS-BERT}.

cs.CV↗
arXiv2022

Learning Decoupling Features Through Orthogonality Regularization

Keyword spotting (KWS) and speaker verification (SV) are two important tasks in speech applications. Research shows that the state-of-art KWS and SV models are trained independently using different datasets since they expect to learn distinctive acoustic features. However, humans can distinguish language content and the speaker identity simultaneously. Motivated by this, we believe it is important to explore a method that can effectively extract common features while decoupling task-specific features. Bearing this in mind, a two-branch deep network (KWS branch and SV branch) with the same network structure is developed and a novel decoupling feature learning method is proposed to push up the performance of KWS and SV simultaneously where speaker-invariant keyword representations and keyword-invariant speaker representations are expected respectively. Experiments are conducted on Google Speech Commands Dataset (GSCD). The results demonstrate that the orthogonality regularization helps the network to achieve SOTA EER of 1.31% and 1.87% on KWS and SV, respectively.

cs.SD↗
arXiv2022

Direct observation of local antiferroelectricity induced phonon softening at a SrTiO3 defect

Defects in oxides usually exhibit exotic properties that may be associated with the local lattice dynamics. Here, at atomic spatial resolution, we directly measure phonon modes of an antiphase boundary (APB) in SrTiO3 freestanding membrane and correlate them with the picometer-level structural distortion. We find that the SrTiO3 APB introduces new defect phonon modes that are absent in bulk SrTiO3. These modes are highly sensitive to the subtle structure distortion, i.e., the SrTiO3 APB generates the local electric dipoles forming an antiferroelectric configuration, which significantly softens the transverse optical (TO) and longitudinal optical (LO) modes at Γ point. Correlating the local phonons with the subtle structural distortion, our findings provide valuable insights into understanding the defect properties in complex oxides and essential information for their applications such as thermoelectric devices.

cond-mat.mtrl-sci↗
arXiv2022

Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning

In this paper, we propose a simple and general framework for self-supervised point cloud representation learning. Human beings understand the 3D world by extracting two levels of information and establishing the relationship between them. One is the global shape of an object, and the other is the local structures of it. However, few existing studies in point cloud representation learning explored how to learn both global shapes and local-to-global relationships without a specified network architecture. Inspired by how human beings understand the world, we utilize knowledge distillation to learn both global shape information and the relationship between global shape and local structures. At the same time, we combine contrastive learning with knowledge distillation to make the teacher network be better updated. Our method achieves the state-of-the-art performance on linear classification and multiple other downstream tasks. Especially, we develop a variant of ViT for 3D point cloud feature extraction, which also achieves comparable results with existing backbones when combined with our framework, and visualization of the attention maps show that our model does understand the point cloud by combining the global shape information and multiple local structural information, which is consistent with the inspiration of our representation learning method. Our code will be released soon.

cs.CV↗
arXiv2022

UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning

It is a challenging task to learn rich and multi-scale spatiotemporal semantics from high-dimensional videos, due to large local redundancy and complex global dependency between video frames. The recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. Although 3D convolution can efficiently aggregate local context to suppress local redundancy from a small 3D neighborhood, it lacks the capability to capture global dependency because of the limited receptive field. Alternatively, vision transformers can effectively capture long-range dependency by self-attention mechanism, while having the limitation on reducing local redundancy with blind similarity comparison among all the tokens in each layer. Based on these observations, we propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution and spatiotemporal self-attention in a concise transformer format, and achieves a preferable balance between computation and accuracy. Different from traditional transformers, our relation aggregator can tackle both spatiotemporal redundancy and dependency, by learning local and global token affinity respectively in shallow and deep layers. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. With only ImageNet-1K pretraining, our UniFormer achieves 82.9%/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods. For Something-Something V1 and V2, our UniFormer achieves new state-of-the-art performances of 60.9% and 71.2% top-1 accuracy respectively. Code is available at https://github.com/Sense-X/UniFormer.

cs.CV↗
arXiv2022

TerViT: An Efficient Ternary Vision Transformer

Vision transformers (ViTs) have demonstrated great potential in various visual tasks, but suffer from expensive computational and memory cost problems when deployed on resource-constrained devices. In this paper, we introduce a ternary vision transformer (TerViT) to ternarize the weights in ViTs, which are challenged by the large loss surface gap between real-valued and ternary parameters. To address the issue, we introduce a progressive training scheme by first training 8-bit transformers and then TerViT, and achieve a better optimization than conventional methods. Furthermore, we introduce channel-wise ternarization, by partitioning each matrix to different channels, each of which is with an unique distribution and ternarization interval. We apply our methods to popular DeiT and Swin backbones, and extensive results show that we can achieve competitive performance. For example, TerViT can quantize Swin-S to 13.1MB model size while achieving above 79% Top-1 accuracy on ImageNet dataset.

cs.CV↗
arXiv2022

Bounds for moments of quadratic Dirichlet $L$-functions of prime-related moduli

In this paper, we study the $k$-th moment of central values of the family of quadratic Dirichlet $L$-functions of moduli $8p$, with $p$ ranging over odd primes. Assuming the truth of the generlized Riemann hypothesis, we establish sharp upper and lower bounds for the $k$-th power moment of these $L$-values for all real $k \geq 0$.

math.NT↗
arXiv2022

Average values of quadratic Hecke character sums

We study smoothed character sums involving $\sum_{m,n} ( \frac{m}{n} )_2$, where $( \frac{m}{n} )_2$ denotes the quadratic symbol in the Gaussian field. We extend previously known results to obtain asymptotic formulas for the sums considered to larger ranges of $m$ and $n$.

math.NT↗