arXiv subjects

Yujun Shi

Publications and source records attributed to Yujun Shi.

26 records · Page 2Linked to original sources

arXiv2021

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Transformers, which are popular for language modeling, have been explored for solving vision tasks recently, e.g., the Vision Transformer (ViT) for image classification. The ViT model splits each image into a sequence of tokens with fixed length and then applies multiple Transformer layers to model their global relation for classification. However, ViT achieves inferior performance to CNNs when trained from scratch on a midsize dataset like ImageNet. We find it is because: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines among neighboring pixels, leading to low training sample efficiency; 2) the redundant attention backbone design of ViT leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we propose a new Tokens-To-Token Vision Transformer (T2T-ViT), which incorporates 1) a layer-wise Tokens-to-Token (T2T) transformation to progressively structurize the image to tokens by recursively aggregating neighboring Tokens into one Token (Tokens-to-Token), such that local structure represented by surrounding tokens can be modeled and tokens length can be reduced; 2) an efficient backbone with a deep-narrow structure for vision transformer motivated by CNN architecture design after empirical study. Notably, T2T-ViT reduces the parameter count and MACs of vanilla ViT by half, while achieving more than 3.0\% improvement when trained from scratch on ImageNet. It also outperforms ResNets and achieves comparable performance with MobileNets by directly training on ImageNet. For example, T2T-ViT with comparable size to ResNet50 (21.5M parameters) can achieve 83.3\% top1 accuracy in image resolution 384$\times$384 on ImageNet. (Code: https://github.com/yitu-opensource/T2T-ViT)

cs.CV↗

arXiv2021

All Tokens Matter: Token Labeling for Training Better Vision Transformers

In this paper, we present token labeling -- a new training objective for training high-performance vision transformers (ViTs). Different from the standard training objective of ViTs that computes the classification loss on an additional trainable class token, our proposed one takes advantage of all the image patch tokens to compute the training loss in a dense manner. Specifically, token labeling reformulates the image classification problem into multiple token-level recognition problems and assigns each patch token with an individual location-specific supervision generated by a machine annotator. Experiments show that token labeling can clearly and consistently improve the performance of various ViT models across a wide spectrum. For a vision transformer with 26M learnable parameters serving as an example, with token labeling, the model can achieve 84.4% Top-1 accuracy on ImageNet. The result can be further increased to 86.4% by slightly scaling the model size up to 150M, delivering the minimal-sized model among previous models (250M+) reaching 86%. We also show that token labeling can clearly improve the generalization capability of the pre-trained models on downstream tasks with dense prediction, such as semantic segmentation. Our code and all the training details will be made publicly available at https://github.com/zihangJiang/TokenLabeling.

cs.CV↗

arXiv2021

Refiner: Refining Self-attention for Vision Transformers

Vision Transformers (ViTs) have shown competitive accuracy in image classification tasks compared with CNNs. Yet, they generally require much more data for model pre-training. Most of recent works thus are dedicated to designing more complex architectures or training methods to address the data-efficiency issue of ViTs. However, few of them explore improving the self-attention mechanism, a key factor distinguishing ViTs from CNNs. Different from existing works, we introduce a conceptually simple scheme, called refiner, to directly refine the self-attention maps of ViTs. Specifically, refiner explores attention expansion that projects the multi-head attention maps to a higher-dimensional space to promote their diversity. Further, refiner applies convolutions to augment local patterns of the attention maps, which we show is equivalent to a distributed local attention features are aggregated locally with learnable kernels and then globally aggregated with self-attention. Extensive experiments demonstrate that refiner works surprisingly well. Significantly, it enables ViTs to achieve 86% top-1 classification accuracy on ImageNet with only 81M parameters.

cs.CV↗

arXiv2021

Universal scaling of the critical temperature and the strange-metal scattering rate in unconventional superconductors

Dramatic evolution of properties with minute change in the doping level is a hallmark of the complex chemistry which governs cuprate superconductivity as manifested in the celebrated superconducting domes as well as quantum criticality taking place at precise compositions. The strange metal state, where the resistivity varies linearly with temperature, has emerged as a central feature in the normal state of cuprate superconductors. The ubiquity of this behavior signals an intimate link between the scattering mechanism and superconductivity. However, a clear quantitative picture of the correlation has been lacking. Here, we report observation of quantitative scaling laws between the superconducting transition temperature $T_{\rm c}$ and the scattering rate associated with the strange metal state in electron-doped cuprate $\rm La_{2-x}Ce_xCuO_4$ (LCCO) as a precise function of the doping level. High-resolution characterization of epitaxial composition-spread films, which encompass the entire overdoped range of LCCO has allowed us to systematically map its structural and transport properties with unprecedented accuracy and increment of $Δx = 0.0015$. We have uncovered the relations $T_{\rm c}\sim(x_{\rm c}-x)^{0.5}\sim(A_1^\square)^{0.5}$, where $x_c$ is the critical doping where superconductivity disappears on the overdoped side and $A_1^\square$ is the scattering rate of perfect $T$-linear resistivity per CuO$_2$ plane. We argue that the striking similarity of the $T_{\rm c}$ vs $A_1^\square$ relation among cuprates, iron-based and organic superconductors is an indication of a common mechanism of the strange metal behavior and unconventional superconductivity in these systems.

cond-mat.supr-con↗

arXiv2019

Understanding Adversarial Behavior of DNNs by Disentangling Non-Robust and Robust Components in Performance Metric

The vulnerability to slight input perturbations is a worrying yet intriguing property of deep neural networks (DNNs). Despite many previous works studying the reason behind such adversarial behavior, the relationship between the generalization performance and adversarial behavior of DNNs is still little understood. In this work, we reveal such relation by introducing a metric characterizing the generalization performance of a DNN. The metric can be disentangled into an information-theoretic non-robust component, responsible for adversarial behavior, and a robust component. Then, we show by experiments that current DNNs rely heavily on optimizing the non-robust component in achieving decent performance. We also demonstrate that current state-of-the-art adversarial training algorithms indeed try to robustify the DNNs by preventing them from using the non-robust component to distinguish samples from different categories. Also, based on our findings, we take a step forward and point out the possible direction for achieving decent standard performance and adversarial robustness simultaneously. We believe that our theory could further inspire the community to make more interesting discoveries about the relationship between standard generalization and adversarial generalization of deep learning models.

stat.ML↗

arXiv2019

Emergent superconductivity in single crystalline $\mathrm{MgTi}_2\mathrm{O}_4$ films via structural engineering

Spinel compounds have demonstrated rich functionalities but rarely shown superconductivity. Here, we report the emergence of superconductivity in the spinel $\mathrm{MgTi}_2\mathrm{O}_4$, known to be an insulator with a complicated order. The superconducting transition is achieved by engineering a superlattice of $\mathrm{MgTi}_2\mathrm{O}_4$ and $\mathrm{SrTiO}_3$. The onset transition temperature in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer can be tuned from 0 to 5 K in such geometry, concurrently with a stretched $c$-axis (from 8.51 to 8.53 Å) compared to the bulk material. Such a positive correlation without saturation suggests ample room for the further enhancement. Intriguingly, the superlattice exhibits isotropic upper critical field $H_{\mathrm{c}2}$ that breaks the Pauli limit, distinct from the highly anisotropic feature of interface superconductivity. The origin of superconductivity in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer is understood in combination with the electron energy loss spectra and the first-principles electronic structure calculations, which point to the birth of superconductivity in the $\mathrm{MgTi}_2\mathrm{O}_4$ layer by preventing the Ti-Ti dimerization. Our discovery not only provides a platform to explore the interplay between the superconductivity and other exotic states, but also opens a new window to realize superconductivity in the spinel compounds as well as other titanium oxides.

cond-mat.supr-con↗

arXiv2019

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

In this work, we propose a novel technique to boost training efficiency of a neural network. Our work is based on an excellent idea that whitening the inputs of neural networks can achieve a fast convergence speed. Given the well-known fact that independent components must be whitened, we introduce a novel Independent-Component (IC) layer before each weight layer, whose inputs would be made more independent. However, determining independent components is a computationally intensive task. To overcome this challenge, we propose to implement an IC layer by combining two popular techniques, Batch Normalization and Dropout, in a new manner that we can rigorously prove that Dropout can quadratically reduce the mutual information and linearly reduce the correlation between any pair of neurons with respect to the dropout layer parameter $p$. As demonstrated experimentally, the IC layer consistently outperforms the baseline approaches with more stable training process, faster convergence speed and better convergence limit on CIFAR10/100 and ILSVRC2012 datasets. The implementation of our IC layer makes us rethink the common practices in the design of neural networks. For example, we should not place Batch Normalization before ReLU since the non-negative responses of ReLU will make the weight layer updated in a suboptimal way, and we can achieve better performance by combining Batch Normalization and Dropout together as an IC layer.

cs.LG↗

arXiv2018

Learning Pixel-wise Labeling from the Internet without Human Interaction

Deep learning stands at the forefront in many computer vision tasks. However, deep neural networks are usually data-hungry and require a huge amount of well-annotated training samples. Collecting sufficient annotated data is very expensive in many applications, especially for pixel-level prediction tasks such as semantic segmentation. To solve this fundamental issue, we consider a new challenging vision task, Internetly supervised semantic segmentation, which only uses Internet data with noisy image-level supervision of corresponding query keywords for segmentation model training. We address this task by proposing the following solution. A class-specific attention model unifying multiscale forward and backward convolutional features is proposed to provide initial segmentation "ground truth". The model trained with such noisy annotations is then improved by an online fine-tuning procedure. It achieves state-of-the-art performance under the weakly-supervised setting on PASCAL VOC2012 dataset. The proposed framework also paves a new way towards learning from the Internet without human interaction and could serve as a strong baseline therein. Code and data will be released upon the paper acceptance.

cs.CV↗