arXiv subjects

Zheng Zhang

Publications and source records attributed to Zheng Zhang.

At least 523 records · Page 29Linked to original sources

arXiv2021

Video Swin Transformer

The vision community is witnessing a modeling shift from CNNs to Transformers, where pure Transformer architectures have attained top accuracy on the major video recognition benchmarks. These video models are all built on Transformer layers that globally connect patches across the spatial and temporal dimensions. In this paper, we instead advocate an inductive bias of locality in video Transformers, which leads to a better speed-accuracy trade-off compared to previous approaches which compute self-attention globally even with spatial-temporal factorization. The locality of the proposed video architecture is realized by adapting the Swin Transformer designed for the image domain, while continuing to leverage the power of pre-trained image models. Our approach achieves state-of-the-art accuracy on a broad range of video recognition benchmarks, including on action recognition (84.9 top-1 accuracy on Kinetics-400 and 86.1 top-1 accuracy on Kinetics-600 with ~20x less pre-training data and ~3x smaller model size) and temporal modeling (69.6 top-1 accuracy on Something-Something v2). The code and models will be made publicly available at https://github.com/SwinTransformer/Video-Swin-Transformer.

cs.CV↗

arXiv2021

DS-TransUNet:Dual Swin Transformer U-Net for Medical Image Segmentation

Automatic medical image segmentation has made great progress benefit from the development of deep learning. However, most existing methods are based on convolutional neural networks (CNNs), which fail to build long-range dependencies and global context connections due to the limitation of receptive field in convolution operation. Inspired by the success of Transformer in modeling the long-range contextual information, some researchers have expended considerable efforts in designing the robust variants of Transformer-based U-Net. Moreover, the patch division used in vision transformers usually ignores the pixel-level intrinsic structural features inside each patch. To alleviate these problems, we propose a novel deep medical image segmentation framework called Dual Swin Transformer U-Net (DS-TransUNet), which might be the first attempt to concurrently incorporate the advantages of hierarchical Swin Transformer into both encoder and decoder of the standard U-shaped architecture to enhance the semantic segmentation quality of varying medical images. Unlike many prior Transformer-based solutions, the proposed DS-TransUNet first adopts dual-scale encoder subnetworks based on Swin Transformer to extract the coarse and fine-grained feature representations of different semantic scales. As the core component for our DS-TransUNet, a well-designed Transformer Interactive Fusion (TIF) module is proposed to effectively establish global dependencies between features of different scales through the self-attention mechanism. Furthermore, we also introduce the Swin Transformer block into decoder to further explore the long-range contextual information during the up-sampling process. Extensive experiments across four typical tasks for medical image segmentation demonstrate the effectiveness of DS-TransUNet, and show that our approach significantly outperforms the state-of-the-art methods.

cs.CV↗

arXiv2021

A Unified Generative Framework for Aspect-Based Sentiment Analysis

Aspect-based Sentiment Analysis (ABSA) aims to identify the aspect terms, their corresponding sentiment polarities, and the opinion terms. There exist seven subtasks in ABSA. Most studies only focus on the subsets of these subtasks, which leads to various complicated ABSA models while hard to solve these subtasks in a unified framework. In this paper, we redefine every subtask target as a sequence mixed by pointer indexes and sentiment class indexes, which converts all ABSA subtasks into a unified generative formulation. Based on the unified formulation, we exploit the pre-training sequence-to-sequence model BART to solve all ABSA subtasks in an end-to-end framework. Extensive experiments on four ABSA datasets for seven subtasks demonstrate that our framework achieves substantial performance gain and provides a real unified end-to-end solution for the whole ABSA subtasks, which could benefit multiple tasks.

cs.CL↗

arXiv2021

A Unified Generative Framework for Various NER Subtasks

Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-the-art (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.

cs.CL↗

arXiv2021

Active Sampling for Accelerated MRI with Low-Rank Tensors

Magnetic resonance imaging (MRI) is a powerful imaging modality that revolutionizes medicine and biology. The imaging speed of high-dimensional MRI is often limited, which constrains its practical utility. Recently, low-rank tensor models have been exploited to enable fast MR imaging with sparse sampling. Most existing methods use some pre-defined sampling design, and active sensing has not been explored for low-rank tensor imaging. In this paper, we introduce an active low-rank tensor model for fast MR imaging. We propose an active sampling method based on a Query-by-Committee model, making use of the benefits of low-rank tensor structure. Numerical experiments on a 3-D MRI data set demonstrate the effectiveness of the proposed method.

cs.CV↗

arXiv2021

Prototype-supervised Adversarial Network for Targeted Attack of Deep Hashing

Due to its powerful capability of representation learning and high-efficiency computation, deep hashing has made significant progress in large-scale image retrieval. However, deep hashing networks are vulnerable to adversarial examples, which is a practical secure problem but seldom studied in hashing-based retrieval field. In this paper, we propose a novel prototype-supervised adversarial network (ProS-GAN), which formulates a flexible generative architecture for efficient and effective targeted hashing attack. To the best of our knowledge, this is the first generation-based method to attack deep hashing networks. Generally, our proposed framework consists of three parts, i.e., a PrototypeNet, a generator, and a discriminator. Specifically, the designed PrototypeNet embeds the target label into the semantic representation and learns the prototype code as the category-level representative of the target label. Moreover, the semantic representation and the original image are jointly fed into the generator for a flexible targeted attack. Particularly, the prototype code is adopted to supervise the generator to construct the targeted adversarial example by minimizing the Hamming distance between the hash code of the adversarial example and the prototype code. Furthermore, the generator is against the discriminator to simultaneously encourage the adversarial examples visually realistic and the semantic representation informative. Extensive experiments verify that the proposed framework can efficiently produce adversarial examples with better targeted attack performance and transferability over state-of-the-art targeted attack methods of deep hashing. The related codes could be available at https://github.com/xunguangwang/ProS-GAN .

cs.CV↗

arXiv2021

Self-Supervised Learning with Swin Transformers

We are witnessing a modeling shift from CNN to Transformers in computer vision. In this work, we present a self-supervised learning approach called MoBY, with Vision Transformers as its backbone architecture. The approach basically has no new inventions, which is combined from MoCo v2 and BYOL and tuned to achieve reasonably high accuracy on ImageNet-1K linear evaluation: 72.8% and 75.0% top-1 accuracy using DeiT-S and Swin-T, respectively, by 300-epoch training. The performance is slightly better than recent works of MoCo v3 and DINO which adopt DeiT as the backbone, but with much lighter tricks. More importantly, the general-purpose Swin Transformer backbone enables us to also evaluate the learnt representations on downstream tasks such as object detection and semantic segmentation, in contrast to a few recent approaches built on ViT/DeiT which only report linear evaluation results on ImageNet-1K due to ViT/DeiT not tamed for these dense prediction tasks. We hope our results can facilitate more comprehensive evaluation of self-supervised learning methods designed for Transformer architectures. Our code and models are available at https://github.com/SwinTransformer/Transformer-SSL, which will be continually enriched.

cs.CV↗

arXiv2021

3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

The deep neural network (DNN) based AI applications on the edge require both low-cost computing platforms and high-quality services. However, the limited memory, computing resources, and power budget of the edge devices constrain the effectiveness of the DNN algorithms. Developing edge-oriented AI algorithms and implementations (e.g., accelerators) is challenging. In this paper, we summarize our recent efforts for efficient on-device AI development from three aspects, including both training and inference. First, we present on-device training with ultra-low memory usage. We propose a novel rank-adaptive tensor-based tensorized neural network model, which offers orders-of-magnitude memory reduction during training. Second, we introduce an ultra-low bitwidth quantization method for DNN model compression, achieving the state-of-the-art accuracy under the same compression ratio. Third, we introduce an ultra-low latency DNN accelerator design, practicing the software/hardware co-design methodology. This paper emphasizes the importance and efficacy of training, quantization and accelerator design, and calls for more research breakthroughs in the area for AI on the edge.

cs.LG↗

arXiv2021

Towards Robust Neural Networks via Close-loop Control

Despite their success in massive engineering applications, deep neural networks are vulnerable to various perturbations due to their black-box nature. Recent study has shown that a deep neural network can misclassify the data even if the input data is perturbed by an imperceptible amount. In this paper, we address the robustness issue of neural networks by a novel close-loop control method from the perspective of dynamic systems. Instead of modifying the parameters in a fixed neural network architecture, a close-loop control process is added to generate control signals adaptively for the perturbed or corrupted data. We connect the robustness of neural networks with optimal control using the geometrical information of underlying data to design the control objective. The detailed analysis shows how the embedding manifolds of state trajectory affect error estimation of the proposed method. Our approach can simultaneously maintain the performance on clean data and improve the robustness against many types of data perturbations. It can also further improve the performance of robustly trained neural networks against different perturbations. To the best of our knowledge, this is the first work that improves the robustness of neural networks with close-loop control.

cs.LG↗

arXiv2021

Group-Free 3D Object Detection via Transformers

Recently, directly detecting 3D objects from 3D point clouds has received increasing attention. To extract object representation from an irregular point cloud, existing methods usually take a point grouping step to assign the points to an object candidate so that a PointNet-like network could be used to derive object features from the grouped points. However, the inaccurate point assignments caused by the hand-crafted grouping scheme decrease the performance of 3D object detection. In this paper, we present a simple yet effective method for directly detecting 3D objects from the 3D point cloud. Instead of grouping local points to each object candidate, our method computes the feature of an object from all the points in the point cloud with the help of an attention mechanism in the Transformers \cite{vaswani2017attention}, where the contribution of each point is automatically learned in the network training. With an improved attention stacking scheme, our method fuses object features in different stages and generates more accurate object detection results. With few bells and whistles, the proposed method achieves state-of-the-art 3D object detection performance on two widely used benchmarks, ScanNet V2 and SUN RGB-D. The code and models are publicly available at \url{https://github.com/zeliu98/Group-Free-3D}

cs.CV↗

arXiv2021

On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method

Various hardware accelerators have been developed for energy-efficient and real-time inference of neural networks on edge devices. However, most training is done on high-performance GPUs or servers, and the huge memory and computing costs prevent training neural networks on edge devices. This paper proposes a novel tensor-based training framework, which offers orders-of-magnitude memory reduction in the training process. We propose a novel rank-adaptive tensorized neural network model, and design a hardware-friendly low-precision algorithm to train this model. We present an FPGA accelerator to demonstrate the benefits of this training method on edge devices. Our preliminary FPGA implementation achieves $59\times$ speedup and $123\times$ energy reduction compared to embedded CPU, and $292\times$ memory reduction over a standard full-size training.

cs.AR↗

arXiv2021

Securing NOMA Networks by Exploiting Intelligent Reflecting Surface

This paper investigates the security enhancement of an intelligent reflecting surface (IRS) assisted non-orthogonal multiple access (NOMA) network, where a distributed IRS enabled NOMA transmission framework is proposed to serve users securely in the presence of a passive eavesdropper. Considering that eavesdropper's instantaneous channel state information (CSI) is challenging to acquire in practice, we utilize secrecy outage probability (SOP) as the security metric. A problem of maximizing the minimum secrecy rate among users, subject to the successive interference cancellation (SIC) decoding constraints and SOP constraints, by jointly optimizing transmit beamforming at the BS and phase shifts of IRSs, is formulated. For special case with a single-antenna BS, we derive the exact closed-form SOP expressions and propose a novel ring-penalty based successive convex approximation (SCA) algorithm to design power allocation and phase shifts jointly. While for the more general and challenging case with a multi-antenna BS, we adopt the Bernstein-type inequality to approximate the SOP constraints by a deterministic convex form. To proceed, an efficient alternating optimization (AO) algorithm is developed to solve the considered problem. Numerical results validate the advantages of the proposed algorithms over the baseline schemes. Particularly, two interesting phenomena on distributed IRS deployment are revealed: 1) the secrecy rate peak is achieved only when distributed IRSs share the reflecting elements equally; and 2) the distributed IRS deployment does not always outperform the centralized IRS deployment, due to the tradeoff between the number of IRSs and the reflecting elements equipped at each IRS.

cs.IT↗

arXiv2021

Effective Model for Rare-earth Kitaev Materials and its Classical Monte Carlo Simulation

Recently, the family of rare-earth chalcohalides were proposed as candidate compounds to realize the Kitaev spin liquid (KSL). In the present work, we firstly propose an effective spin Hamiltonian consistents with the symmetry group of the crystal structure. Then we apply classical Monte Carlo simulations to preliminarily study the model and establish a phase diagram. When approaching to the low temperature limit, several magnetic long range orders are observed, including the stripe, the zigzag, the antiferromagnetic (AFM), the ferromagnetic (FM), the incommensurate spiral (IS), the Multi-$\pmb {Q}$ and the 120°. We further calculate the thermodynamic properties of the system, such as the temperature dependence of the magnetic susceptibility and the heat capacity. The ordering transition temperatures reflected in the two quantities agree with each other. For most interaction regions, the system is magnetically more susceptible in the $ab$-plane than in the $c$-direction. The stripe phase is special, where the susceptibility is fairly isotropic in the whole temperature region. These features provide useful information to understand the magnetic properties of related materials.

cond-mat.str-el↗

arXiv2021

Rare-earth chalcohalides: A family of van der Waals layered Kitaev spin liquid candidates

Kitaev spin liquid (KSL) system has attracted tremendous attention in past years because of its fundamental significance in condensed matter physics and promising applications in fault-tolerant topological quantum computation. Material realization of such a system remains a major challenge in the field due to the unusual configuration of anisotropic spin interactions, though great effort has been made before. Here we reveal that rare-earth chalcohalides REChX (RE=rare earth, Ch=O, S, Se, Te, X=F, Cl, Br, I) can serve as a family of KSL candidates. Most family members have the typical SmSI-type structure with a high symmetry of R-3m and rare-earth magnetic ions form an undistorted honeycomb lattice. The strong spin-orbit coupling of 4f electrons intrinsically offers anisotropic spin interactions as required by Kitaev model. We have grown the crystals of YbOCl and synthesized the polycrystals of SmSI, ErOF, HoOF and DyOF, and made careful structural characterizations. We carry out magnetic and heat capacity measurements down to 1.8 K and find no obvious magnetic transition in all the samples but DyOF. The van der Waals interlayer coupling highlights the true two-dimensionality of the family which is vital for the exact realization of Abelian/non-Abelian anyons, and the graphene-like feature will be a prominent advantage for developing miniaturized devices. The family is expected to act as an inspiring material platform for the exploration of KSL physics.

cond-mat.str-el↗

arXiv2021

Trojaning Language Models for Fun and Profit

Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems. Specifically, we present TROJAN-LM, a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-critical NLP tasks (toxic comment detection, question answering, text completion) as well as user studies on crowdsourcing platforms, we demonstrate that TROJAN-LM possesses the following properties: (i) flexibility - the adversary is able to flexibly dene logical combinations (e.g., 'and', 'or', 'xor') of arbitrary words as triggers, (ii) efficacy - the host systems misbehave as desired by the adversary with high probability when trigger-embedded inputs are present, (iii) specificity - the trojan LMs function indistinguishably from their benign counterparts on clean inputs, and (iv) fluency - the trigger-embedded inputs appear as fluent natural language and highly relevant to their surrounding contexts. We provide analytical justification for the practicality of TROJAN-LM, and further discuss potential countermeasures and their challenges, which lead to several promising research directions.

cs.CR↗

arXiv2021

Propagate Yourself: Exploring Pixel-Level Consistency for Unsupervised Visual Representation Learning

Contrastive learning methods for unsupervised visual representation learning have reached remarkable levels of transfer performance. We argue that the power of contrastive learning has yet to be fully unleashed, as current methods are trained only on instance-level pretext tasks, leading to representations that may be sub-optimal for downstream tasks requiring dense pixel predictions. In this paper, we introduce pixel-level pretext tasks for learning dense feature representations. The first task directly applies contrastive learning at the pixel level. We additionally propose a pixel-to-propagation consistency task that produces better results, even surpassing the state-of-the-art approaches by a large margin. Specifically, it achieves 60.2 AP, 41.4 / 40.5 mAP and 77.2 mIoU when transferred to Pascal VOC object detection (C4), COCO object detection (FPN / C4) and Cityscapes semantic segmentation using a ResNet-50 backbone network, which are 2.6 AP, 0.8 / 1.0 mAP and 1.0 mIoU better than the previous best methods built on instance-level contrastive learning. Moreover, the pixel-level pretext tasks are found to be effective for pre-training not only regular backbone networks but also head networks used for dense downstream tasks, and are complementary to instance-level contrastive methods. These results demonstrate the strong potential of defining pretext tasks at the pixel level, and suggest a new path forward in unsupervised visual representation learning. Code is available at \url{https://github.com/zdaxie/PixPro}.

cs.CV↗

arXiv2021

Directional-TV Algorithm for Image Reconstruction from Limited-Angular-Range Data

Investigation of image reconstruction from data collected over a limited angular range in X-ray CT remains a topic of active research because it may yield insight into the development of imaging workflow of practical significance. This reconstruction problem is well-known to be challenging, however, because it is highly ill-conditioned. In the work, we investigate optimization-based image reconstruction from data acquired over a limited-angular range that is considerably smaller than the angular range in short-scan CT. We first formulate the reconstruction problem as a convex optimization program with directional total-variation (TV) constraints applied to the image, and then develop an iterative algorithm, referred to as the directional-TV (DTV) algorithm for image reconstruction through solving the optimization program. We use the DTV algorithm to reconstruct images from data collected over a variety of limited-angular ranges for breast and bar phantoms of clinical- and industrial-application relevance. The study demonstrates that the DTV algorithm accurately recovers the phantoms from data generated over a significantly reduced angular range, and that it considerably diminishes artifacts observed otherwise in reconstructions of existing algorithms. We have also obtained empirical conditions on minimal angular ranges sufficient for numerically accurate image reconstruction with the DTV algorithm.

physics.med-ph↗

arXiv2021

Bayesian Inference with Certifiable Adversarial Robustness

We consider adversarial training of deep neural networks through the lens of Bayesian learning, and present a principled framework for adversarial training of Bayesian Neural Networks (BNNs) with certifiable guarantees. We rely on techniques from constraint relaxation of non-convex optimisation problems and modify the standard cross-entropy error model to enforce posterior robustness to worst-case perturbations in $ε$-balls around input points. We illustrate how the resulting framework can be combined with methods commonly employed for approximate inference of BNNs. In an empirical investigation, we demonstrate that the presented approach enables training of certifiably robust models on MNIST, FashionMNIST and CIFAR-10 and can also be beneficial for uncertainty calibration. Our method is the first to directly train certifiable BNNs, thus facilitating their deployment in safety-critical applications.

cs.LG↗