arXiv subjects

Wenhui Wang

Publications and source records attributed to Wenhui Wang.

At least 55 records · Page 3Linked to original sources

arXiv2021

Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains

Large pre-trained models have achieved great success in many natural language processing tasks. However, when they are applied in specific domains, these models suffer from domain shift and bring challenges in fine-tuning and online serving for latency and capacity constraints. In this paper, we present a general approach to developing small, fast and effective pre-trained models for specific domains. This is achieved by adapting the off-the-shelf general pre-trained models and performing task-agnostic knowledge distillation in target domains. Specifically, we propose domain-specific vocabulary expansion in the adaptation stage and employ corpus level occurrence probability to choose the size of incremental vocabulary automatically. Then we systematically explore different strategies to compress the large pre-trained models for specific domains. We conduct our experiments in the biomedical and computer science domain. The experimental results demonstrate that our approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3x smaller and 5.1x faster than BERT BASE. The code and pre-trained models are available at https://aka.ms/adalm.

cs.CL↗

arXiv2021

MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

We generalize deep self-attention distillation in MiniLM (Wang et al., 2020) by only using self-attention relation distillation for task-agnostic compression of pretrained Transformers. In particular, we define multi-head self-attention relations as scaled dot-product between the pairs of query, key, and value vectors within each self-attention module. Then we employ the above relational knowledge to train the student model. Besides its simplicity and unified principle, more favorably, there is no restriction in terms of the number of student's attention heads, while most previous work has to guarantee the same head number between teacher and student. Moreover, the fine-grained self-attention relations tend to fully exploit the interaction knowledge learned by Transformer. In addition, we thoroughly examine the layer selection strategy for teacher models, rather than just relying on the last layer as in MiniLM. We conduct extensive experiments on compressing both monolingual and multilingual pretrained models. Experimental results demonstrate that our models distilled from base-size and large-size teachers (BERT, RoBERTa and XLM-R) outperform the state-of-the-art.

cs.CL↗

arXiv2021

Consistency Regularization for Cross-Lingual Fine-Tuning

Fine-tuning pre-trained cross-lingual language models can transfer task-specific supervision from one language to the others. In this work, we propose to improve cross-lingual fine-tuning with consistency regularization. Specifically, we use example consistency regularization to penalize the prediction sensitivity to four types of data augmentations, i.e., subword sampling, Gaussian noise, code-switch substitution, and machine translation. In addition, we employ model consistency to regularize the models trained with two augmented versions of the same training set. Experimental results on the XTREME benchmark show that our method significantly improves cross-lingual fine-tuning across various tasks, including text classification, question answering, and sequence labeling.

cs.CL↗

arXiv2021

InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training

In this work, we present an information-theoretic framework that formulates cross-lingual language model pre-training as maximizing mutual information between multilingual-multi-granularity texts. The unified view helps us to better understand the existing methods for learning cross-lingual representations. More importantly, inspired by the framework, we propose a new pre-training task based on contrastive learning. Specifically, we regard a bilingual sentence pair as two views of the same meaning and encourage their encoded representations to be more similar than the negative examples. By leveraging both monolingual and parallel corpora, we jointly train the pretext tasks to improve the cross-lingual transferability of pre-trained models. Experimental results on several benchmarks show that our approach achieves considerably better performance. The code and pre-trained models are available at https://aka.ms/infoxlm.

cs.CL↗

arXiv2020

Harvesting and Refining Question-Answer Pairs for Unsupervised QA

Question Answering (QA) has shown great success thanks to the availability of large-scale datasets and the effectiveness of neural models. Recent research works have attempted to extend these successes to the settings with few or no labeled data available. In this work, we introduce two approaches to improve unsupervised QA. First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA). Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA. We conduct experiments on SQuAD 1.1, and NewsQA by fine-tuning BERT without access to manually annotated data. Our approach outperforms previous unsupervised approaches by a large margin and is competitive with early supervised models. We also show the effectiveness of our approach in the few-shot learning setting.

cs.CL↗

arXiv2020

Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences

Neuromorphic data, recording frameless spike events, have attracted considerable attention for the spatiotemporal information components and the event-driven processing fashion. Spiking neural networks (SNNs) represent a family of event-driven models with spatiotemporal dynamics for neuromorphic computing, which are widely benchmarked on neuromorphic data. Interestingly, researchers in the machine learning community can argue that recurrent (artificial) neural networks (RNNs) also have the capability to extract spatiotemporal features although they are not event-driven. Thus, the question of "what will happen if we benchmark these two kinds of models together on neuromorphic data" comes out but remains unclear. In this work, we make a systematic study to compare SNNs and RNNs on neuromorphic data, taking the vision datasets as a case study. First, we identify the similarities and differences between SNNs and RNNs (including the vanilla RNNs and LSTM) from the modeling and learning perspectives. To improve comparability and fairness, we unify the supervised learning algorithm based on backpropagation through time (BPTT), the loss function exploiting the outputs at all timesteps, the network structure with stacked fully-connected or convolutional layers, and the hyper-parameters during training. Especially, given the mainstream loss function used in RNNs, we modify it inspired by the rate coding scheme to approach that of SNNs. Furthermore, we tune the temporal resolution of datasets to test model robustness and generalization. At last, a series of contrast experiments are conducted on two types of neuromorphic datasets: DVS-converted (N-MNIST) and DVS-captured (DVS Gesture).

cs.CV↗

arXiv2020

MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

Pre-trained language models (e.g., BERT (Devlin et al., 2018) and its variants) have achieved remarkable success in varieties of NLP tasks. However, these models usually consist of hundreds of millions of parameters which brings challenges for fine-tuning and online serving in real-life applications due to latency and capacity constraints. In this work, we present a simple and effective approach to compress large Transformer (Vaswani et al., 2017) based pre-trained models, termed as deep self-attention distillation. The small model (student) is trained by deeply mimicking the self-attention module, which plays a vital role in Transformer networks, of the large model (teacher). Specifically, we propose distilling the self-attention module of the last Transformer layer of the teacher, which is effective and flexible for the student. Furthermore, we introduce the scaled dot-product between values in the self-attention module as the new deep self-attention knowledge, in addition to the attention distributions (i.e., the scaled dot-product of queries and keys) that have been used in existing works. Moreover, we show that introducing a teacher assistant (Mirzadeh et al., 2019) also helps the distillation of large pre-trained Transformer models. Experimental results demonstrate that our monolingual model outperforms state-of-the-art baselines in different parameter size of student models. In particular, it retains more than 99% accuracy on SQuAD 2.0 and several GLUE benchmark tasks using 50% of the Transformer parameters and computations of the teacher model. We also obtain competitive results in applying deep self-attention distillation to multilingual pre-trained models.

cs.CL↗

arXiv2020

UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training

We propose to pre-train a unified language model for both autoencoding and partially autoregressive language modeling tasks using a novel training procedure, referred to as a pseudo-masked language model (PMLM). Given an input text with masked tokens, we rely on conventional masks to learn inter-relations between corrupted tokens and context via autoencoding, and pseudo masks to learn intra-relations between masked spans via partially autoregressive modeling. With well-designed position embeddings and self-attention masks, the context encodings are reused to avoid redundant computation. Moreover, conventional masks used for autoencoding provide global masking information, so that all the position embeddings are accessible in partially autoregressive language modeling. In addition, the two tasks pre-train a unified language model as a bidirectional encoder and a sequence-to-sequence decoder, respectively. Our experiments show that the unified language models pre-trained using PMLM achieve new state-of-the-art results on a wide range of natural language understanding and generation tasks across several widely used benchmarks.

cs.CL↗

arXiv2019

Cross-Lingual Natural Language Generation via Pre-Training

In this work we focus on transferring supervision signals of natural language generation (NLG) tasks between multiple languages. We propose to pretrain the encoder and the decoder of a sequence-to-sequence model under both monolingual and cross-lingual settings. The pre-training objective encourages the model to represent different languages in the shared space, so that we can conduct zero-shot cross-lingual transfer. After the pre-training procedure, we use monolingual data to fine-tune the pre-trained model on downstream NLG tasks. Then the sequence-to-sequence model trained in a single language can be directly evaluated beyond that language (i.e., accepting multi-lingual input and producing multi-lingual output). Experimental results on question generation and abstractive summarization show that our model outperforms the machine-translation-based pipeline methods for zero-shot cross-lingual generation. Moreover, cross-lingual transfer improves NLG performance of low-resource languages by leveraging rich-resource language data. Our implementation and data are available at https://github.com/CZWin32768/xnlg.

cs.CL↗

arXiv2019

Unified Language Model Pre-training for Natural Language Understanding and Generation

This paper presents a new Unified pre-trained Language Model (UniLM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirectional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UniLM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UniLM achieves new state-of-the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm.

cs.CL↗

arXiv2019

High-performance silicon-graphene hybrid plasmonic waveguide photodetectors beyond 1.55 μm

A fast silicon-graphene hybrid plasmonic waveguide photodetectors beyond 1.55 μm is proposed and realized by introducing an ultra-thin wide silicon-on-insulator ridge core region with a narrow metal cap. With this novel design, the light absorption in graphene is enhanced while the metal absorption loss is reduced simultaneously, which helps greatly improve the responsivity as well as shorten the absorption region for achieving fast responses. Furthermore, metal-graphene-metal sandwiched electrodes are introduced to reduce the metal-graphene contact resistance, which is also helpful for improving the response speed. When the photodetector operates at 2 μm, the measured 3dB-bandwidth is >20 GHz (which is limited by the experimental setup) while the 3dB-bandwith calculated from the equivalent circuit with the parameters extracted from the measured S11 is as high as ~100 GHz. To the best of our knowledge, it is the first time to report the waveguide photodetector at 2 μm with a 3dB-bandwidth over 20 GHz. Besides, the present photodetectors also work very well at 1.55 μm. The measured responsivity is about 0.4 A/W under a bias voltage of -0.3 V for an optical power of 0.16 mW, while the measured 3dB-bandwidth is over 40 GHz (limited by the test setup) and the 3 dB-bandwidth estimated from the equivalent circuit is also as high as ~100 GHz, which is one of the best results reported for silicon-graphene photodetectors at 1.55 μm.

physics.app-ph↗

arXiv2019

Isolating hydrogen in hexagonal boron nitride bubbles by a plasma treatment

Atomically thin hexagonal boron nitride (h-BN) is often regarded as an elastic film that is impermeable to gases. The high stabilities in thermal and chemical properties allow h-BN to serve as a gas barrier under extreme conditions.In this work, we demonstrate the isolation of hydrogen in bubbles of h-BN via plasma treatment.Detailed characterizations reveal that the substrates do not show chemical change after treatment. The bubbles are found to withstand thermal treatment in air,even at 800 degree celsius. Scanning transmission electron microscopy investigation shows that the h-BN multilayer has a unique aligned porous stacking nature, which is essential for the character of being transparent to atomic hydrogen but impermeable to hydrogen molecules. We successfully demonstrated the extraction of hydrogen gases from gaseous compounds or mixtures containing hydrogen element. The successful production of hydrogen bubbles on h-BN flakes has potential for further application in nano/micro-electromechanical systems and hydrogen storage.

physics.app-ph↗

arXiv2019

Learning to Ask Unanswerable Questions for Machine Reading Comprehension

Machine reading comprehension with unanswerable questions is a challenging task. In this work, we propose a data augmentation technique by automatically generating relevant unanswerable questions according to an answerable question paired with its corresponding paragraph that contains the answer. We introduce a pair-to-sequence model for unanswerable question generation, which effectively captures the interactions between the question and the paragraph. We also present a way to construct training data for our question generation models by leveraging the existing reading comprehension dataset. Experimental results show that the pair-to-sequence model performs consistently better compared with the sequence-to-sequence baseline. We further use the automatically generated unanswerable questions as a means of data augmentation on the SQuAD 2.0 dataset, yielding 1.9 absolute F1 improvement with BERT-base model and 1.7 absolute F1 improvement with BERT-large model.

cs.CL↗

arXiv2018

Defect Engineering for Modulating the Trap States in Two-dimensional Photoconductor

Defect induced trap states are essential in determining the performance of semiconductor photodetectors. The de-trap time of carriers from a deep trap could be prolonged by several orders of magnitude as compared to shallow trap, resulting in additional decay/response time of the device. Here, we demonstrate that the trap states in two-dimensional ReS2 could be efficiently modulated by defect engineering through molecule decoration. The deep traps that greatly prolong the response time could be mostly filled by Protoporphyrin (H2PP) molecules. At the same time, carrier recombination and shallow traps would in-turn play dominant roles in determining the decay time of the device, which can be several orders of magnitude faster than the as-prepared device. Moreover, the specific detectivity of the device is enhanced (as high as ~1.89 x 10^13 Jones) due to the significant reduction of dark current through charge transfer between ReS2 and molecules. Defect engineering of trap states therefore provides a solution to achieve photodetectors with both high responsivity and fast response.

cond-mat.mtrl-sci↗

arXiv2018

Enhancing the Sensitivity of Searches for Gravitational Waves from Core-Collapse Supernovae with a Bayesian classification of candidate events

We demonstrate how a morphological veto involving Bayesian statistics can improve the receiver-operating characteristic (ROC) curves of the current search for core-collapse supernovae (CCSNe) as implemented by the coherent Waveburst (cWB) algorithm. Examples involving two implementations of BayesWave (BW), one that makes no assumption of the polarization state of the gravitational wave (GW) and one that uses the same elliptical polarization settings adopted in previous usages for Binary systems are provided on the set of waveforms currently adopted for the first and second Advanced LIGO (aLIGO) science runs for the targeted CCSNe search. A comparison of the performance for all-sky searches versus the targeted searches with optical triggers is provided. The average cWB+BW ROC range improvements with respect to a fixed false-alarm rate (FAR) for slowly-rotating waveforms ranged from [+1.30%, +15.76%] while the improvement for rapidly-rotating waveforms were on the order of [+1.19%, +22.05%]. The application of BW to CCSN GW triggers also shows a significant reduction of the FAR while maintaining detection efficiency and remaining sensitive to a wide range of morphological CCSNe signals. It appears that the code developed for arbitrarily polarized signals outperforms the previous code for the GW morphologies tested.

astro-ph.HE↗

arXiv2017

High performance position-sensitive-detector based on graphene-silicon heterojunction

Position-sensitive-detectors (PSDs) based on lateral photoeffect have been widely used in diverse applications, including optical engineering, aerospace and military fields. With increasing demands in long working distance, low energy consumption, and weak signal sensing systems, the poor responsivity of conventional Silicon-based PSDs has become a bottleneck limiting their applications. Herein, we propose a high-performance passive PSD based on graphene-Si heterostructure. The graphene is adapted as a photon absorbing and charge separation layer working together with Si as a junction, while the high mobility provides promising ultra-long carrier diffusion length and facilitates large active area of the device. A PSD with working area of 8 mm x 8 mm is demonstrated to present excellent position sensitivity to weak light at nWs level (much better than the limit of ~μWs of Si p-i-n PSDs). More importantly, it shows very fast response and low degree of non-linearity of ~3%, and extends the operating wavelength to the near infrared (IR) region (1319 and 1550 nm). This work therefore provides a new strategy for high performance and broadband PSDs.

physics.app-ph↗

arXiv2017

High-performance graphene-based electrostatic field sensor

Electrostatic sensing technology is widely utilized in both military and civilian applications, including electrostatic prevention in gas stations and various electronic devices. The high sensitivity of electrostatic sensor is capable to detect not only weak electrostatic charges, but also the weak disturbance of electrostatic field in distant. Here, we present a high-performance graphene-based electrostatic sensor. Combining the ultrahigh mobility of graphene and the long lifetime of carriers in lightly doped SiO2/Si substrate, our device achieves a fast response of ~2 us and detection limit of electrostatic potential as low as ~5 V, which is improved by an order of magnitude as compared to commercial product. The proposed device structure opens a promising pathway to high-sensitive electrostatic detection, and also greatly facilitates the development of novel sensors, e.g. portable and flexible electrostatic sensor.

cond-mat.mtrl-sci↗

arXiv2016

High-performance graphene photodetector by interfacial gating

Graphene based photo-detecting has received great attentions and the performance of such detector is stretching to both ends of high sensitivity and ultra-fast response. However, limited by the current photo-gating mechanism, the price for achieving ultra-high sensitivity is sacrificing the response time. Detecting weak signal within short response time is crucial especially in applications such as optical positioning, remote sensing, and biomedical imaging. In this work, we bridge the gap between ultra-fast response and ultra-high sensitivity by employing a graphene/SiO2/lightly-doped-Si architecture with revolutionary interfacial gating mechanism. Such device is capable to detect < 1 nW signal (with responsivity of ~1000 A W-1) and the spectral response extends from visible to near-infrared. More importantly, the photoresponse time of our device has been pushed to ~400 ns. The current device structure does not need complicated fabrication process and is fully compatible with the silicon technology. This work will not only open up a route to graphene-based high performance optoelectronic devices, but also have great potential in ultra-fast weak signal detection.

physics.ins-det↗