arXiv subjects

Weixin Liu

Publications and source records attributed to Weixin Liu.

23 records · Page 2Linked to original sources

arXiv2021

ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

cs.CL↗

arXiv2021

Integrating Auxiliary Information in Self-supervised Learning

This paper presents to integrate the auxiliary information (e.g., additional attributes for data such as the hashtags for Instagram images) in the self-supervised learning process. We first observe that the auxiliary information may bring us useful information about data structures: for instance, the Instagram images with the same hashtags can be semantically similar. Hence, to leverage the structural information from the auxiliary information, we present to construct data clusters according to the auxiliary information. Then, we introduce the Clustering InfoNCE (Cl-InfoNCE) objective that learns similar representations for augmented variants of data from the same cluster and dissimilar representations for data from different clusters. Our approach contributes as follows: 1) Comparing to conventional self-supervised representations, the auxiliary-information-infused self-supervised representations bring the performance closer to the supervised representations; 2) The presented Cl-InfoNCE can also work with unsupervised constructed clusters (e.g., k-means clusters) and outperform strong clustering-based self-supervised learning approaches, such as the Prototypical Contrastive Learning (PCL) method; 3) We show that Cl-InfoNCE may be a better approach to leverage the data clustering information, by comparing it to the baseline approach - learning to predict the clustering assignments with cross-entropy loss. For analysis, we connect the goodness of the learned representations with the statistical relationships: i) the mutual information between the labels and the clusters and ii) the conditional entropy of the clusters given the labels.

cs.LG↗

arXiv2021

ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression

Pretrained language models (PLMs) such as BERT adopt a training paradigm which first pretrain the model in general data and then finetune the model on task-specific data, and have recently achieved great success. However, PLMs are notorious for their enormous parameters and hard to be deployed on real-life applications. Knowledge distillation has been prevailing to address this problem by transferring knowledge from a large teacher to a much smaller student over a set of data. We argue that the selection of thee three key components, namely teacher, training data, and learning objective, is crucial to the effectiveness of distillation. We, therefore, propose a four-stage progressive distillation framework ERNIE-Tiny to compress PLM, which varies the three components gradually from general level to task-specific level. Specifically, the first stage, General Distillation, performs distillation with guidance from pretrained teacher, gerenal data and latent distillation loss. Then, General-Enhanced Distillation changes teacher model from pretrained teacher to finetuned teacher. After that, Task-Adaptive Distillation shifts training data from general data to task-specific data. In the end, Task-Specific Distillation, adds two additional losses, namely Soft-Label and Hard-Label loss onto the last stage. Empirical results demonstrate the effectiveness of our framework and generalization gain brought by ERNIE-Tiny.In particular, experiments show that a 4-layer ERNIE-Tiny maintains over 98.0%performance of its 12-layer teacher BERT base on GLUE benchmark, surpassing state-of-the-art (SOTA) by 1.0% GLUE score with the same amount of parameters. Moreover, ERNIE-Tiny achieves a new compression SOTA on five Chinese NLP tasks, outperforming BERT base by 0.4% accuracy with 7.5x fewer parameters and9.4x faster inference speed.

cs.CL↗

arXiv2021

Population of Bright Plume Threads in Solar Polar Coronal Holes

Coronal holes are well accepted to be source regions of the fast solar wind. As one of the common structures in coronal holes, coronal plumes might contribute to the origin of the nascent solar wind. To estimate the contribution of coronal plumes to the nascent solar wind, we make the first attempt to estimate their populations in solar polar coronal holes. By comparing the observations viewed from two different angles taken by the twin satellites of STEREO and the results of Monte Carlo simulations, we estimate about 16--27 plumes rooted in an area of $4\times10^4$ arcsec$^2$ of the polar coronal holes near the solar minimum, which occupy about 2--3.4% of the area. Based on these values, the contribution of coronal plumes to the nascent solar wind has also been discussed. A further investigation indicates that more precise number of coronal plumes can be worked out with observations from three or more viewing angles.

astro-ph.SR↗

arXiv2019

On the relation between transition region network jets and coronal plumes

Both coronal plumes and network jets are rooted in network lanes. The relationship between the two, however, has yet to be addressed. For this purpose, we perform an observational analysis using images acquired with the Atmospheric Imaging Assembly (AIA) 171Å passband to follow the evolution of coronal plumes, the observations taken by the Interface Region Imaging Spectrograph (IRIS) slit-jaw 1330Å to study the network jets, and the line-of-sight magnetograms taken by the Helioseismic and Magnetic Imager (HMI) to overview the the photospheric magnetic features in the regions. Four regions in the network lanes are identified, and labeled ``R1--R4''. We find that coronal plumes are clearly seen only in ``R1''&''R2'' but not in ``R3''&``R4'', even though network jets abound in all these regions. Furthermore, while magnetic features in all these regions are dominated by positive polarity, they are more compact (suggesting stronger convergence) in ``R1''&``R2'' than that in ``R3''&``R4''. We develop an automated method to identify and track the network jets in the regions. We find that the network jets rooted in ``R1''&``R2'' are higher and faster than that in ``R3''&``R4'',indicating that network regions producing stronger coronal plumes also tend to produce more dynamic network jets. We suggest that the stronger convergence in ``R1''&``R2'' might provide a condition for faster shocks and/or more small-scale magnetic reconnection events that power more dynamic network jets and coronal plumes.

astro-ph.SR↗