arXiv subjects

Huiwen Xue

Publications and source records attributed to Huiwen Xue.

4 recordsLinked to original sources

arXiv2023

RPN: A Word Vector Level Data Augmentation Algorithm in Deep Learning for Language Understanding

Data augmentation is a widely used technique in machine learning to improve model performance. However, existing data augmentation techniques in natural language understanding (NLU) may not fully capture the complexity of natural language variations, and they can be challenging to apply to large datasets. This paper proposes the Random Position Noise (RPN) algorithm, a novel data augmentation technique that operates at the word vector level. RPN modifies the word embeddings of the original text by introducing noise based on the existing values of selected word vectors, allowing for more fine-grained modifications and better capturing natural language variations. Unlike traditional data augmentation methods, RPN does not require gradients in the computational graph during virtual sample updates, making it simpler to apply to large datasets. Experimental results demonstrate that RPN consistently outperforms existing data augmentation techniques across various NLU tasks, including sentiment analysis, natural language inference, and paraphrase detection. Moreover, RPN performs well in low-resource settings and is applicable to any model featuring a word embeddings layer. The proposed RPN algorithm is a promising approach for enhancing NLU performance and addressing the challenges associated with traditional data augmentation techniques in large-scale NLU tasks. Our experimental results demonstrated that the RPN algorithm achieved state-of-the-art performance in all seven NLU tasks, thereby highlighting its effectiveness and potential for real-world NLU applications.

cs.CL↗

arXiv2023

Hulk: Graph Neural Networks for Optimizing Regionally Distributed Computing Systems

Large deep learning models have shown great potential for delivering exceptional results in various applications. However, the training process can be incredibly challenging due to the models' vast parameter sizes, often consisting of hundreds of billions of parameters. Common distributed training methods, such as data parallelism, tensor parallelism, and pipeline parallelism, demand significant data communication throughout the process, leading to prolonged wait times for some machines in physically distant distributed systems. To address this issue, we propose a novel solution called Hulk, which utilizes a modified graph neural network to optimize distributed computing systems. Hulk not only optimizes data communication efficiency between different countries or even different regions within the same city, but also provides optimal distributed deployment of models in parallel. For example, it can place certain layers on a machine in a specific region or pass specific parameters of a model to a machine in a particular location. By using Hulk in experiments, we were able to improve the time efficiency of training large deep learning models on distributed systems by more than 20\%. Our open source collection of unlabeled data:https://github.com/DLYuanGod/Hulk.

cs.DC↗

arXiv2023

EvoText: Enhancing Natural Language Generation Models via Self-Escalation Learning for Up-to-Date Knowledge and Improved Performance

In recent years, pretrained models have been widely used in various fields, including natural language understanding, computer vision, and natural language generation. However, the performance of these language generation models is highly dependent on the model size and the dataset size. While larger models excel in some aspects, they cannot learn up-to-date knowledge and are relatively difficult to relearn. In this paper, we introduce EvoText, a novel training method that enhances the performance of any natural language generation model without requiring additional datasets during the entire training process (although a prior dataset is necessary for pretraining). EvoText employs two models: $G$, a text generation model, and $D$, a model that can determine whether the data generated by $G$ is legitimate. Initially, the fine-tuned $D$ model serves as the knowledge base. The text generated by $G$ is then input to $D$ to determine whether it is legitimate. Finally, $G$ is fine-tuned based on $D$'s output. EvoText enables the model to learn up-to-date knowledge through a self-escalation process that builds on a priori knowledge. When EvoText needs to learn something new, it simply fine-tunes the $D$ model. Our approach applies to autoregressive language modeling for all Transformer classes. With EvoText, eight models achieved stable improvements in seven natural language processing tasks without any changes to the model structure.

cs.CL↗

arXiv2021

High order finite difference WENO methods with unequal-sized sub-stencils for the Degasperis-Procesi type equations

In this paper, we develop two finite difference weighted essentially non-oscillatory (WENO) schemes with unequal-sized sub-stencils for solving the Degasperis-Procesi (DP) and $μ$-Degasperis-Procesi ($μ$DP) equations, which contain nonlinear high order derivatives, and possibly peakon solutions or shock waves. By introducing auxiliary variable(s), we rewrite the DP equation as a hyperbolic-elliptic system, and the \mdp equation as a first order system. Then we choose a linear finite difference scheme with suitable order of accuracy for the auxiliary variable(s), and two finite difference WENO schemes with unequal-sized sub-stencils for the primal variable. One WENO scheme uses one large stencil and several smaller stencils, and the other WENO scheme is based on the multi-resolution framework which uses a series of unequal-sized hierarchical central stencils. Comparing with the classical WENO scheme which uses several small stencils of the same size to make up a big stencil, both WENO schemes with unequal-sized sub-stencils are simple in the choice of the stencil and enjoy the freedom of arbitrary positive linear weights. Another advantage is that the final reconstructed polynomial on the target cell is a polynomial of the same degree as the polynomial over the big stencil, while the classical finite difference WENO reconstruction can only be obtained for specific points inside the target interval. Numerical tests are provided to demonstrate the high order accuracy and non-oscillatory properties of the proposed schemes.

math.NA↗