arXiv subjects

Dezhou Shen

Publications and source records attributed to Dezhou Shen.

4 recordsLinked to original sources

arXiv2022

FPM: A Collection of Large-scale Foundation Pre-trained Language Models

Large-scale Transformer models have significantly promoted the recent development of natural language processing applications. However, little effort has been made to unify the effective models. In this paper, driven by providing a new set of baseline models in the future, we adopt various novel transformer architectures and launch a model set with the help of recent mainstream technologies. We focus the discussions on optimizing the depth of the networks based on the existing powerful encode-decoder structures. We show that by properly avoiding training defects such as non-convergence and degradation, scaling up off-the-shelf transformer architectures consistently delivers better performance. To stimulate future research on large-scale language model pretraining, we present extensive results and detailed discussions on network performance improvements with respect to the network depth and confirm the existence of the optimal number of layers under specific tasks. To the best of our knowledge, we provide the largest Chinese generative model and the largest Chinese encoding model. The BERT language models we trained on English datasets deliver a 14.45% higher F1 score than the Turing-NLR.

cs.CL↗

arXiv2022

FoundationLayerNorm: Scaling BERT and GPT to 1,000 Layers

The mainstream BERT/GPT model contains only 10 to 20 layers, and there is little literature to discuss the training of deep BERT/GPT. This paper proposes a simple yet effective method to stabilize BERT and GPT training. We successfully scale up BERT and GPT to 1,000 layers, which is an order of magnitude deeper than previous BERT and GPT. The proposed method FoundationLayerNormalization enables efficient training of deep neural networks and is validated at the 1000-layer scale.

cs.CL↗

arXiv2020

Lower Bounds on Rate of Convergence of Matrix Products in All Pairs Shortest Path of Social Network

With the rapid development of social network applications, social network has become an important medium for people to interact. For the minimum distance computation of all pairs in networks, Alon N[4] proposed an algorithm with matrix multiplication, combining with distance product association law and block matrix multiplication, all pairs shortest path length algorithm on networks has time bound O((2n^3)/B logn). In practical applications, considering the scale-free characteristics of social networks and the precision limitations of floating-point operations on computer hardware, I found that the shortest path algorithm has an improved time bound O((14n^3)/B). Based on the above theory, I propose an all pairs shortest path algorithm that combines sparseness judgment and convergence judgment, leveraging the distance product algorithm with matrix multiplication, distance product association law, block matrix multiplication, scale-free characteristics of social networks, and limitation of floating-point operations on hardware. Testing on a social network dataset with 8508 actors, compared to Alon N algorithm, proposed algorithm has a performance improvement of 39% to 36.2 times on CPU and GPU.

cs.DS↗

arXiv2020

Movie Box office Prediction via Joint Actor Representations and Social Media Sentiment

In recent years, driven by the Asian film industry, such as China and India, the global box office has maintained a steady growth trend. Previous studies have rarely used long-term, full-sample film data in analysis, lack of research on actors' social networks. Existing film box office prediction algorithms only use film meta-data, lack of using social network characteristics and the model is less interpretable. I propose a FC-GRU-CNN binary classification model in of box office prediction task, combining five characteristics, including the film meta-data, Sina Weibo text sentiment, actors' social network measurement, all pairs shortest path and actors' art contribution. Exploiting long-term memory ability of GRU layer in long sequences and the mapping ability of CNN layer in retrieving all pairs shortest path matrix features, proposed model is 14% higher in accuracy than the current best C-LSTM model.

cs.SI↗