arXiv subjects

Chengyi Wang

Publications and source records attributed to Chengyi Wang.

39 records · Page 3Linked to original sources

arXiv2019

Source Dependency-Aware Transformer with Supervised Self-Attention

Recently, Transformer has achieved the state-of-the-art performance on many machine translation tasks. However, without syntax knowledge explicitly considered in the encoder, incorrect context information that violates the syntax structure may be integrated into source hidden states, leading to erroneous translations. In this paper, we propose a novel method to incorporate source dependencies into the Transformer. Specifically, we adopt the source dependency tree and define two matrices to represent the dependency relations. Based on the matrices, two heads in the multi-head self-attention module are trained in a supervised manner and two extra cross entropy losses are introduced into the training objective function. Under this training objective, the model is trained to learn the source dependency relations directly. Without requiring pre-parsed input during inference, our model can generate better translations with the dependency-aware context information. Experiments on bi-directional Chinese-to-English, English-to-Japanese and English-to-German translation tasks show that our proposed method can significantly improve the Transformer baseline.

cs.CL↗

arXiv2019

Accelerating Transformer Decoding via a Hybrid of Self-attention and Recurrent Neural Network

Due to the highly parallelizable architecture, Transformer is faster to train than RNN-based models and popularly used in machine translation tasks. However, at inference time, each output word requires all the hidden states of the previously generated words, which limits the parallelization capability, and makes it much slower than RNN-based ones. In this paper, we systematically analyze the time cost of different components of both the Transformer and RNN-based model. Based on it, we propose a hybrid network of self-attention and RNN structures, in which, the highly parallelizable self-attention is utilized as the encoder, and the simpler RNN structure is used as the decoder. Our hybrid network can decode 4-times faster than the Transformer. In addition, with the help of knowledge distillation, our hybrid network achieves comparable translation quality to the original Transformer.

cs.CL↗

arXiv2019

Direct detection of the cosmic expansion: the redshift drift and the flux drift

The redshift drift, the change of cosmological redshift with time, is a direct consequence of the expansion of the Universe. Thus the measurement of the cosmological redshift drift will offer a direct test of our models of cosmology. The magnitude of the effect is very small, i.e. the spectral shift is of order of $10^{-10} - 10^{-9}$ over the period of a decade, but the next generation facilities such as ELT and SKA will be able to directly detect the expansion of our Universe by the year 2040. In this paper we focus on detectebility of this effect, including strategies of overcoming the kinematic contamination of the cosmological signal. We also show that the redshift drift directly impacts the change of flux. Thus apart from the redshift drift, measurements of the flux drift will provide an additional tool of detecting the expansion of the universe, including its acceleration. We discuss the strategies of detecting the flux drift and show that by including the flux drift signal to the redshift drift signal we boost the chances of a direct detection of the expansion of the Universe. We show that if only the stability of flux is at the level of $ΔF/F \sim 10^{-6}$ then the SKA1-mid Array should be able to detect these effects, before the ETL and the full SKA. Thus, by including the flux drift into the SKA1-mid Array's analysis pipeline, we could be able to provide by mid-2030s a direct evidence of the expansion of the universe including its accelerating phase.

astro-ph.CO↗