Searcharxiv⌕ Search

arXiv subjects

Dongdong Zhang

Publications and source records attributed to Dongdong Zhang.

At least 73 records · Page 4Linked to original sources

Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation

This paper demonstrates that multilingual pretraining and multilingual fine-tuning are both critical for facilitating cross-lingual transfer in zero-shot translation, where the neural machine translation (NMT) model is tested on source languages unseen during supervised training. Following this idea, we present SixT+, a strong many-to-English NMT model that supports 100 source languages but is trained with a parallel dataset in only six source languages. SixT+ initializes the decoder embedding and the full encoder with XLM-R large and then trains the encoder and decoder layers with a simple two-stage training strategy. SixT+ achieves impressive performance on many-to-English translation. It significantly outperforms CRISS and m2m-100, two strong multilingual NMT systems, with an average gain of 7.2 and 5.0 BLEU respectively. Additionally, SixT+ offers a set of model parameters that can be further fine-tuned to other unsupervised tasks. We demonstrate that adding SixT+ initialization outperforms state-of-the-art explicitly designed unsupervised NMT models on Si<->En and Ne<->En by over 1.2 average BLEU. When applied to zero-shot cross-lingual abstractive summarization, it produces an average performance gain of 12.3 ROUGE-L over mBART-ft. We conduct detailed analyses to understand the key ingredients of SixT+, including multilinguality of the auxiliary parallel data, positional disentangled encoder, and the cross-lingual transferability of its encoder.

cs.CL↗

DeepNet: Scaling Transformers to 1,000 Layers

In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the residual connection in Transformer, accompanying with theoretically derived initialization. In-depth theoretical analysis shows that model updates can be bounded in a stable way. The proposed method combines the best of two worlds, i.e., good performance of Post-LN and stable training of Pre-LN, making DeepNorm a preferred alternative. We successfully scale Transformers up to 1,000 layers (i.e., 2,500 attention and feed-forward network sublayers) without difficulty, which is one order of magnitude deeper than previous deep Transformers. Remarkably, on a multilingual benchmark with 7,482 translation directions, our 200-layer model with 3.2B parameters significantly outperforms the 48-layer state-of-the-art model with 12B parameters by 5 BLEU points, which indicates a promising scaling direction.

cs.CL↗

SMDT: Selective Memory-Augmented Neural Document Translation

Existing document-level neural machine translation (NMT) models have sufficiently explored different context settings to provide guidance for target generation. However, little attention is paid to inaugurate more diverse context for abundant context information. In this paper, we propose a Selective Memory-augmented Neural Document Translation model to deal with documents containing large hypothesis space of the context. Specifically, we retrieve similar bilingual sentence pairs from the training corpus to augment global context and then extend the two-stream attention model with selective mechanism to capture local context and diverse global contexts. This unified approach allows our model to be trained elegantly on three publicly document-level machine translation datasets and significantly outperforms previous document-level NMT models.

cs.CL↗

New source for tuning the effective Rabi frequency discovered in multiphoton ionization

The Autler-Townes effect due to near resonance transition between 4s-4p states in potassium atoms is mapped out in the photo-electron-momentum distribution and manifests itself as a splitting in the photo-electron kinetic energy spectra. The energy splitting fits well with the calculated Rabi frequency at low laser intensities and shows clear deviation at laser intensities above 1.5x10^11 W/cm^2. An effective Rabi frequency formulae including the ionization process explains the observed results. Our results reveal the possibility to tune the effective coupling strength with the cost of the number of level-populations.

physics.atom-ph↗

Future prospects on constraining neutrino cosmology with the Ali CMB Polarization Telescope

We forecast the constraints on the parameters of neutrino physics with the constructions of Cosmic Microwave Background (CMB) temperature, E-mode polarization and lensing spectra for the ground-based Ali Cosmic Microwave Background Polarization Telescope (AliCPT). To implement the forecast calculations, we perform numerical simulations which show that AliCPT would yield the result $σ(N_{\mathrm{eff}})=0.42$ and $σ(M_ν)=0.18$ for the first year observation. Moreover, we investigate how the instrumental parameters, such as noise level, full width at half maxima (FWHM), and sky coverage can affect the constraints on these two parameters. Our forecasting results find that a large aperture telescope with a large sky survey strategy would significant improve the current constraints.

astro-ph.CO↗

Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders

Previous work mainly focuses on improving cross-lingual transfer for NLU tasks with a multilingual pretrained encoder (MPE), or improving the performance on supervised machine translation with BERT. However, it is under-explored that whether the MPE can help to facilitate the cross-lingual transferability of NMT model. In this paper, we focus on a zero-shot cross-lingual transfer task in NMT. In this task, the NMT model is trained with parallel dataset of only one language pair and an off-the-shelf MPE, then it is directly tested on zero-shot language pairs. We propose SixT, a simple yet effective model for this task. SixT leverages the MPE with a two-stage training schedule and gets further improvement with a position disentangled encoder and a capacity-enhanced decoder. Using this method, SixT significantly outperforms mBART, a pretrained multilingual encoder-decoder model explicitly designed for NMT, with an average improvement of 7.1 BLEU on zero-shot any-to-English test sets across 14 source languages. Furthermore, with much less training computation cost and training data, our model achieves better performance on 15 any-to-English test sets than CRISS and m2m-100, two strong multilingual NMT baselines.

cs.CL↗

Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task

This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. We participated in all three evaluation tracks including Large Track and two Small Tracks where the former one is unconstrained and the latter two are fully constrained. Our model submissions to the shared task were initialized with DeltaLM\footnote{\url{https://aka.ms/deltalm}}, a generic pre-trained multilingual encoder-decoder model, and fine-tuned correspondingly with the vast collected parallel data and allowed data sources according to track settings, together with applying progressive learning and iterative back-translation approaches to further improve the performance. Our final submissions ranked first on three tracks in terms of the automatic evaluation metric.

cs.CL↗

On the existence of Auslander-Reiten $n$-exangles in $n$-exangulated categories

Let $\mathscr{C}$ be an $n$-exangulated category. In this note, we show that if $\mathscr{C}$ is locally finite, then $\mathscr{C}$ has Auslander-Reiten $n$-exangles. This unifies and extends results of Xiao-Zhu, Zhu-Zhuang, Zhou and Xie-Lu-Wang for triangulated, extriangulated, $(n+2)$-angulated and $n$-abelian categories, respectively.

math.RT↗

Balanced pairs on triangulated categories

Let $\mathcal{C}$ be a triangulated category. We first introduce the notion of balanced pairs in $\mathcal{C}$, and then establish the bijective correspondence between balanced pairs and proper classes $ξ$ with enough $ξ$-projectives and enough $ξ$-injectives. Assume that $ξ:=ξ_{\mathcal{X}}=ξ^{\mathcal{Y}}$ is the proper class induced by a balanced pair $(\mathcal{X},\mathcal{Y})$. We prove that $(\mathcal{C}, \mathbb{E}_ξ, \mathfrak{s}_ξ)$ is an extriangulated category. Moreover, it is proved that $(\mathcal{C}, \mathbb{E}_ξ, \mathfrak{s}_ξ)$ is a triangulated category if and only if $\mathcal{X}=\mathcal{Y}=0$; and that $(\mathcal{C}, \mathbb{E}_ξ, \mathfrak{s}_ξ)$ is an exact category if and only if $\mathcal{X}=\mathcal{Y}=\mathcal{C}$. As an application, we produce a large variety of examples of extriangulated categories which are neither exact nor triangulated.

math.RA↗

Higher Auslander's defect and classifying substructures of n-exangulated categories

Herschend-Liu-Nakaoka introduced the notion of $n$-exangulated categories. It is not only a higher dimensional analogue of extriangulated categories defined by Nakaoka-Palu, but also gives a simultaneous generalization of $n$-exact categories and $(n+2)$-angulated categories. In this article, we give an $n$-exangulated version of Auslander's defect and Auslander-Reiten duality formula. Moreover, we also give a classification of substructures (=closed subbifunctors) of a given skeletally small $n$-exangulated category by using the category of defects.

math.RT↗

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG). NLG tasks are often based on the encoder-decoder framework, where the pretrained encoders can only benefit part of it. To reduce this gap, we introduce DeltaLM, a pretrained multilingual encoder-decoder model that regards the decoder as the task layer of off-the-shelf pretrained encoders. Specifically, we augment the pretrained multilingual encoder with a decoder and pre-train it in a self-supervised way. To take advantage of both the large-scale monolingual data and bilingual data, we adopt the span corruption and translation span corruption as the pre-training tasks. Experiments show that DeltaLM outperforms various strong baselines on both natural language generation and translation tasks, including machine translation, abstractive text summarization, data-to-text, and question generation. The code and pretrained models are available at \url{https://aka.ms/deltalm}.

cs.CL↗

A New Design for a Traveling-Wave Zeeman Decelerator: I. Theory

The concept of a novel traveling wave Zeeman deccelerator based on a double-helix wire geometry capable of decelerating paramagnetic species with high efficiency is presented. A moving magnetic trap is created by running time-dependent currents through the decelerator coils. Paramagnetic species in low-field-seeking Zeeman states are confined in the moving traps and transported to the end of the decelerator with programmable velocities. Here, we present the theoretical foundations underlying the working principle of the traveling-trap decelerator. Using trajectory simulations, we characterise the performance of the new device and explore the conditions for phase-space stability of the transported molecules.

physics.chem-ph↗

A New Design for a Traveling-Wave Zeeman Decelerator: II. Experiment

A novel traveling-wave Zeeman decelerator based on a double-helix coil geometry capable of decelerating paramagnetic molecules with high efficiency is presented. Moving magnetic traps are generated by applying time-dependent currents through the decelerator coils. Paramagnetic molecules in low-field-seeking Zeeman states are confined inside the moving traps which are decelerated to lower forward velocities. As a prototypical example, we demonstrate the deceleration of OH radicals from an initial velocity of 445 m/s down to various final velocities. The experimental results are analyzed and numerically reproduced with the help of trajectory simulations confirming the phase-space stability and efficiency of the deceleration of the molecules in the new device.

physics.chem-ph↗

How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?

While non-autoregressive (NAR) models are showing great promise for machine translation, their use is limited by their dependence on knowledge distillation from autoregressive models. To address this issue, we seek to understand why distillation is so effective. Prior work suggests that distilled training data is less complex than manual translations. Based on experiments with the Levenshtein Transformer and the Mask-Predict NAR models on the WMT14 German-English task, this paper shows that different types of complexity have different impacts: while reducing lexical diversity and decreasing reordering complexity both help NAR learn better alignment between source and target, and thus improve translation quality, lexical diversity is the main reason why distillation increases model confidence, which affects the calibration of different NAR models differently.

cs.CL↗

The Impact of COVID-19 on Urban Energy Consumption of the Commercial Tourism City

In 2020, the COVID-19 pandemic spreads all over the world. In order to alleviate the spread of the epidemic, various blockade policies have been implemented in many areas. In order to formulate a better epidemic prevention policy for urban energy consumption of the commercial tourism cities, this paper first analyses the energy characteristics of Macao during the epidemic period from two aspects, based on the energy consumption data of Macao. On this basis, the power consumption characteristics of commercial tourism cities during the epidemic were analyzed. Then, this paper provides analysis of the characteristics of the energy consumption in different fields of commercial tourism cities from the aspects of hotel, transportation, tourism culture and public utilities. Finally, a detailed analysis of the energy consumption characteristics of commercial tourism cities represented by Macao during the epidemic period is provided, by comparing with some typical countries.

physics.soc-ph↗

M3P: Learning Universal Representations via Multitask Multilingual Multimodal Pre-training

We present M3P, a Multitask Multilingual Multimodal Pre-trained model that combines multilingual pre-training and multimodal pre-training into a unified framework via multitask pre-training. Our goal is to learn universal representations that can map objects occurred in different modalities or texts expressed in different languages into a common semantic space. In addition, to explicitly encourage fine-grained alignment between images and non-English languages, we also propose Multimodal Code-switched Training (MCT) to combine monolingual pre-training and multimodal pre-training via a code-switch strategy. Experiments are performed on the multilingual image retrieval task across two benchmark datasets, including MSCOCO and Multi30K. M3P can achieve comparable results for English and new state-of-the-art results for non-English languages.

cs.CL↗

XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders

Multilingual machine translation enables a single model to translate between different languages. Most existing multilingual machine translation systems adopt a randomly initialized Transformer backbone. In this work, inspired by the recent success of language model pre-training, we present XLM-T, which initializes the model with an off-the-shelf pretrained cross-lingual Transformer encoder and fine-tunes it with multilingual parallel data. This simple method achieves significant improvements on a WMT dataset with 10 language pairs and the OPUS-100 corpus with 94 pairs. Surprisingly, the method is also effective even upon the strong baseline with back-translation. Moreover, extensive analysis of XLM-T on unsupervised syntactic parsing, word alignment, and multilingual classification explains its effectiveness for machine translation. The code will be at https://aka.ms/xlm-t.

cs.CL↗

A proximal dual semismooth Newton method for computing zero-norm penalized QR estimator

This paper is concerned with the computation of the high-dimensional zero-norm penalized quantile regression estimator, defined as a global minimizer of the zero-norm penalized check loss function. To seek a desirable approximation to the estimator, we reformulate this NP-hard problem as an equivalent augmented Lipschitz optimization problem, and exploit its coupled structure to propose a multi-stage convex relaxation approach (MSCRA\_PPA), each step of which solves inexactly a weighted $\ell_1$-regularized check loss minimization problem with a proximal dual semismooth Newton method. Under a restricted strong convexity condition, we provide the theoretical guarantee for the MSCRA\_PPA by establishing the error bound of each iterate to the true estimator and the rate of linear convergence in a statistical sense. Numerical comparisons on some synthetic and real data show that MSCRA\_PPA not only has comparable even better estimation performance, but also requires much less CPU time.

math.OC↗