arXiv subjects

Haoran Xu

Publications and source records attributed to Haoran Xu.

133 records · Page 8Linked to original sources

arXiv2021

BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation

The success of bidirectional encoders using masked language models, such as BERT, on numerous natural language processing tasks has prompted researchers to attempt to incorporate these pre-trained models into neural machine translation (NMT) systems. However, proposed methods for incorporating pre-trained models are non-trivial and mainly focus on BERT, which lacks a comparison of the impact that other pre-trained models may have on translation performance. In this paper, we demonstrate that simply using the output (contextualized embeddings) of a tailored and suitable bilingual pre-trained language model (dubbed BiBERT) as the input of the NMT encoder achieves state-of-the-art translation performance. Moreover, we also propose a stochastic layer selection approach and a concept of dual-directional translation model to ensure the sufficient utilization of contextualized embeddings. In the case of without using back translation, our best models achieve BLEU scores of 30.45 for En->De and 38.61 for De->En on the IWSLT'14 dataset, and 31.26 for En->De and 34.94 for De->En on the WMT'14 dataset, which exceeds all published numbers.

cs.CL↗

arXiv2021

Gradual Fine-Tuning for Low-Resource Domain Adaptation

Fine-tuning is known to improve NLP models by adapting an initial model trained on more plentiful but less domain-salient examples to data in a target domain. Such domain adaptation is typically done using one stage of fine-tuning. We demonstrate that gradually fine-tuning in a multi-stage process can yield substantial further gains and can be applied without modifying the model or learning objective.

cs.CL↗

arXiv2021

Cross-Lingual BERT Contextual Embedding Space Mapping with Isotropic and Isometric Conditions

Typically, a linearly orthogonal transformation mapping is learned by aligning static type-level embeddings to build a shared semantic space. In view of the analysis that contextual embeddings contain richer semantic features, we investigate a context-aware and dictionary-free mapping approach by leveraging parallel corpora. We illustrate that our contextual embedding space mapping significantly outperforms previous multilingual word embedding methods on the bilingual dictionary induction (BDI) task by providing a higher degree of isomorphism. To improve the quality of mapping, we also explore sense-level embeddings that are split from type-level representations, which can align spaces in a finer resolution and yield more precise mapping. Moreover, we reveal that contextual embedding spaces suffer from their natural properties -- anisotropy and anisometry. To mitigate these two problems, we introduce the iterative normalization algorithm as an imperative preprocessing step. Our findings unfold the tight relationship between isotropy, isometry, and isomorphism in normalized contextual embedding spaces.

cs.CL↗

arXiv2021

Zero-Shot Cross-Lingual Dependency Parsing through Contextual Embedding Transformation

Linear embedding transformation has been shown to be effective for zero-shot cross-lingual transfer tasks and achieve surprisingly promising results. However, cross-lingual embedding space mapping is usually studied in static word-level embeddings, where a space transformation is derived by aligning representations of translation pairs that are referred from dictionaries. We move further from this line and investigate a contextual embedding alignment approach which is sense-level and dictionary-free. To enhance the quality of the mapping, we also provide a deep view of properties of contextual embeddings, i.e., anisotropy problem and its solution. Experiments on zero-shot dependency parsing through the concept-shared space built by our embedding transformation substantially outperform state-of-the-art methods using multilingual embeddings.

cs.CL↗

arXiv2019

Self-organization of swimmers drives long-range fluid transport in bacterial colonies

Bacteria commonly live in structured communities that affect human health and influence ecological systems. Heterogeneous populations, such as motile and non-motile populations, often coexist in bacteria communities. Motile subpopulations in microbial communities are believed to be important to dispersal, quest for food, and material transport. However, except in circumstances where motile cells drive colony expansion (e.g. bacterial swarming), the physiological functions of motile subpopulations in bacterial communities are largely unclear. Here we discovered that motile cells in routinely cultured sessile colonies of peritrichously flagellated bacteria can self-organize into two adjacent, centimeter-scale motile rings surrounding the entire colony. The motile rings arise from spontaneous segregation of a homogeneous swimmer suspension that mimics a phase separation; the process is mediated by intercellular interactions and shear-induced depletion. As a result of this self-organization, cells drive fluid flows to circulate around the colony at a constant peak speed of approximately 30 microns per second, providing a stable and high-speed avenue for directed material transport at the macroscopic scale. These findings present a unique form of bacterial self-organization that influences population structure and material distribution in bacterial communities.

physics.bio-ph↗

arXiv2018

Cimple: Instruction and Memory Level Parallelism

Modern out-of-order processors have increased capacity to exploit instruction level parallelism (ILP) and memory level parallelism (MLP), e.g., by using wide superscalar pipelines and vector execution units, as well as deep buffers for in-flight memory requests. These resources, however, often exhibit poor utilization rates on workloads with large working sets, e.g., in-memory databases, key-value stores, and graph analytics, as compilers and hardware struggle to expose ILP and MLP from the instruction stream automatically. In this paper, we introduce the IMLP (Instruction and Memory Level Parallelism) task programming model. IMLP tasks execute as coroutines that yield execution at annotated long-latency operations, e.g., memory accesses, divisions, or unpredictable branches. IMLP tasks are interleaved on a single thread, and integrate well with thread parallelism and vectorization. Our DSL embedded in C++, Cimple, allows exploration of task scheduling and transformations, such as buffering, vectorization, pipelining, and prefetching. We demonstrate state-of-the-art performance on core algorithms used in in-memory databases that operate on arrays, hash tables, trees, and skip lists. Cimple applications reach 2.5x throughput gains over hardware multithreading on a multi-core, and 6.4x single thread speedup.

cs.PL↗

arXiv2014

Electric-Field-Modulated Nonvolatile Resistance Switching in VO2/PMN-PT(111) Heterostructures

The electric-field-modulated resistance switching in VO2 thin films grown on piezoelectric (111)-0.68Pb(Mg1/3Nb2/3)O3-0.32PbTiO3 (PMN-PT) substrates has been investigated. Large relative change in resistance (10.7%) was observed in VO2/PMN-PT(111) hererostructures at room temperature. For a substrate with a given polarization direction, stable resistive states of VO2 films can be realized even when the applied electric fields are removed from the heterostructures. By sweeping electric fields across the heterostructure appropriately, multiple resistive states can be achieved. These stable resistive states result from the different stable remnant strain states of substrate, which is related to the rearrangements of ferroelectric domain structures in PMN-PT(111) substrate. The resistance switching tuned by electric field in our work may have potential applications for novel electronic devices.

cond-mat.str-el↗