arXiv subjects

Kenneth Heafield

Publications and source records attributed to Kenneth Heafield.

24 records · Page 2Linked to original sources

arXiv2018

Marian: Cost-effective High-Quality Neural Machine Translation in C++

This paper describes the submissions of the "Marian" team to the WNMT 2018 shared task. We investigate combinations of teacher-student training, low-precision matrix products, auto-tuning and other methods to optimize the Transformer model on GPU and CPU. By further integrating these methods with the new averaging attention networks, a recently introduced faster Transformer variant, we create a number of high-quality, high-performance models on the GPU and CPU, dominating the Pareto frontier for this shared task.

cs.CL↗

arXiv2018

Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task

Previously, neural methods in grammatical error correction (GEC) did not reach state-of-the-art results compared to phrase-based statistical machine translation (SMT) baselines. We demonstrate parallels between neural GEC and low-resource neural MT and successfully adapt several methods from low-resource MT to neural GEC. We further establish guidelines for trustable results in neural GEC and propose a set of model-independent methods for neural GEC that can be easily applied in most GEC settings. Proposed methods include adding source-side noise, domain-adaptation techniques, a GEC-specific training-objective, transfer learning with monolingual data, and ensembling of independently trained GEC models and language models. The combined effects of these methods result in better than state-of-the-art neural GEC models that outperform previously best neural GEC systems by more than 10% M$^2$ on the CoNLL-2014 benchmark and 5.9% on the JFLEG test set. Non-neural state-of-the-art systems are outperformed by more than 2% on the CoNLL-2014 benchmark and by 4% on JFLEG.

cs.CL↗

arXiv2018

Marian: Fast Neural Machine Translation in C++

We present Marian, an efficient and self-contained Neural Machine Translation framework with an integrated automatic differentiation engine based on dynamic computation graphs. Marian is written entirely in C++. We describe the design of the encoder-decoder framework and demonstrate that a research-friendly toolkit can achieve high training and translation speed.

cs.CL↗

arXiv2017

The University of Edinburgh's Neural MT Systems for WMT17

This paper describes the University of Edinburgh's submissions to the WMT17 shared news translation and biomedical translation tasks. We participated in 12 translation directions for news, translating between English and Czech, German, Latvian, Russian, Turkish and Chinese. For the biomedical task we submitted systems for English to Czech, German, Polish and Romanian. Our systems are neural machine translation systems trained with Nematus, an attentional encoder-decoder. We follow our setup from last year and build BPE-based models with parallel and back-translated monolingual training data. Novelties this year include the use of deep architectures, layer normalization, and more compact models due to weight tying and improvements in BPE segmentations. We perform extensive ablative experiments, reporting on the effectivenes of layer normalization, deep architectures, and different ensembling techniques.

cs.CL↗

arXiv2017

Sparse Communication for Distributed Gradient Descent

We make distributed stochastic gradient descent faster by exchanging sparse updates instead of dense updates. Gradient updates are positively skewed as most updates are near zero, so we map the 99% smallest updates (by absolute value) to zero then exchange sparse matrices. This method can be combined with quantization to further improve the compression. We explore different configurations and apply them to neural machine translation and MNIST image classification tasks. Most configurations work on MNIST, whereas different configurations reduce convergence rate on the more complex translation task. Our experiments show that we can achieve up to 49% speed up on MNIST and 22% on NMT without damaging the final accuracy or BLEU.

cs.CL↗

arXiv2005

The Galex Ultraviolet Variability (GUVV) Catalog

We present Version 1.0 of the NASA Galaxy Evolution Explorer (GALEX) ultraviolet variability catalog (GUVV) that contains information on 84 time-variable and transient sources gained with simultaneous near and far ultraviolet photometric observations. These time-variable sources were serendipitously revealed in the various 1.2 degree star fields currently being surveyed by the GALEX satellite in two ultraviolet bands (NUV 1750-2750A, FUV 1350-1750A) with limiting AB magnitudes of 23-25. The largest-amplitude variable objects presently detected by GALEX are M-dwarf flare stars, which can brighten by 5-10 mag in both the NUV and FUV bands during short duration (< 500s) outbursts. Other types of large-amplitude ultraviolet variable objects include ab-type RR Lyrae stars, which can vary periodically by 2-5mag in the GALEX FUV band. This first GUVV catalog lists galactic positions and possible source identifications in order to provide the astronomical community with a list of time-variable objects that can now be repeatedly observed at other wavelengths. We expect the total number of time-variable source detections to increase as the GALEX mission progresses, such that later version numbers of the GUVV catalog will contain substantially more variable sources.

astro-ph↗