arXiv subjects

José A. R. Fonollosa

Publications and source records attributed to José A. R. Fonollosa.

26 records · Page 2Linked to original sources

arXiv2019

From Bilingual to Multilingual Neural Machine Translation by Incremental Training

Multilingual Neural Machine Translation approaches are based on the use of task-specific models and the addition of one more language can only be done by retraining the whole system. In this work, we propose a new training schedule that allows the system to scale to more languages without modification of the previous components based on joint training and language-independent encoder/decoder modules allowing for zero-shot translation. This work in progress shows close results to the state-of-the-art in the WMT task.

cs.CL↗

arXiv2019

Joint Source-Target Self Attention with Locality Constraints

The dominant neural machine translation models are based on the encoder-decoder structure, and many of them rely on an unconstrained receptive field over source and target sequences. In this paper we study a new architecture that breaks with both conventions. Our simplified architecture consists in the decoder part of a transformer model, based on self-attention, but with locality constraints applied on the attention receptive field. As input for training, both source and target sentences are fed to the network, which is trained as a language model. At inference time, the target tokens are predicted autoregressively starting with the source sequence as previous tokens. The proposed model achieves a new state of the art of 35.7 BLEU on IWSLT'14 German-English and matches the best reported results in the literature on the WMT'14 English-German and WMT'14 English-French translation benchmarks.

cs.CL↗

arXiv2018

(Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation

Universal language representation is the holy grail in machine translation (MT). Thanks to the new neural MT approach, it seems that there are good perspectives towards this goal. In this paper, we propose a new architecture based on combining variational autoencoders with encoder-decoders and introducing an interlingual loss as an additional training objective. By adding and forcing this interlingual loss, we are able to train multiple encoders and decoders for each language, sharing a common universal representation. Since the final objective of this universal representation is producing close results for similar input sentences (in any language), we propose to evaluate it by encoding the same sentence in two different languages, decoding both latent representations into the same language and comparing both outputs. Preliminary results on the WMT 2017 Turkish/English task shows that the proposed architecture is capable of learning a universal language representation and simultaneously training both translation directions with state-of-the-art results.

cs.CL↗

arXiv2018

SAT encodings for sorting networks, single-exception sorting networks and $ε-$halvers

Sorting networks are oblivious sorting algorithms with many practical applications and rich theoretical properties. Propositional encodings of sorting networks are a key tool for proving concrete bounds on the minimum number of comparators or depth (number of parallel steps) of sorting networks. In this paper, we present new SAT encodings that reduce the number of variables and clauses of the sorting constraint of optimality problems. Moreover, the proposed SAT encodings can be applied to a broader class of problems, such as the search of optimal single-exception sorting networks and $ε-$halvers. We obtain optimality results for single-exception sorting networks on $n \le 10$ inputs.

cs.DS↗

arXiv2018

Joint Size and Depth Optimization of Sorting Networks

Sorting networks are oblivious sorting algorithms with many interesting theoretical properties and practical applications. One of the related classical challenges is the search of optimal networks respect to size (number of comparators) of depth (number of layers). However, up to our knowledge, the joint size-depth optimality of small sorting networks has not been addressed before. This paper presents size-depth optimality results for networks up to $12$ channels. Our results show that there are sorting networks for $n\leq9$ inputs that are optimal in both size and depth, but this is not the case for $10$ and $12$ channels. For $n=10$ inputs, we were able to proof that optimal-depth optimal sorting networks with $7$ layers require $31$ comparators while optimal-size networks with $29$ comparators need $8$ layers. For $n=11$ inputs we show that networks with $8$ or $9$ layers require at least $35$ comparators (the best known upper bound for the minimal size). And for networks with $n=12$ inputs and $8$ layers we need $40$ comparators, while for $9$ layers the best known size is $39$.

cs.DS↗

arXiv2017

Character-level Intra Attention Network for Natural Language Inference

Natural language inference (NLI) is a central problem in language understanding. End-to-end artificial neural networks have reached state-of-the-art performance in NLI field recently. In this paper, we propose Character-level Intra Attention Network (CIAN) for the NLI task. In our model, we use the character-level convolutional network to replace the standard word embedding layer, and we use the intra attention to capture the intra-sentence semantics. The proposed CIAN model provides improved results based on a newly published MNLI corpus.

cs.CL↗

arXiv2016

Character-based Neural Machine Translation

Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional and highway layers to replace the standard lookup-based word representations. The resulting unlimited-vocabulary and affix-aware source word embeddings are tested in a state-of-the-art neural MT based on an attention-based bidirectional recurrent neural network. The proposed MT scheme provides improved results even when the source language is not morphologically rich. Improvements up to 3 BLEU points are obtained in the German-English WMT task.

cs.CL↗

arXiv2016

Conditional distribution variability measures for causality detection

In this paper we derive variability measures for the conditional probability distributions of a pair of random variables, and we study its application in the inference of causal-effect relationships. We also study the combination of the proposed measures with standard statistical measures in the the framework of the ChaLearn cause-effect pair challenge. The developed model obtains an AUC score of 0.82 on the final test database and ranked second in the challenge.

stat.ML↗