arXiv subjects

Ivan Titov

Publications and source records attributed to Ivan Titov.

At least 127 records · Page 7Linked to original sources

arXiv2019

Boosting Entity Linking Performance by Leveraging Unlabeled Documents

Modern entity linking systems rely on large collections of documents specifically annotated for the task (e.g., AIDA CoNLL). In contrast, we propose an approach which exploits only naturally occurring information: unlabeled documents and Wikipedia. Our approach consists of two stages. First, we construct a high recall list of candidate entities for each mention in an unlabeled document. Second, we use the candidate lists as weak supervision to constrain our document-level entity linking model. The model treats entities as latent variables and, when estimated on a collection of unlabelled texts, learns to choose entities relying both on local context of each mention and on coherence with other entities in the document. The resulting approach rivals fully-supervised state-of-the-art systems on standard test sets. It also approaches their performance in the very challenging setting: when tested on a test set sampled from the data used to estimate the supervised systems. By comparing to Wikipedia-only training of our model, we demonstrate that modeling unlabeled documents is beneficial.

cs.CL↗

arXiv2019

Block Neural Autoregressive Flow

Normalising flows (NFS) map two density functions via a differentiable bijection whose Jacobian determinant can be computed efficiently. Recently, as an alternative to hand-crafted bijections, Huang et al. (2018) proposed neural autoregressive flow (NAF) which is a universal approximator for density functions. Their flow is a neural network (NN) whose parameters are predicted by another NN. The latter grows quadratically with the size of the former and thus an efficient technique for parametrization is needed. We propose block neural autoregressive flow (B-NAF), a much more compact universal approximator of density functions, where we model a bijection directly using a single feed-forward network. Invertibility is ensured by carefully designing each affine transformation with block matrices that make the flow autoregressive and (strictly) monotone. We compare B-NAF to NAF and other established flows on density estimation and approximate inference for latent variable models. Our proposed flow is competitive across datasets while using orders of magnitude fewer parameters.

stat.ML↗

arXiv2019

Evidence for the formation of nanoprecipitates with magnetically disordered regions in bulk $\mathrm{Ni}_{50}\mathrm{Mn}_{45}\mathrm{In}_{5}$ Heusler alloys

Shell ferromagnetism is a new functional property of certain Heusler alloys which has been recently observed in $\mathrm{Ni}_{50}\mathrm{Mn}_{45}\mathrm{In}_{5}$. We report the results of a comparative study of the magnetic microstructure of bulk $\mathrm{Ni}_{50}\mathrm{Mn}_{45}\mathrm{In}_{5}$ Heusler alloys using magnetometry, synchrotron x-ray diffraction, and magnetic small-angle neutron scattering (SANS). By combining unpolarized and spin-polarized SANS (POLARIS) we demonstrate that a number of important conclusions regarding the mesoscopic spin structure can be made. In particular, the analysis of the magnetic neutron data suggests that nanoprecipitates with an effective ferromagnetic component form in an antiferromagnetic matrix on field annealing at $700 \, \mathrm{K}$. These particles represent sources of perturbation, which seem to give rise to magnetically disordered regions in the vicinity of the particle-matrix interface. Analysis of the spin-flip SANS cross section via the computation of the correlation function yields a value of $\sim 55 \, \mathrm{nm}$ for the particle size and $\sim 20 \, \mathrm{nm}$ for the size of the spin-canted region.

cond-mat.mes-hall↗

arXiv2019

Differentiable Perturb-and-Parse: Semi-Supervised Parsing with a Structured Variational Autoencoder

Human annotation for syntactic parsing is expensive, and large resources are available only for a fraction of languages. A question we ask is whether one can leverage abundant unlabeled texts to improve syntactic parsers, beyond just using the texts to obtain more generalisable lexical features (i.e. beyond word embeddings). To this end, we propose a novel latent-variable generative model for semi-supervised syntactic dependency parsing. As exact inference is intractable, we introduce a differentiable relaxation to obtain approximate samples and compute gradients with respect to the parser parameters. Our method (Differentiable Perturb-and-Parse) relies on differentiable dynamic programming over stochastically perturbed edge scores. We demonstrate effectiveness of our approach with experiments on English, French and Swedish.

cs.CL↗

arXiv2018

Microstructural-defect-induced Dzyaloshinskii-Moriya interaction

The antisymmetric Dzyaloshinskii-Moriya interaction (DMI) plays a decisive role for the stabilization and control of chirality of skyrmion textures in various magnetic systems exhibiting a noncentrosymmetric crystal structure. A less studied aspect of the DMI is that this interaction is believed to be operative in the vicinity of lattice imperfections in crystalline magnetic materials, due to the local structural inversion symmetry breaking. If this scenario leads to an effect of sizable magnitude, it implies that the DMI introduces chirality into a very large class of magnetic materials---defect-rich systems such as polycrystalline magnets. Here, we show experimentally that the microstructural-defect-induced DMI gives rise to a polarization-dependent asymmetric term in the small-angle neutron scattering (SANS) cross section of polycrystalline ferromagnets with a centrosymmetric crystal structure. The results are supported by theoretical predictions using the continuum theory of micromagnetics. This effect, conjectured already by Arrott in 1963, is demonstrated for nanocrystalline terbium and holmium (with a large grain-boundary density), and for mechanically-deformed microcrystalline cobalt (with a large dislocation density). Analysis of the scattering asymmetry allows one to determine the defect-induced DMI constant, $D = 0.45 \pm 0.07 \, \mathrm{mJ/m^2}$ for Tb at $100 \, \mathrm{K}$. Our study proves the generic relevance of the DMI for the magnetic microstructure of defect-rich ferromagnets with vanishing intrinsic DMI. Polarized SANS is decisive for disclosing the signature of the defect-induced DMI, which is related to the unique dependence of the polarized SANS cross section on the chiral interactions. The findings open up the way to study defect-induced skyrmionic magnetization textures in disordered materials.

cond-mat.mtrl-sci↗

arXiv2018

Embedding Words as Distributions with a Bayesian Skip-gram Model

We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods, in our Bayesian model we generate it from a word-specific prior density for each occurrence of a given word. Intuitively, for each word, the prior density encodes the distribution of its potential 'meanings'. These prior densities are conceptually similar to Gaussian embeddings. Interestingly, unlike the Gaussian embeddings, we can also obtain context-specific densities: they encode uncertainty about the sense of a word given its context and correspond to posterior distributions within our model. The context-dependent densities have many potential applications: for example, we show that they can be directly used in the lexical substitution task. We describe an effective estimation method based on the variational autoencoding framework. We also demonstrate that our embeddings achieve competitive results on standard benchmarks.

cs.CL↗

arXiv2018

Context-Aware Neural Machine Translation Learns Anaphora Resolution

Standard machine translation systems process sentences in isolation and hence ignore extra-sentential information, even though extended context can both prevent mistakes in ambiguous cases and improve translation coherence. We introduce a context-aware neural machine translation model designed in such way that the flow of information from the extended context to the translation model can be controlled and analyzed. We experiment with an English-Russian subtitles dataset, and observe that much of what is captured by our model deals with improving pronoun translation. We measure correspondences between induced attention distributions and coreference relations and observe that the model implicitly captures anaphora. It is consistent with gains for sentences where pronouns need to be gendered in translation. Beside improvements in anaphoric cases, the model also improves in overall BLEU, both over its context-agnostic version (+0.7) and over simple concatenation of the context and source sentences (+0.6).

cs.CL↗

arXiv2018

AMR Parsing as Graph Prediction with Latent Alignment

Abstract meaning representations (AMRs) are broad-coverage sentence-level semantic representations. AMRs represent sentences as rooted labeled directed acyclic graphs. AMR parsing is challenging partly due to the lack of annotated alignments between nodes in the graphs and words in the corresponding sentences. We introduce a neural parser which treats alignments as latent variables within a joint probabilistic model of concepts, relations and alignments. As exact inference requires marginalizing over alignments and is infeasible, we use the variational auto-encoding framework and a continuous relaxation of the discrete alignments. We show that joint modeling is preferable to using a pipeline of align and parse. The parser achieves the best reported results on the standard benchmark (74.4% on LDC2016E25).

cs.CL↗

arXiv2018

Improving Entity Linking by Modeling Latent Relations between Mentions

Entity linking involves aligning textual mentions of named entities to their corresponding entries in a knowledge base. Entity linking systems often exploit relations between textual mentions in a document (e.g., coreference) to decide if the linking decisions are compatible. Unlike previous approaches, which relied on supervised systems or heuristics to predict these relations, we treat relations as latent variables in our neural entity-linking model. We induce the relations without any supervision while optimizing the entity-linking system in an end-to-end fashion. Our multi-relational model achieves the best reported scores on the standard benchmark (AIDA-CoNLL) and substantially outperforms its relation-agnostic version. Its training also converges much faster, suggesting that the injected structural bias helps to explain regularities in the training data.

cs.CL↗

arXiv2017

Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols

Learning to communicate through interaction, rather than relying on explicit supervision, is often considered a prerequisite for developing a general AI. We study a setting where two agents engage in playing a referential game and, from scratch, develop a communication protocol necessary to succeed in this game. Unlike previous work, we require that messages they exchange, both at train and test time, are in the form of a language (i.e. sequences of discrete symbols). We compare a reinforcement learning approach and one using a differentiable relaxation (straight-through Gumbel-softmax estimator) and observe that the latter is much faster to converge and it results in more effective protocols. Interestingly, we also observe that the protocol we induce by optimizing the communication success exhibits a degree of compositionality and variability (i.e. the same information can be phrased in different ways), both properties characteristic of natural languages. As the ultimate goal is to ensure that communication is accomplished in natural language, we also perform experiments where we inject prior information about natural language into our model and study properties of the resulting protocol.

cs.LG↗

arXiv2017

Modeling Relational Data with Graph Convolutional Networks

Knowledge graphs enable a wide variety of applications, including question answering and information retrieval. Despite the great effort invested in their creation and maintenance, even the largest (e.g., Yago, DBPedia or Wikidata) remain incomplete. We introduce Relational Graph Convolutional Networks (R-GCNs) and apply them to two standard knowledge base completion tasks: Link prediction (recovery of missing facts, i.e. subject-predicate-object triples) and entity classification (recovery of missing entity attributes). R-GCNs are related to a recent class of neural networks operating on graphs, and are developed specifically to deal with the highly multi-relational data characteristic of realistic knowledge bases. We demonstrate the effectiveness of R-GCNs as a stand-alone model for entity classification. We further show that factorization models for link prediction such as DistMult can be significantly improved by enriching them with an encoder model to accumulate evidence over multiple inference steps in the relational graph, demonstrating a large improvement of 29.8% on FB15k-237 over a decoder-only baseline.

stat.ML↗

arXiv2017

Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard NLP pipeline. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of neural networks operating on graphs, suited to model syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark (CoNLL-2009) both for Chinese and English.

cs.CL↗

arXiv2017

Optimizing Differentiable Relaxations of Coreference Evaluation Metrics

Coreference evaluation metrics are hard to optimize directly as they are non-differentiable functions, not easily decomposable into elementary decisions. Consequently, most approaches optimize objectives only indirectly related to the end goal, resulting in suboptimal performance. Instead, we propose a differentiable relaxation that lends itself to gradient-based optimisation, thus bypassing the need for reinforcement learning or heuristic modification of cross-entropy. We show that by modifying the training objective of a competitive neural coreference system, we obtain a substantial gain in performance. This suggests that our approach can be regarded as a viable alternative to using reinforcement learning or more computationally expensive imitation learning.

cs.CL↗

arXiv2017

A Simple and Accurate Syntax-Agnostic Neural Model for Dependency-based Semantic Role Labeling

We introduce a simple and accurate neural model for dependency-based semantic role labeling. Our model predicts predicate-argument dependencies relying on states of a bidirectional LSTM encoder. The semantic role labeler achieves competitive performance on English, even without any kind of syntactic information and only using local inference. However, when automatically predicted part-of-speech tags are provided as input, it substantially outperforms all previous local models and approaches the best reported results on the English CoNLL-2009 dataset. We also consider Chinese, Czech and Spanish where our approach also achieves competitive results. Syntactic parsers are unreliable on out-of-domain data, so standard (i.e., syntactically-informed) SRL models are hindered when tested in this setting. Our syntax-agnostic model appears more robust, resulting in the best reported results on standard out-of-domain test sets.

cs.CL↗

arXiv2017

Modeling Semantic Expectation: Using Script Knowledge for Referent Prediction

Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content. Prediction also affects perception and might be a key to robustness in human language processing. In this paper, we investigate the factors that affect human prediction by building a computational model that can predict upcoming discourse referents based on linguistic knowledge alone vs. linguistic knowledge jointly with common-sense knowledge in the form of scripts. We find that script knowledge significantly improves model estimates of human predictions. In a second study, we test the highly controversial hypothesis that predictability influences referring expression type but do not find evidence for such an effect.

cs.CL↗

arXiv2016

Spin structures of textured and isotropic Nd-Fe-B-based nanocomposites: Evidence for correlated crystallographic and spin texture

We report the results of a comparative study of the magnetic microstructure of textured and isotropic $\mathrm{Nd}_2\mathrm{Fe}_{14}\mathrm{B}/α$-$\mathrm{Fe}$ nanocomposites using magnetometry, transmission electron microscopy, synchrotron x-ray diffraction, and, in particular, magnetic small-angle neutron scattering (SANS). Analysis of the magnetic neutron data of the textured specimen and computation of the correlation function of the spin misalignment SANS cross section suggests the existence of inhomogeneously magnetized regions on an intraparticle nanometer length scale, about $40-50 \, \mathrm{nm}$ in the remanent state. Possible origins for this spin disorder are discussed: it may originate in thin grain-boundary layers (where the materials parameters are different than in the $\mathrm{Nd}_2\mathrm{Fe}_{14}\mathrm{B}$ grains), or it may reflect the presence of crystal defects (introduced via hot pressing), or the dispersion in the orientation distribution of the magnetocrystalline anisotropy axes of the $\mathrm{Nd}_2\mathrm{Fe}_{14}\mathrm{B}$ grains. X-ray powder diffraction data reveal a crystallographic texture in the direction perpendicular to the pressing direction -- a finding which might be related to the presence of a texture in the magnetization distribution, as inferred from the magnetic SANS data.

cond-mat.mes-hall↗

arXiv2016

Bilingual Learning of Multi-sense Embeddings with Discrete Autoencoders

We present an approach to learning multi-sense word embeddings relying both on monolingual and bilingual information. Our model consists of an encoder, which uses monolingual and bilingual context (i.e. a parallel sentence) to choose a sense for a given word, and a decoder which predicts context words based on the chosen sense. The two components are estimated jointly. We observe that the word representations induced from bilingual data outperform the monolingual counterparts across a range of evaluation tasks, even though crosslingual information is not available at test time.

cs.CL↗

arXiv2016

Word Representations, Tree Models and Syntactic Functions

Word representations induced from models with discrete latent variables (e.g.\ HMMs) have been shown to be beneficial in many NLP applications. In this work, we exploit labeled syntactic dependency trees and formalize the induction problem as unsupervised learning of tree-structured hidden Markov models. Syntactic functions are used as additional observed variables in the model, influencing both transition and emission components. Such syntactic information can potentially lead to capturing more fine-grain and functional distinctions between words, which, in turn, may be desirable in many NLP applications. We evaluate the word representations on two tasks -- named entity recognition and semantic frame identification. We observe improvements from exploiting syntactic function information in both cases, and the results rivaling those of state-of-the-art representation learning methods. Additionally, we revisit the relationship between sequential and unlabeled-tree models and find that the advantage of the latter is not self-evident.

cs.CL↗