arXiv subjects

Xiaodong He

Publications and source records attributed to Xiaodong He.

180 records · Page 10Linked to original sources

arXiv2016

Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

This paper develops a model that addresses sentence embedding, a hot topic in current natural language processing research, using recurrent neural networks with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long term memory, the LSTM-RNN accumulates increasingly richer information as it goes through the sentence, and when it reaches the last word, the hidden layer of the network provides a semantic representation of the whole sentence. In this paper, the LSTM-RNN is trained in a weakly supervised manner on user click-through data logged by a commercial web search engine. Visualization and analysis are performed to understand how the embedding process works. The model is found to automatically attenuate the unimportant words and detects the salient keywords in the sentence. Furthermore, these detected keywords are found to automatically activate different cells of the LSTM-RNN, where words belonging to a similar topic activate the same cell. As a semantic representation of the sentence, the embedding vector can be used in many different applications. These automatic keyword detection and topic allocation abilities enabled by the LSTM-RNN allow the network to perform document retrieval, a difficult language processing task, where the similarity between the query and documents can be measured by the distance between their corresponding sentence embedding vectors computed by the LSTM-RNN. On a web search task, the LSTM-RNN embedding is shown to significantly outperform several existing state of the art methods. We emphasize that the proposed model generates sentence embedding vectors that are specially useful for web document retrieval tasks. A comparison with a well known general sentence embedding method, the Paragraph Vector, is performed. The results show that the proposed method in this paper significantly outperforms it for web document retrieval task.

cs.CL↗

arXiv2016

Basic Reasoning with Tensor Product Representations

In this paper we present the initial development of a general theory for mapping inference in predicate logic to computation over Tensor Product Representations (TPRs; Smolensky (1990), Smolensky & Legendre (2006)). After an initial brief synopsis of TPRs (Section 0), we begin with particular examples of inference with TPRs in the 'bAbI' question-answering task of Weston et al. (2015) (Section 1). We then present a simplification of the general analysis that suffices for the bAbI task (Section 2). Finally, we lay out the general treatment of inference over TPRs (Section 3). We also show the simplification in Section 2 derives the inference methods described in Lee et al. (2016); this shows how the simple methods of Lee et al. (2016) can be formally extended to more general reasoning tasks.

cs.AI↗

arXiv2016

Long-range states of the NaRb molecule near the Na($3^2S_{1/2}$)+Rb($5^2P_{3/2}$) asymptote

We report a high-resolution spectroscopic investigation of the long-range states of the $^{23}$Na$^{87}$Rb molecule near its Na($3^2S_{1/2}$)+Rb($5^2P_{3/2}$) asymptote. This study was performed with weakly bound ultracold molecules produced via magneto-association with an inter-species Feshbach resonance. We observed several regular vibrational series, which are assigned to the 5 attractive long-range states correlated with this asymptote. The vibrational levels of two of these states have sharp but complex structures due to hyperfine and Zeeman interactions. For the other states, we observed significant linewidth broadenings due to strong predissociation caused by spin-orbit couplings with states correlated to the lower Na($3^2S_{1/2}$)+Rb($5^2P_{1/2}$) asymptote. The long-range $C_6$ van der Waals coefficients extracted from our spectrum are in good agreement with theoretical values.

physics.atom-ph↗

arXiv2015

Recurrent Reinforcement Learning: A Hybrid Approach

Successful applications of reinforcement learning in real-world problems often require dealing with partially observable states. It is in general very challenging to construct and infer hidden states as they often depend on the agent's entire interaction history and may require substantial domain knowledge. In this work, we investigate a deep-learning approach to learning the representation of states in partially observable tasks, with minimal prior knowledge of the domain. In particular, we propose a new family of hybrid models that combines the strength of both supervised learning (SL) and reinforcement learning (RL), trained in a joint fashion: The SL component can be a recurrent neural networks (RNN) or its long short-term memory (LSTM) version, which is equipped with the desired property of being able to capture long-term dependency on history, thus providing an effective way of learning the representation of hidden states. The RL component is a deep Q-network (DQN) that learns to optimize the control for maximizing long-term rewards. Extensive experiments in a direct mailing campaign problem demonstrate the effectiveness and advantages of the proposed approach, which performs the best among a set of previous state-of-the-art methods.

cs.LG↗

arXiv2015

End-to-end Learning of LDA by Mirror-Descent Back Propagation over a Deep Architecture

We develop a fully discriminative learning approach for supervised Latent Dirichlet Allocation (LDA) model using Back Propagation (i.e., BP-sLDA), which maximizes the posterior probability of the prediction variable given the input document. Different from traditional variational learning or Gibbs sampling approaches, the proposed learning method applies (i) the mirror descent algorithm for maximum a posterior inference and (ii) back propagation over a deep architecture together with stochastic gradient/mirror descent for model parameter estimation, leading to scalable and end-to-end discriminative learning of the model. As a byproduct, we also apply this technique to develop a new learning method for the traditional unsupervised LDA model (i.e., BP-LDA). Experimental results on three real-world regression and classification tasks show that the proposed methods significantly outperform the previous supervised topic models, neural networks, and is on par with deep neural networks.

cs.LG↗

arXiv2015

Language Models for Image Captioning: The Quirks and What Works

Two recent approaches have achieved state-of-the-art results in image captioning. The first uses a pipelined process where a set of candidate words is generated by a convolutional neural network (CNN) trained on images, and then a maximum entropy (ME) language model is used to arrange these words into a coherent sentence. The second uses the penultimate activation layer of the CNN as input to a recurrent neural network (RNN) that then generates the caption sequence. In this paper, we compare the merits of these different language modeling approaches for the first time by using the same state-of-the-art CNN as input. We examine issues in the different approaches, including linguistic irregularities, caption repetition, and data set overlap. By combining key aspects of the ME and RNN methods, we achieve a new record performance over previously published results on the benchmark COCO dataset. However, the gains we see in BLEU do not translate to human judgments.

cs.CL↗

arXiv2015

Embedding Entities and Relations for Learning and Inference in Knowledge Bases

We consider learning representations of entities and relations in KBs using the neural-embedding approach. We show that most existing models, including NTN (Socher et al., 2013) and TransE (Bordes et al., 2013b), can be generalized under a unified learning framework, where entities are low-dimensional vectors learned from a neural network and relations are bilinear and/or linear mapping functions. Under this framework, we compare a variety of embedding models on the link prediction task. We show that a simple bilinear formulation achieves new state-of-the-art results for the task (achieving a top-10 accuracy of 73.2% vs. 54.7% by TransE on Freebase). Furthermore, we introduce a novel approach that utilizes the learned relation embeddings to mine logical rules such as "BornInCity(a,b) and CityInCountry(b,c) => Nationality(a,c)". We find that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication. More interestingly, we demonstrate that our embedding-based rule extraction approach successfully outperforms a state-of-the-art confidence-based rule mining approach in mining Horn rules that involve compositional reasoning.

cs.CL↗

arXiv2015

A Deep Embedding Model for Co-occurrence Learning

Co-occurrence Data is a common and important information source in many areas, such as the word co-occurrence in the sentences, friends co-occurrence in social networks and products co-occurrence in commercial transaction data, etc, which contains rich correlation and clustering information about the items. In this paper, we study co-occurrence data using a general energy-based probabilistic model, and we analyze three different categories of energy-based model, namely, the $L_1$, $L_2$ and $L_k$ models, which are able to capture different levels of dependency in the co-occurrence data. We also discuss how several typical existing models are related to these three types of energy models, including the Fully Visible Boltzmann Machine (FVBM) ($L_2$), Matrix Factorization ($L_2$), Log-BiLinear (LBL) models ($L_2$), and the Restricted Boltzmann Machine (RBM) model ($L_k$). Then, we propose a Deep Embedding Model (DEM) (an $L_k$ model) from the energy model in a \emph{principled} manner. Furthermore, motivated by the observation that the partition function in the energy model is intractable and the fact that the major objective of modeling the co-occurrence data is to predict using the conditional probability, we apply the \emph{maximum pseudo-likelihood} method to learn DEM. In consequence, the developed model and its learning method naturally avoid the above difficulties and can be easily used to compute the conditional probability in prediction. Interestingly, our method is equivalent to learning a special structured deep neural network using back-propagation and a special sampling strategy, which makes it scalable on large-scale datasets. Finally, in the experiments, we show that the DEM can achieve comparable or better results than state-of-the-art methods on datasets across several application domains.

cs.LG↗

arXiv2015

Joint Learning of Distributed Representations for Images and Texts

This technical report provides extra details of the deep multimodal similarity model (DMSM) which was proposed in (Fang et al. 2015, arXiv:1411.4952). The model is trained via maximizing global semantic similarity between images and their captions in natural language using the public Microsoft COCO database, which consists of a large set of images and their corresponding captions. The learned representations attempt to capture the combination of various visual concepts and cues.

cs.CV↗

arXiv2015

From Captions to Visual Concepts and Back

This paper presents a novel approach for automatically generating image descriptions: visual detectors, language models, and multimodal similarity models learnt directly from a dataset of image captions. We use multiple instance learning to train visual detectors for words that commonly occur in captions, including many different parts of speech such as nouns, verbs, and adjectives. The word detector outputs serve as conditional inputs to a maximum-entropy language model. The language model learns from a set of over 400,000 image descriptions to capture the statistics of word usage. We capture global semantics by re-ranking caption candidates using sentence-level features and a deep multimodal similarity model. Our system is state-of-the-art on the official Microsoft COCO benchmark, producing a BLEU-4 score of 29.1%. When human judges compare the system captions to ones written by other people on our held-out test set, the system captions have equal or better quality 34% of the time.

cs.CV↗

arXiv2015

Coherent spin mixing dynamics in thermal $^{87}$Rb spin-1 and spin-2 gases

We study the non-equilibrium coherent spin mixing dynamics in ferromagnetic spin-1 and antiferromagnetic spin-2 thermal gases of ultracold $^{87}$Rb atoms. Long lasting spin population oscillations with magnetic field dependent resonances are observed in both cases. Our observations are well reproduced by Boltzmann equations of the Wigner distribution function. Compared to the equation of motion of spinor Bose-Einstein condensates, the only difference here is a factor of two increase in the spin-dependent interaction, which is confirmed directly in the spin-2 case by measuring the relation between the oscillation amplitude and the sample's density.

physics.atm-clus↗

arXiv2015

Interaction induced decay of a heteronuclear two-atom system

Two-atom systems in small traps are of fundamental interest, first of all for understanding the role of interactions in degenerate cold gases and for the creation of quantum gates in quantum information processing with single-atom traps. One of the key quantities is the inelastic relaxation (decay) time when one of the atoms or both are in a higher hyperfine state. Here we measure this quantity in a heteronuclear system of $^{87}$Rb and $^{85}$Rb in a micro optical trap and demonstrate experimentally and theoretically the presence of both fast and slow relaxation processes, depending on the choice of the initial hyperfine states. The developed experimental method allows us to single out a particular relaxation process and, in this sense, our experiment is a "superclean platform" for collisional physics studies. Our results have also implications for engineering of quantum states via controlled collisions and creation of two-qubit quantum gates.

physics.atom-ph↗

arXiv2015

Formation of Ultracold NaRb Feshbach Molecules

We report the creation of ultracold bosonic $^{23}$Na$^{87}$Rb Feshbach molecules via magneto-association. By ramping the magnetic field across an interspecies Feshbach resonance, at least 4000 molecules can be produced out of the near degenerate ultracold mixture. Fast loss due to inelastic atom-molecule collisions is observed, which limits the pure molecule number, after residual atoms removal, to 1700. The pure molecule sample can live for 21.8(8) ms in the optical trap, long enough for future molecular spectroscopy studies toward coherently transferring to the singlet ro-vibrational ground state, where these molecules are stable against chemical reaction and have a permanent electric dipole moment of 3.3 Debye. We have also measured the Feshbach molecule's binding energy near the Feshbach resonance by the oscillating magnetic field method and found these molecules have a large closed-channel fraction.

physics.atom-ph↗

arXiv2015

Coherent heteronuclear spin dynamics in an ultracold spin-1 mixture

We report the observation of coherent heteronuclear spin dynamics driven by inter-species spin-spin interaction in an ultracold spinor mixture, which manifests as periodical and well correlated spin oscillations between two atomic species. In particular, we investigate the magnetic field dependence of the oscillations and find a resonance behavior which depends on {\em both} the linear and quadratic Zeeman effects and the spin-dependent interaction. We also demonstrate a unique knob for controlling the spin dynamics in the spinor mixture with species-dependent vector light shifts. Our finds are in agreement with theoretical simulations without any fitting parameters.

physics.atom-ph↗

arXiv2014

Learning Multi-Relational Semantics Using Neural-Embedding Models

In this paper we present a unified framework for modeling multi-relational representations, scoring, and learning, and conduct an empirical study of several recent multi-relational embedding models under the framework. We investigate the different choices of relation operators based on linear and bilinear transformations, and also the effects of entity representations by incorporating unsupervised vectors pre-trained on extra textual resources. Our results show several interesting findings, enabling the design of a simple embedding model that achieves the new state-of-the-art performance on a popular knowledge base completion task evaluated on Freebase.

cs.CL↗

arXiv2013

Learning Semantic Representations for the Phrase Translation Model

This paper presents a novel semantic-based phrase translation model. A pair of source and target phrases are projected into continuous-valued vector representations in a low-dimensional latent semantic space, where their translation score is computed by the distance between the pair in this new space. The projection is performed by a multi-layer neural network whose weights are learned on parallel training data. The learning is aimed to directly optimize the quality of end-to-end machine translation results. Experimental evaluation has been performed on two Europarl translation tasks, English-French and German-English. The results show that the new semantic-based phrase translation model significantly improves the performance of a state-of-the-art phrase-based statistical machine translation sys-tem, leading to a gain of 0.7-1.0 BLEU points.

cs.CL↗

arXiv2013

Suppressing phase decoherence of a single atom qubit with CPMG sequence

We experimentally demonstrate the strong suppression of dephasing of a qubit stored in a single \textsuperscript{87}Rb atom in an optical dipole trap by using Carr-Purcell-Meiboom-Gill(CPMG) pulse sequences. Regarded as a repetition of spin echo, CPMG sequence is an optimal choice for suppressing both inhomogeneous and homogeneous phase decoherence mechanisms. In the trap with 830 nm wavelength and 0.7 mK potential depth, the spin relaxation time of single atoms is showed to be 830.8 ms. We obtain the reversible inhomogeneous dephasing time of $T_{2}^{\ast}$=1.4 ms. The homogeneous dephasing time is $T_{2}^{\prime}$ =102.7 ms in the spin echo process, by employing CPMG sequence with pulse number n = 6 the homogeneous dephasing is further suppressed by a factor of 3, and its corresponding coherence time is extended to $T_{2}^{\prime}$=304.5 ms.

quant-ph↗

arXiv2012

Combining red- and blue-detuned optical potentials to form a Lamb-Dicke trap for a single neutral atom

We propose and demonstrate a scheme for strongly radially confining a single neutral atom in a bichromatic far-off resonance optical dipole trap(BFORT) . BFORT is composed of a blue-detuned Laguerre-Gaussian $LG^1_ 0$ beam and a red-detuned Gaussian beam. The trapping radial dimension of a single atom trapped in the Gaussian FORT can be greatly compressed by imposing a blue-detuned Laguerre-Gaussian $LG^1_ 0$ beam with moderate potential depth. By modulating the potential depth of the Gaussian FORT we observed that the resonant and parametric excitation of the oscillatory motion of a single atom in this BFORT and obtained the oscillation frequency that well fits prediction from the theoretical model. The frequency measurement shows that effective trapping dimension can be greatly sharper than that diffraction limited of microscopic objective we used. Then we show that the excess scattering rate due to imposing blue detuned light can be eliminated when single atoms is close to ground-state theoretically. So BFORT suits the purpose of acting as a Lamb-Dicke trap for further cooling a single neutral atom to motion ground-state and finding application in quantum information progressing.

physics.atom-ph↗