arXiv subjects

Zhi Jin

Publications and source records attributed to Zhi Jin.

227 records · Page 13Linked to original sources

arXiv2016

Distilling Word Embeddings: An Encoding Approach

Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems. This paper addresses the problem of distilling word embeddings for NLP tasks. We propose an encoding approach to distill task-specific knowledge from a set of high-dimensional embeddings, which can reduce model complexity by a large margin as well as retain high accuracy, showing a good compromise between efficiency and performance. Experiments in two tasks reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings.

cs.CL↗

arXiv2016

Natural Language Inference by Tree-Based Convolution and Heuristic Matching

In this paper, we propose the TBCNN-pair model to recognize entailment and contradiction between two sentences. In our model, a tree-based convolutional neural network (TBCNN) captures sentence-level semantics; then heuristic matching layers like concatenation, element-wise product/difference combine the information in individual sentences. Experimental results show that our model outperforms existing sentence encoding-based approaches by a large margin.

cs.CL↗

arXiv2016

Backward and Forward Language Modeling for Constrained Sentence Generation

Recent language models, especially those based on recurrent neural networks (RNNs), make it possible to generate natural language from a learned probability. Language generation has wide applications including machine translation, summarization, question answering, conversation systems, etc. Existing methods typically learn a joint probability of words conditioned on additional information, which is (either statically or dynamically) fed to RNN's hidden layer. In many applications, we are likely to impose hard constraints on the generated texts, i.e., a particular word must appear in the sentence. Unfortunately, existing approaches could not solve this problem. In this paper, we propose a novel backward and forward language model. Provided a specific word, we use RNNs to generate previous words and future words, either simultaneously or asynchronously, resulting in two model variants. In this way, the given word could appear at any position in the sentence. Experimental results show that the generated texts are comparable to sequential LMs in quality.

cs.CL↗

arXiv2015

Convolutional Neural Networks over Tree Structures for Programming Language Processing

Programming language processing (similar to natural language processing) is a hot research topic in the field of software engineering; it has also aroused growing interest in the artificial intelligence community. However, different from a natural language sentence, a program contains rich, explicit, and complicated structural information. Hence, traditional NLP models may be inappropriate for programs. In this paper, we propose a novel tree-based convolutional neural network (TBCNN) for programming language processing, in which a convolution kernel is designed over programs' abstract syntax trees to capture structural information. TBCNN is a generic architecture for programming language processing; our experiments show its effectiveness in two different program analysis tasks: classifying programs according to functionality, and detecting code snippets of certain patterns. TBCNN outperforms baseline methods, including several neural models for NLP.

cs.LG↗

arXiv2015

On End-to-End Program Generation from User Intention by Deep Neural Networks

This paper envisions an end-to-end program generation scenario using recurrent neural networks (RNNs): Users can express their intention in natural language; an RNN then automatically generates corresponding code in a characterby-by-character fashion. We demonstrate its feasibility through a case study and empirical analysis. To fully make such technique useful in practice, we also point out several cross-disciplinary challenges, including modeling user intention, providing datasets, improving model architectures, etc. Although much long-term research shall be addressed in this new field, we believe end-to-end program generation would become a reality in future decades, and we are looking forward to its practice.

cs.SE↗

arXiv2015

Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Path

Relation classification is an important research arena in the field of natural language processing (NLP). In this paper, we present SDP-LSTM, a novel neural network to classify the relation of two entities in a sentence. Our neural architecture leverages the shortest dependency path (SDP) between two entities; multichannel recurrent neural networks, with long short term memory (LSTM) units, pick up heterogeneous information along the SDP. Our proposed model has several distinct features: (1) The shortest dependency paths retain most relevant information (to relation classification), while eliminating irrelevant words in the sentence. (2) The multichannel LSTM networks allow effective information integration from heterogeneous sources over the dependency paths. (3) A customized dropout strategy regularizes the neural network to alleviate overfitting. We test our model on the SemEval 2010 relation classification task, and achieve an $F_1$-score of 83.7\%, higher than competing methods in the literature.

cs.CL↗

arXiv2015

A Comparative Study on Regularization Strategies for Embedding-based Neural Networks

This paper aims to compare different regularization strategies to address a common phenomenon, severe overfitting, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparameter tuning, and combining different regularizations. The results provide a picture on tuning hyperparameters for neural NLP models.

cs.CL↗

arXiv2015

Discriminative Neural Sentence Modeling by Tree-Based Convolution

This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling. Our models leverage either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences' structural features, and these features are aggregated by max pooling. Such architecture allows short propagation paths between the output layer and underlying feature detectors, which enables effective structural feature learning and extraction. We evaluate our models on two tasks: sentiment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.

cs.CL↗

arXiv2014

Building Program Vector Representations for Deep Learning

Deep learning has made significant breakthroughs in various fields of artificial intelligence. Advantages of deep learning include the ability to capture highly complicated features, weak involvement of human engineering, etc. However, it is still virtually impossible to use deep learning to analyze programs since deep architectures cannot be trained effectively with pure back propagation. In this pioneering paper, we propose the "coding criterion" to build program vector representations, which are the premise of deep learning for program analysis. Our representation learning approach directly makes deep learning a reality in this new field. We evaluate the learned vector representations both qualitatively and quantitatively. We conclude, based on the experiments, the coding criterion is successful in building program representations. To evaluate whether deep learning is beneficial for program analysis, we feed the representations to deep neural networks, and achieve higher accuracy in the program classification task than "shallow" methods, such as logistic regression and the support vector machine. This result confirms the feasibility of deep learning to analyze programs. It also gives primary evidence of its success in this new field. We believe deep learning will become an outstanding technique for program analysis in the near future.

cs.SE↗

arXiv2013

Proving Inequalities and Solving Global Optimization Problems via Simplified CAD Projection

Let $\xx_n=(x_1,\ldots,x_n)$ and $f\in \R[\xx_n,k]$. The problem of finding all $k_0$ such that $f(\xx_n,k_0)\ge 0$ on $\mathbb{R}^n$ is considered in this paper, which obviously takes as a special case the problem of computing the global infimum or proving the semi-definiteness of a polynomial. For solving the problems, we propose a simplified Brown's CAD projection operator, \Nproj, of which the projection scale is always no larger than that of Brown's. For many problems, the scale is much smaller than that of Brown's. As a result, the lifting phase is also simplified. Some new algorithms based on \Nproj\ for solving those problems are designed and proved to be correct. Comparison to some existing tools on some examples is reported to illustrate the effectiveness of our new algorithms.

cs.SC↗

arXiv2006

Evolving Quantum Oracles with Hybrid Quantum-inspired Evolutionary Algorithm

Quantum oracles play key roles in the studies of quantum computation and quantum information. But implementing quantum oracles efficiently with universal quantum gates is a hard work. Motivated by genetic programming, this paper proposes a novel approach to evolve quantum oracles with a hybrid quantum-inspired evolutionary algorithm. The approach codes quantum circuits with numerical values and combines the cost and correctness of quantum circuits into the fitness function. To speed up the calculation of matrix multiplication in the evaluation of individuals, a fast algorithm of matrix multiplication with Kronecker product is also presented. The experiments show the validity and the effects of some parameters of the presented approach. And some characteristics of the novel approach are discussed too.

quant-ph↗