arXiv subjects

Xu Sun

Publications and source records attributed to Xu Sun.

216 records · Page 12Linked to original sources

arXiv2017

Minimal Effort Back Propagation for Convolutional Neural Networks

As traditional neural network consumes a significant amount of computing resources during back propagation, \citet{Sun2017mePropSB} propose a simple yet effective technique to alleviate this problem. In this technique, only a small subset of the full gradients are computed to update the model parameters. In this paper we extend this technique into the Convolutional Neural Network(CNN) to reduce calculation in back propagation, and the surprising results verify its validity in CNN: only 5\% of the gradients are passed back but the model still achieves the same effect as the traditional CNN, or even better. We also show that the top-$k$ selection of gradients leads to a sparse calculation in back propagation, which may bring significant computational benefits for high computational complexity of convolution operation in CNN.

cs.LG↗

arXiv2017

Transfer Deep Learning for Low-Resource Chinese Word Segmentation with a Novel Neural Network

Recent studies have shown effectiveness in using neural networks for Chinese word segmentation. However, these models rely on large-scale data and are less effective for low-resource datasets because of insufficient training data. We propose a transfer learning method to improve low-resource word segmentation by leveraging high-resource corpora. First, we train a teacher model on high-resource corpora and then use the learned knowledge to initialize a student model. Second, a weighted data similarity method is proposed to train the student model on low-resource data. Experiment results show that our work significantly improves the performance on low-resource datasets: 2.3% and 1.5% F-score on PKU and CTB datasets. Furthermore, this paper achieves state-of-the-art results: 96.1%, and 96.2% F-score on PKU and CTB datasets.

cs.CL↗

arXiv2017

Improving Semantic Relevance for Sequence-to-Sequence Learning of Chinese Social Media Text Summarization

Current Chinese social media text summarization models are based on an encoder-decoder framework. Although its generated summaries are similar to source texts literally, they have low semantic relevance. In this work, our goal is to improve semantic relevance between source texts and summaries for Chinese social media summarization. We introduce a Semantic Relevance Based neural model to encourage high semantic similarity between texts and summaries. In our model, the source text is represented by a gated attention encoder, while the summary representation is produced by a decoder. Besides, the similarity score between the representations is maximized during training. Our experiments show that the proposed model outperforms baseline systems on a social media corpus.

cs.CL↗

arXiv2017

F-Score Driven Max Margin Neural Network for Named Entity Recognition in Chinese Social Media

We focus on named entity recognition (NER) for Chinese social media. With massive unlabeled text and quite limited labelled corpus, we propose a semi-supervised learning model based on B-LSTM neural network. To take advantage of traditional methods in NER such as CRF, we combine transition probability with deep learning in our model. To bridge the gap between label accuracy and F-score of NER, we construct a model which can be directly trained on F-score. When considering the instability of F-score driven method and meaningful information provided by label accuracy, we propose an integrated method to train on both F-score and label accuracy. Our integrated model yields 7.44\% improvement over previous state-of-the-art result.

cs.CL↗

arXiv2017

Lock-Free Parallel Perceptron for Graph-based Dependency Parsing

Dependency parsing is an important NLP task. A popular approach for dependency parsing is structured perceptron. Still, graph-based dependency parsing has the time complexity of $O(n^3)$, and it suffers from slow training. To deal with this problem, we propose a parallel algorithm called parallel perceptron. The parallel algorithm can make full use of a multi-core computer which saves a lot of training time. Based on experiments we observe that dependency parsing with parallel perceptron can achieve 8-fold faster training speed than traditional structured perceptron methods when using 10 threads, and with no loss at all in accuracy.

cs.CL↗

arXiv2017

A Generic Online Parallel Learning Framework for Large Margin Models

To speed up the training process, many existing systems use parallel technology for online learning algorithms. However, most research mainly focus on stochastic gradient descent (SGD) instead of other algorithms. We propose a generic online parallel learning framework for large margin models, and also analyze our framework on popular large margin algorithms, including MIRA and Structured Perceptron. Our framework is lock-free and easy to implement on existing systems. Experiments show that systems with our framework can gain near linear speed up by increasing running threads, and with no loss in accuracy.

cs.CL↗

arXiv2016

A New Recurrent Neural CRF for Learning Non-linear Edge Features

Conditional Random Field (CRF) and recurrent neural models have achieved success in structured prediction. More recently, there is a marriage of CRF and recurrent neural models, so that we can gain from both non-linear dense features and globally normalized CRF objective. These recurrent neural CRF models mainly focus on encode node features in CRF undirected graphs. However, edge features prove important to CRF in structured prediction. In this work, we introduce a new recurrent neural CRF model, which learns non-linear edge features, and thus makes non-linear features encoded completely. We compare our model with different neural models in well-known structured prediction tasks. Experiments show that our model outperforms state-of-the-art methods in NP chunking, shallow parsing, Chinese word segmentation and POS tagging.

cs.CL↗

arXiv2016

Governing equations for Probability densities of stochastic differential equations with discrete time delays

The time evolution of probability densities for solutions to stochastic differential equations (SDEs) without delay is usually described by Fokker-Planck equations, which require the adjoint of the infinitesimal generator for the solutions. However, Fokker-Planck equations do not exist for stochastic delay differential equations (SDDEs) because the solutions to SDDEs are not Markov processes and have no corresponding infinitesimal generators. In this paper, we address the open question of finding the governing equations for probability densities of SDDEs with discrete time delays. The governing equation is given in a simple form that facilitates theoretical analysis and numerical computation. An illustrative example is presented to verify the proposed governing equations.

math.DS↗

arXiv2016

Fokker-Planck equations for Marcus stochastic differential equations driven by Levy processes

Marcus stochastic differential equations (SDEs) often are appropriate models for stochastic dynamical systems driven by non-Gaussian Levy processes and have wide applications in engineering and physical sciences. The probability density of the solution to an SDE offers complete statistical information on the underlying stochastic process. Explicit formula for the Fokker-Planck equation, the governing equation for the probability density, is well-known when the SDE is driven by a Brownian motion. In this paper, we address the open question of finding the Fokker-Plank equations for Marcus SDEs in arbitrary dimensions driven by non-Gaussian Levy processes. The equations are given in a simple form that facilitates theoretical analysis and numerical computation. Several examples are presented to illustrate how the theoretical results can be applied to obtain Fokker-Planck equations for Marcus SDEs driven by Levy processes.

math.DS↗

arXiv2016

Derivation of Fokker-Planck equations for stochastic dynamical systems under excitation of multiplicative non-Gaussian white noise

Fokker-Planck equations describe time evolution of probability densities of stochastic dynamical systems and play an important role in quantifying propagation and evolution of uncertainty. Although Fokker-Planck equations can be written explicitly for nonlinear dynamical systems excited by Gaussian white noise, they are not available in general for nonlinear dynamical systems excited by multiplicative non-Gaussian white noise. Marcus stochastic differential equations are often appropriate models in engineering and physics for stochastic dynamical systems excited by non-Gaussian white noise. In this paper, we derive explicit forms of Fokker-Planck equations for one dimensional systems modeled by Marcus stochastic differential equations under multiplicative non-Gaussian white noise. As examples to illustrate the theoretical results, the derived formula is used to obtain Fokker-Plank equations for nonlinear dynamical systems under excitation of (i) $α$-stable white noise; (ii) combined Gaussian and Poisson white noise, respectively.

math.DS↗

arXiv2015

Elliptic and Triangular Flow of Identified Particles from the AMPT Model at RHIC Energies

The elliptic flow ($v_{2}$) at $\sqrt{s_{\rm NN}} = $ 11.5, 39, and 200 GeV and triangular flow ($v_{3}$) at $\sqrt{s_{\rm NN}} = $ 200 GeV of identified particles ($π^{\pm}, K^{\pm}, K^{0}_{S}, p, \bar{p}, ϕ, Λ$ and $\barΛ$) from 0\%--80\% central Au+Au collisions are analyzed using a multiphase transport (AMPT) model. It is shown that the experimental results from the $η$-sub event plane method can be reproduced with a parton scattering cross section between 1.5 and 3 mb. We also studied the differential and integrated $v_{2}/v_{3}$ ratios and conclude that they are anti-corrected with the parton scattering cross section.

nucl-ex↗

arXiv2015

Structure Regularization for Structured Prediction: Theories and Experiments

While there are many studies on weight regularization, the study on structure regularization is rare. Many existing systems on structured prediction focus on increasing the level of structural dependencies within the model. However, this trend could have been misdirected, because our study suggests that complex structures are actually harmful to generalization ability in structured prediction. To control structure-based overfitting, we propose a structure regularization framework via \emph{structure decomposition}, which decomposes training samples into mini-samples with simpler structures, deriving a model with better generalization power. We show both theoretically and empirically that structure regularization can effectively control overfitting risk and lead to better accuracy. As a by-product, the proposed method can also substantially accelerate the training speed. The method and the theoretical results can apply to general graphical models with arbitrary structures. Experiments on well-known tasks demonstrate that our method can easily beat the benchmark systems on those highly-competitive tasks, achieving state-of-the-art accuracies yet with substantially faster training speed.

cs.LG↗

arXiv2014

Exact Decoding on Latent Variable Conditional Models is NP-Hard

Latent variable conditional models, including the latent conditional random fields as a special case, are popular models for many natural language processing and vision processing tasks. The computational complexity of the exact decoding/inference in latent conditional random fields is unclear. In this paper, we try to clarify the computational complexity of the exact decoding. We analyze the complexity and demonstrate that it is an NP-hard problem even on a sequential labeling setting. Furthermore, we propose the latent-dynamic inference (LDI-Naive) method and its bounded version (LDI-Bounded), which are able to perform exact-inference or almost-exact-inference by using top-$n$ search and dynamic programming.

cs.AI↗

arXiv2013

State estimation under non-Gaussian Levy noise: A modified Kalman filtering method

The Kalman filter is extensively used for state estimation for linear systems under Gaussian noise. When non-Gaussian Lévy noise is present, the conventional Kalman filter may fail to be effective due to the fact that the non-Gaussian Lévy noise may have infinite variance. A modified Kalman filter for linear systems with non-Gaussian Lévy noise is devised. It works effectively with reasonable computational cost. Simulation results are presented to illustrate this non-Gaussian filtering method.

math.DS↗

arXiv2012

Modeling nonlinear random vibration: Implication of the energy conservation law

Nonlinear random vibration under excitations of both Gaussian and Poisson white noises is considered. The model is based on stochastic differential equations, and the corresponding stochastic integrals are defined in such a way that the energy conservation law is satisfied. It is shown that Stratonovich integral and Di Paola-Falsone integral should be used for excitations of Gaussian and Poisson white noises, respectively, in order for the model to satisfy the underlining physical laws (e.g., energy conservation). Numerical examples are presented to illustrate the theoretical results.

math.DS↗

arXiv2012

An alternative expression of Di Paola and Falson's formula for stochastic dynamics

Di Paola and Falsone's formula is widely used in studying stochastic dynamics of nonlinear systems under Poisson white noise. In this short communication, an alternative expression is presented. Compared to Di Paola and Falsone's original expression, the alternative one is applicable under more general condition, and shows significantly improved performance in numerical implementation. The alternative expression turns out to be a special case of the Marcus integrals.

math.DS↗

arXiv2012

Fokker-Planck equations for nonlinear dynamical systems driven by non-Gaussian Levy processes

The Fokker-Planck equations describe time evolution of probability densities of stochastic dynamical systems and are thus widely used to quantify random phenomena such as uncertainty propagation. For dynamical systems driven by non-Gaussian Lévy processes, however, it is difficult to obtain explicit forms of Fokker-Planck equations because the adjoint operators of the associated infinitesimal generators usually do not have exact formulation. In the present paper, Fokker- Planck equations are derived in terms of infinite series for nonlinear stochastic differential equations with non-Gaussian Lévy processes. A few examples are presented to illustrate the method.

math.DS↗

arXiv2011

Approximation of invariant foliations for stochastic dynamical systems

Invariant foliations are geometric structures for describing and understanding the qualitative behaviors of nonlinear dynamical systems. For stochastic dynamical systems, however, these geometric structures themselves are complicated random sets. Thus it is desirable to have some techniques to approximate random invariant foliations. In this paper, invariant foliations are approximated for dynamical systems with small noisy perturbations, via asymptotic analysis. Namely, random invariant foliations are represented as a perturbation of the deterministic invariant foliations, with deviation errors estimated.

math.DS↗