Searcharxiv⌕ Search

arXiv subjects

Liang Ding

Publications and source records attributed to Liang Ding.

At least 181 records · Page 10Linked to original sources
arXiv2021

Bridging the Gap Between Clean Data Training and Real-World Inference for Spoken Language Understanding

Spoken language understanding (SLU) system usually consists of various pipeline components, where each component heavily relies on the results of its upstream ones. For example, Intent detection (ID), and slot filling (SF) require its upstream automatic speech recognition (ASR) to transform the voice into text. In this case, the upstream perturbations, e.g. ASR errors, environmental noise and careless user speaking, will propagate to the ID and SF models, thus deteriorating the system performance. Therefore, the well-performing SF and ID models are expected to be noise resistant to some extent. However, existing models are trained on clean data, which causes a \textit{gap between clean data training and real-world inference.} To bridge the gap, we propose a method from the perspective of domain adaptation, by which both high- and low-quality samples are embedding into similar vector space. Meanwhile, we design a denoising generation model to reduce the impact of the low-quality samples. Experiments on the widely-used dataset, i.e. Snips, and large scale in-house dataset (10 million training examples) demonstrate that this method not only outperforms the baseline models on real-world (noisy) corpus but also enhances the robustness, that is, it produces high-quality results under a noisy environment. The source code will be released.

cs.CL↗
arXiv2021

Eulerian collinear configuration for 3-body problem

For 3-body problem with any given masses $m_1, \,m_2,\,m_3>0$, there exist only Eulerian collinear central configuration and Lagrangian equilateral-triangle central configuration, and in this paper, for planar 3-body problem, we prove that there exists another non-collision trajectory $q$, which is not the variational minimizer of the Lagrangian action on the loop space $\overline{Λ_1}$, is also an Eulerian collinear central configuration at any instant. Moreover, we do not need the restriction condition on the winding number $deg(q_i-q_j)\neq0 \,(i\neq j)$.

math.DS↗
arXiv2021

Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning

Encoder layer fusion (EncoderFusion) is a technique to fuse all the encoder layers (instead of the uppermost layer) for sequence-to-sequence (Seq2Seq) models, which has proven effective on various NLP tasks. However, it is still not entirely clear why and when EncoderFusion should work. In this paper, our main contribution is to take a step further in understanding EncoderFusion. Many of previous studies believe that the success of EncoderFusion comes from exploiting surface and syntactic information embedded in lower encoder layers. Unlike them, we find that the encoder embedding layer is more important than other intermediate encoder layers. In addition, the uppermost decoder layer consistently pays more attention to the encoder embedding layer across NLP tasks. Based on this observation, we propose a simple fusion method, SurfaceFusion, by fusing only the encoder embedding layer for the softmax layer. Experimental results show that SurfaceFusion outperforms EncoderFusion on several NLP benchmarks, including machine translation, text summarization, and grammatical error correction. It obtains the state-of-the-art performance on WMT16 Romanian-English and WMT14 English-French translation tasks. Extensive analyses reveal that SurfaceFusion learns more expressive bilingual word embeddings by building a closer relationship between relevant source and target embedding. Source code is freely available at https://github.com/SunbowLiu/SurfaceFusion.

cs.CL↗
arXiv2021

Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation

Dialogue generation models face the challenge of producing generic and repetitive responses. Unlike previous augmentation methods that mostly focus on token manipulation and ignore the essential variety within a single sample using hard labels, we propose to promote the generation diversity of the neural dialogue models via soft embedding augmentation along with soft labels in this paper. Particularly, we select some key input tokens and fuse their embeddings together with embeddings from their semantic-neighbor tokens. The new embeddings serve as the input of the model to replace the original one. Besides, soft labels are used in loss calculation, resulting in multi-target supervision for a given input. Our experimental results on two datasets illustrate that our proposed method is capable of generating more diverse responses than raw models while remains a similar n-gram accuracy that ensures the quality of generated responses.

cs.CL↗
arXiv2021

Understanding and Improving Lexical Choice in Non-Autoregressive Translation

Knowledge distillation (KD) is essential for training non-autoregressive translation (NAT) models by reducing the complexity of the raw data with an autoregressive teacher model. In this study, we empirically show that as a side effect of this training, the lexical choice errors on low-frequency words are propagated to the NAT model from the teacher model. To alleviate this problem, we propose to expose the raw data to NAT models to restore the useful information of low-frequency words, which are missed in the distilled data. To this end, we introduce an extra Kullback-Leibler divergence term derived by comparing the lexical choice of NAT model and that embedded in the raw data. Experimental results across language pairs and model architectures demonstrate the effectiveness and universality of the proposed approach. Extensive analyses confirm our claim that our approach improves performance by reducing the lexical choice errors on low-frequency words. Encouragingly, our approach pushes the SOTA NAT performance on the WMT14 English-German and WMT16 Romanian-English datasets up to 27.8 and 33.8 BLEU points, respectively. The source code will be released.

cs.CL↗
arXiv2021

Strong solutions of stochastic differential equations with square integrable drift

We prove the existence and uniqueness of strong solutions for stochastic differential equations in which the drift coefficient is square integrable in time variable and Hölder continuous in space variable. Moreover, we prove that the unique strong solution has a continuous modification, which is $β$-Hölder continuous in space variable for every $β\in (0,1)$, and as an $L^2(Ω\times (0,T))$ valued function, it is differentiable as well.

math.AP↗
arXiv2020

Self-Attention with Cross-Lingual Position Representation

Position encoding (PE), an essential part of self-attention networks (SANs), is used to preserve the word order information for natural language processing tasks, generating fixed position indices for input sequences. However, in cross-lingual scenarios, e.g. machine translation, the PEs of source and target sentences are modeled independently. Due to word order divergences in different languages, modeling the cross-lingual positional relationships might help SANs tackle this problem. In this paper, we augment SANs with \emph{cross-lingual position representations} to model the bilingually aware latent structure for the input sentence. Specifically, we utilize bracketing transduction grammar (BTG)-based reordering information to encourage SANs to learn bilingual diagonal alignments. Experimental results on WMT'14 English$\Rightarrow$German, WAT'17 Japanese$\Rightarrow$English, and WMT'17 Chinese$\Leftrightarrow$English translation tasks demonstrate that our approach significantly and consistently improves translation quality over strong baselines. Extensive analyses confirm that the performance gains come from the cross-lingual information.

cs.CL↗
arXiv2020

Predicting Terrain Mechanical Properties in Sight for Planetary Rovers with Semantic Clues

Non-geometric mobility hazards such as rover slippage and sinkage posing great challenges to costly planetary missions are closely related to the mechanical properties of terrain. In-situ proprioceptive processes for rovers to estimate terrain mechanical properties need to experience different slip as well as sinkage and are helpless to untraversed regions. This paper proposes to predict terrain mechanical properties with vision in the distance, which expands the sensing range to the whole view and can partly halt potential slippage and sinkage hazards in the planning stage. A semantic-based method is designed to predict bearing and shearing properties of terrain in two stages connected with semantic clues. The former segmentation phase segments terrain with a light-weighted network promising to be applied onboard with competitive 93% accuracy and high recall rate over 96%, while the latter inference phase predicts terrain properties in a quantitative manner based on human-like inference principles. The prediction results in several test routes are 12.5% and 10.8% in full-scale error and help to plan appropriate strategies to avoid suffering non-geometric hazards.

cs.RO↗
arXiv2020

Context-Aware Cross-Attention for Non-Autoregressive Translation

Non-autoregressive translation (NAT) significantly accelerates the inference process by predicting the entire target sequence. However, due to the lack of target dependency modelling in the decoder, the conditional generation process heavily depends on the cross-attention. In this paper, we reveal a localness perception problem in NAT cross-attention, for which it is difficult to adequately capture source context. To alleviate this problem, we propose to enhance signals of neighbour source tokens into conventional cross-attention. Experimental results on several representative datasets show that our approach can consistently improve translation quality over strong NAT baselines. Extensive analyses demonstrate that the enhanced cross-attention achieves better exploitation of source contexts by leveraging both local and global information.

cs.CL↗
arXiv2020

SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling

Slot filling and intent detection are two main tasks in spoken language understanding (SLU) system. In this paper, we propose a novel non-autoregressive model named SlotRefine for joint intent detection and slot filling. Besides, we design a novel two-pass iteration mechanism to handle the uncoordinated slots problem caused by conditional independence of non-autoregressive model. Experiments demonstrate that our model significantly outperforms previous models in slot filling task, while considerably speeding up the decoding (up to X 10.77). In-depth analyses show that 1) pretraining schemes could further enhance our model; 2) two-pass mechanism indeed remedy the uncoordinated slots.

cs.CL↗
arXiv2020

Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns

This paper describes our submission of the WMT 2020 Shared Task on Sentence Level Direct Assessment, Quality Estimation (QE). In this study, we empirically reveal the \textit{mismatching issue} when directly adopting BERTScore to QE. Specifically, there exist lots of mismatching errors between the source sentence and translated candidate sentence with token pairwise similarity. In response to this issue, we propose to expose explicit cross-lingual patterns, \textit{e.g.} word alignments and generation score, to our proposed zero-shot models. Experiments show that our proposed QE model with explicit cross-lingual patterns could alleviate the mismatching issue, thereby improving the performance. Encouragingly, our zero-shot QE method could achieve comparable performance with supervised QE method, and even outperforms the supervised counterpart on 2 out of 6 directions. We expect our work could shed light on the zero-shot QE model improvement.

cs.CL↗
arXiv2020

A projected gradient method for $α\ell_{1}-β\ell_{2}$ sparsity regularization

The non-convex $α\|\cdot\|_{\ell_1}-β\| \cdot\|_{\ell_2}$ $(α\geβ\geq0)$ regularization has attracted attention in the field of sparse recovery. One way to obtain a minimizer of this regularization is the ST-($α\ell_1-β\ell_2$) algorithm which is similar to the classical iterative soft thresholding algorithm (ISTA). It is known that ISTA converges quite slowly, and a faster alternative to ISTA is the projected gradient (PG) method. However, the conventional PG method is limited to the classical $\ell_1$ sparsity regularization. In this paper, we present two accelerated alternatives to the ST-($α\ell_1-β\ell_2$) algorithm by extending the PG method to the non-convex $α\ell_1-β\ell_2$ sparsity regularization. Moreover, we discuss a strategy to determine the radius $R$ of the $\ell_1$-ball constraint by Morozov's discrepancy principle. Numerical results are reported to illustrate the efficiency of the proposed approach.

math.NA↗
arXiv2020

$α\ell_{1}-β\ell_{2}$ sparsity regularization for nonlinear ill-posed problems

In this paper, we consider the $α\| \cdot\|_{\ell_1}-β\| \cdot\|_{\ell_2}$ sparsity regularization with parameter $α\geqβ\geq0$ for nonlinear ill-posed inverse problems. We investigate the well-posedness of the regularization. Compared to the case where $α>β\geq0$, the results for the case $α=β\geq0$ are weaker due to the lack of coercivity and Radon-Riesz property of the regularization term. Under certain condition on the nonlinearity of $F$, we prove that every minimizer of $ α\| \cdot\|_{\ell_1}-β\| \cdot\|_{\ell_2}$ regularization is sparse. For the case $α>β\geq0$, if the exact solution is sparse, we derive convergence rate $O(δ^{\frac{1}{2}})$ and $O(δ)$ of the regularized solution under two commonly adopted conditions on the nonlinearity of $F$, respectively. In particular, it is shown that the iterative soft thresholding algorithm can be utilized to solve the $ α\| \cdot\|_{\ell_1}-β\| \cdot\|_{\ell_2}$ regularization problem for nonlinear ill-posed equations. Numerical results illustrate the efficiency of the proposed method.

math.NA↗
arXiv2020

Fault Tolerant Free Gait and Footstep Planning for Hexapod Robot Based on Monte-Carlo Tree

Legged robots can pass through complex field environments by selecting gaits and discrete footholds carefully. Traditional methods plan gait and foothold separately and treat them as the single-step optimal process. However, such processing causes its poor passability in a sparse foothold environment. This paper novelly proposes a coordinative planning method for hexapod robots that regards the planning of gait and foothold as a sequence optimization problem with the consideration of dealing with the harshness of the environment as leg fault. The Monte Carlo tree search algorithm(MCTS) is used to optimize the entire sequence. Two methods, FastMCTS, and SlidingMCTS are proposed to solve some defeats of the standard MCTS applicating in the field of legged robot planning. The proposed planning algorithm combines the fault-tolerant gait method to improve the passability of the algorithm. Finally, compared with other planning methods, experiments on terrains with different densities of footholds and artificially-designed challenging terrain are carried out to verify our methods. All results show that the proposed method dramatically improves the hexapod robot's ability to pass through sparse footholds environment.

cs.RO↗
arXiv2020

Payload-agnostic Decoupling and Hybrid Vibration Isolation Control for a Maglev Platform with Redundant Actuation

Payload-specific vibration control may be suitable for a particular task but lacks generality and transferability required for adapting to the various payload. Self-decoupling and robust vibration control are the crucial problems to achieve payload-agnostic vibration control. However, there are problems still unsolved. In this article, we present a maglev vibration isolation platform (MVIP), which aims to attenuate vibration in the payload-agnostic task under a dynamic environment. Since efforts trying to suppress disturbance will encounter inevitable coupling problems, we analyzed the reasons resulting in it and proposed unique and effective solutions. To achieve payload-agnostic vibration control, we proposed a new control strategy, which is the main contribution of this article. It consists of a self-construct radial basis function neural network inversion (SRBFNNI) decoupling scheme and hybrid adaptive feed-forward internal model control (HAFIMC). The former one enables the MVIP to create a self inverse model with little prior knowledge and achieving self-decoupling. For the unique structure of MVIP, the vibration control problem is stated and addressed by the proposed HAFIMC, which utilizes the adaptive part to deal with the periodical disturbance and the internal mode part to deal with the stability.

eess.SY↗
arXiv2020

System Integration and Control Design of a Maglev Platform for Space Vibration Isolation

Micro-vibration has been a dominant factor impairing the performance of scientific experiments which are expected to be deployed in a micro-gravity environment such as Spacelab. The micro-vibration has a serious impact on scientific experiments requiring a quasi-static environment. Therefore, we proposed a maglev vibration isolation platform (MVIP) operating in six degrees of freedom (DOF) to fulfill the environmental requirements. In view of non-contact and large stroke requirements for micro-vibration isolation, an optimization method was utilized to design the actuator. Mathematical models of the actuator's remarkable nonlinearity were established so that its output can be compensated according to floater's varying position and the system's performance may be satisfied. Furthermore, aiming to adapt to an energy-limited environment such as Spacelab, an optimum allocation scheme was put forward. Considering the actuator's nonlinearity, accuracy and minimum energy-consumption can be obtained simultaneously. In view of operating in six DOF, methods for nonlinear compensation and system decoupling were discussed, the necessary controller was also presented. Simulation and experiments validate the system's performance. With a movement range of 10x10x8 mm and rotations of 200 mrad, the decay ratio of -40 dB/Dec between 1-10 Hz was obtained under closed-loop control.

eess.SY↗