arXiv subjects

Xin Jiang

Publications and source records attributed to Xin Jiang.

At least 217 records · Page 12Linked to original sources

arXiv2021

Integrating Regular Expressions with Neural Networks via DFA

Human-designed rules are widely used to build industry applications. However, it is infeasible to maintain thousands of such hand-crafted rules. So it is very important to integrate the rule knowledge into neural networks to build a hybrid model that achieves better performance. Specifically, the human-designed rules are formulated as Regular Expressions (REs), from which the equivalent Minimal Deterministic Finite Automatons (MDFAs) are constructed. We propose to use the MDFA as an intermediate model to capture the matched RE patterns as rule-based features for each input sentence and introduce these additional features into neural networks. We evaluate the proposed method on the ATIS intent classification task. The experiment results show that the proposed method achieves the best performance compared to neural networks and four other methods that combine REs and neural networks when the training dataset is relatively small.

cs.CL↗

arXiv2021

Generate & Rank: A Multi-task Framework for Math Word Problems

Math word problem (MWP) is a challenging and critical task in natural language processing. Many recent studies formalize MWP as a generation task and have adopted sequence-to-sequence models to transform problem descriptions to mathematical expressions. However, mathematical expressions are prone to minor mistakes while the generation objective does not explicitly handle such mistakes. To address this limitation, we devise a new ranking task for MWP and propose Generate & Rank, a multi-task framework based on a generative pre-trained language model. By joint training with generation and ranking, the model learns from its own mistakes and is able to distinguish between correct and incorrect expressions. Meanwhile, we perform tree-based disturbance specially designed for MWP and an online update to boost the ranker. We demonstrate the effectiveness of our proposed method on the benchmark and the results show that our method consistently outperforms baselines in all datasets. Particularly, in the classical Math23k, our method is 7% (78.4% $\rightarrow$ 85.4%) higher than the state-of-the-art.

cs.CL↗

arXiv2021

Blindfolded Attackers Still Threatening: Strict Black-Box Adversarial Attacks on Graphs

Adversarial attacks on graphs have attracted considerable research interests. Existing works assume the attacker is either (partly) aware of the victim model, or able to send queries to it. These assumptions are, however, unrealistic. To bridge the gap between theoretical graph attacks and real-world scenarios, in this work, we propose a novel and more realistic setting: strict black-box graph attack, in which the attacker has no knowledge about the victim model at all and is not allowed to send any queries. To design such an attack strategy, we first propose a generic graph filter to unify different families of graph-based models. The strength of attacks can then be quantified by the change in the graph filter before and after attack. By maximizing this change, we are able to find an effective attack strategy, regardless of the underlying model. To solve this optimization problem, we also propose a relaxation technique and approximation theories to reduce the difficulty as well as the computational expense. Experiments demonstrate that, even with no exposure to the model, the Macro-F1 drops 6.4% in node classification and 29.5% in graph classification, which is a significant result compared with existent works.

cs.LG↗

arXiv2021

Consolidating Kinematic Models to Promote Coordinated Mobile Manipulations

We construct a Virtual Kinematic Chain (VKC) that readily consolidates the kinematics of the mobile base, the arm, and the object to be manipulated in mobile manipulations. Accordingly, a mobile manipulation task is represented by altering the state of the constructed VKC, which can be converted to a motion planning problem, formulated, and solved by trajectory optimization. This new VKC perspective of mobile manipulation allows a service robot to (i) produce well-coordinated motions, suitable for complex household environments, and (ii) perform intricate multi-step tasks while interacting with multiple objects without an explicit definition of intermediate goals. In simulated experiments, we validate these advantages by comparing the VKC-based approach with baselines that solely optimize individual components. The results manifest that VKC-based joint modeling and planning promote task success rates and produce more efficient trajectories.

cs.RO↗

arXiv2021

AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models

Pre-trained language models (PLMs) have achieved great success in natural language processing. Most of PLMs follow the default setting of architecture hyper-parameters (e.g., the hidden dimension is a quarter of the intermediate dimension in feed-forward sub-networks) in BERT (Devlin et al., 2019). Few studies have been conducted to explore the design of architecture hyper-parameters in BERT, especially for the more efficient PLMs with tiny sizes, which are essential for practical deployment on resource-constrained devices. In this paper, we adopt the one-shot Neural Architecture Search (NAS) to automatically search architecture hyper-parameters. Specifically, we carefully design the techniques of one-shot learning and the search space to provide an adaptive and efficient development way of tiny PLMs for various latency constraints. We name our method AutoTinyBERT and evaluate its effectiveness on the GLUE and SQuAD benchmarks. The extensive experiments show that our method outperforms both the SOTA search-based baseline (NAS-BERT) and the SOTA distillation-based methods (such as DistilBERT, TinyBERT, MiniLM and MobileBERT). In addition, based on the obtained architectures, we propose a more efficient development method that is even faster than the development of a single PLM.

cs.CL↗

arXiv2021

Testing Kerr Black Hole Mimickers with Quasi-Periodic Oscillations from GRO J1655-40

The measurements of quasi-periodic oscillations (QPOs) provide a quite powerful tool to test the nature of astrophysical black hole candidates in the strong gravitational field regime. In this paper, we use QPOs within the relativistic precession model to test a recently proposed family of rotating black hole mimickers, which reduce to the Kerr metric in a limiting case, and can represent traversable wormholes or regular black holes with one or two horizons, depending on the values of the parameters. In particular, assuming that the compact object of GRO J1655-40 is described by a rotating black hole mimicker, we perform a $χ$-square analysis to fit the parameters of the mimicker with two sets of observed QPO frequencies from GRO J1655-40. Our results indicate that although the metric around the compact object of GRO J1655-40 is consistent with the Kerr metric, a regular black hole with one horizon is favored by the observation data of GRO J1655-40.

gr-qc↗

arXiv2021

BinaryBERT: Pushing the Limit of BERT Quantization

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

cs.CL↗

arXiv2021

Learning Multilingual Representation for Natural Language Understanding with Enhanced Cross-Lingual Supervision

Recently, pre-training multilingual language models has shown great potential in learning multilingual representation, a crucial topic of natural language processing. Prior works generally use a single mixed attention (MA) module, following TLM (Conneau and Lample, 2019), for attending to intra-lingual and cross-lingual contexts equivalently and simultaneously. In this paper, we propose a network named decomposed attention (DA) as a replacement of MA. The DA consists of an intra-lingual attention (IA) and a cross-lingual attention (CA), which model intralingual and cross-lingual supervisions respectively. In addition, we introduce a language-adaptive re-weighting strategy during training to further boost the model's performance. Experiments on various cross-lingual natural language understanding (NLU) tasks show that the proposed architecture and learning strategy significantly improve the model's cross-lingual transferability.

cs.CL↗

arXiv2021

Training Multilingual Pre-trained Language Model with Byte-level Subwords

The pre-trained language models have achieved great successes in various natural language understanding (NLU) tasks due to its capacity to capture the deep contextualized information in text by pre-training on large-scale corpora. One of the fundamental components in pre-trained language models is the vocabulary, especially for training multilingual models on many different languages. In the technical report, we present our practices on training multilingual pre-trained language models with BBPE: Byte-Level BPE (i.e., Byte Pair Encoding). In the experiment, we adopted the architecture of NEZHA as the underlying pre-trained language model and the results show that NEZHA trained with byte-level subwords consistently outperforms Google multilingual BERT and vanilla NEZHA by a notable margin in several multilingual NLU tasks. We release the source code of our byte-level vocabulary building tools and the multilingual pre-trained language models.

cs.CL↗

arXiv2021

Exploring Discourse Structures for Argument Impact Classification

Discourse relations among arguments reveal logical structures of a debate conversation. However, no prior work has explicitly studied how the sequence of discourse relations influence a claim's impact. This paper empirically shows that the discourse relations between two arguments along the context path are essential factors for identifying the persuasive power of an argument. We further propose DisCOC to inject and fuse the sentence-level structural discourse information with contextualized features derived from large-scale language models. Experimental results and extensive analysis show that the attention and gate mechanisms that explicitly model contexts and texts can indeed help the argument impact classification task defined by Durmus et al. (2019), and discourse structures among the context path of the claim to be classified can further boost the performance.

cs.CL↗

arXiv2021

Deformation Control of a Deformable Object Based on Visual and Tactile Feedback

In this paper, we presented a new method for deformation control of deformable objects, which utilizes both visual and tactile feedback. At present, manipulation of deformable objects is basically formulated by assuming positional constraints. But in fact, in many situations manipulation has to be performed under actively applied force constraints. This scenario is considered in this research. In the proposed scheme a tactile feedback is integrated to ensure a stable contact between the robot end-effector and the soft object to be manipulated. The controlled contact force is also utilized to regulate the deformation of the soft object with its shape measured by a vision sensor. The effectiveness of the proposed method is demonstrated by a book page turning and shaping experiment.

cs.RO↗

arXiv2021

Unsupervised Adversarially-Robust Representation Learning on Graphs

Unsupervised/self-supervised pre-training methods for graph representation learning have recently attracted increasing research interests, and they are shown to be able to generalize to various downstream applications. Yet, the adversarial robustness of such pre-trained graph learning models remains largely unexplored. More importantly, most existing defense techniques designed for end-to-end graph representation learning methods require pre-specified label definitions, and thus cannot be directly applied to the pre-training methods. In this paper, we propose an unsupervised defense technique to robustify pre-trained deep graph models, so that the perturbations on the input graph can be successfully identified and blocked before the model is applied to different downstream tasks. Specifically, we introduce a mutual information-based measure, \textit{graph representation vulnerability (GRV)}, to quantify the robustness of graph encoders on the representation space. We then formulate an optimization problem to learn the graph representation by carefully balancing the trade-off between the expressive power and the robustness (\emph{i.e.}, GRV) of the graph encoder. The discrete nature of graph topology and the joint space of graph data make the optimization problem intractable to solve. To handle the above difficulty and to reduce computational expense, we further relax the problem and thus provide an approximate solution. Additionally, we explore a provable connection between the robustness of the unsupervised graph encoder and that of models on downstream tasks. Extensive experiments demonstrate that even without access to labels and tasks, our model is still able to enhance robustness against adversarial attacks on three downstream tasks (node classification, link prediction, and community detection) by an average of +16.5% compared with existing methods.

cs.LG↗

arXiv2021

Improved OOD Generalization via Adversarial Training and Pre-training

Recently, learning a model that generalizes well on out-of-distribution (OOD) data has attracted great attention in the machine learning community. In this paper, after defining OOD generalization via Wasserstein distance, we theoretically show that a model robust to input perturbation generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we theoretically show that adversarially trained models have converged excess risk on OOD data, and empirically verify it on both image classification and natural language understanding tasks. Besides, in the paradigm of first pre-training and then fine-tuning, we theoretically show that a pre-trained model that is more robust to input perturbation provides a better initialization for generalization on downstream OOD data. Empirically, after fine-tuning, this better-initialized model from adversarial pre-training also has better OOD generalization.

cs.LG↗

arXiv2021

A novel feed rate scheduling method based on Sigmoid function with chord error and kinematics constraints

In high speed CNC (Compute Numerical Control) machining, the feed rate scheduling has played an important role to ensure machining quality and machining efficiency. In this paper, a novel feed rate scheduling method is proposed for generating smooth feed rate profile conveniently with the consideration of both geometric error and kinematic error. First, a relationship between feed rate value and chord error is applied to determine the feed rate curve. Then, breaking points, which can split whole curve into several blocks, can be found out using proposed two step screening method. For every block, a feed rate profile based on the Sigmoid function is generated. With the consideration of kinematic limitation and machining efficiency, a time-optimal feed rate adjustment algorithm is proposed to further adjust feed rate value at breaking points. After planning feed rate profile for each block, all blocks feed rate profile will be connected smoothly. The resulting feed rate profile is more concise compared with the polynomial profile and more efficient compared with the trigonometric profile. Finally, simulations with two free-form NURBS curves are conducted and comparison with the sine-curve method are carried out to verify the feasibility and applicability of the proposed method.

eess.SY↗

arXiv2021

PanGu-$α$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$α$, with up to 200 billion parameters. PanGu-$α$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$α$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$α$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$α$ in performing various tasks under few-shot or zero-shot settings.

cs.CL↗

arXiv2021

Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation

Task-agnostic knowledge distillation, a teacher-student framework, has been proved effective for BERT compression. Although achieving promising results on NLP tasks, it requires enormous computational resources. In this paper, we propose Extract Then Distill (ETD), a generic and flexible strategy to reuse the teacher's parameters for efficient and effective task-agnostic distillation, which can be applied to students of any size. Specifically, we introduce two variants of ETD, ETD-Rand and ETD-Impt, which extract the teacher's parameters in a random manner and by following an importance metric respectively. In this way, the student has already acquired some knowledge at the beginning of the distillation process, which makes the distillation process converge faster. We demonstrate the effectiveness of ETD on the GLUE benchmark and SQuAD. The experimental results show that: (1) compared with the baseline without an ETD strategy, ETD can save 70\% of computation cost. Moreover, it achieves better results than the baseline when using the same computing resource. (2) ETD is generic and has been proven effective for different distillation methods (e.g., TinyBERT and MiniLM) and students of different sizes. The source code will be publicly available upon publication.

cs.CL↗

arXiv2021

A novel S-shape based NURBS interpolation with acc-jerk- Continuity and round-off error elimination

Feedrate scheduling is a key step in computer numerical control (CNC) machining, as it has a close relationship with machining time and surface quality, and has now become a hot issue in industry and academia. To reduce high chord errors and round-off errors, and generate continuous velocity, acceleration, and jerk profile of parametric interpolation, a novel and complete S-shape based feedrate scheduling algorithm is presented in this paper. The algorithm consists of three modules: bidirectional scanning module, velocity scheduling module and round-off error elimination module. The bidirectional scanning module with the limitations of chord error, normal acceleration/jerk and command feedrate aims to guarantee the continuity of the feed rate at the junctions between successive NURBS sub-curves. After the NURBS sub-curves have been classified into two cases, the velocity scheduling module firstly calculates the actual maximum federate, and then generates the feed rate profiles of all NURBS sub-curves according to our velocity scheduling function. Later, the round-off error elimination module is proposed to make the total interpolating time become an integer multiple of the interpolation period, which leads to the elimination of round-off errors. Finally, benchmarks are conducted to verify the applicability of the proposed method compared with some other methods.

eess.SY↗

arXiv2021

An Approach to Improve Robustness of NLP Systems against ASR Errors

Speech-enabled systems typically first convert audio to text through an automatic speech recognition (ASR) model and then feed the text to downstream natural language processing (NLP) modules. The errors of the ASR system can seriously downgrade the performance of the NLP modules. Therefore, it is essential to make them robust to the ASR errors. Previous work has shown it is effective to employ data augmentation methods to solve this problem by injecting ASR noise during the training process. In this paper, we utilize the prevalent pre-trained language model to generate training samples with ASR-plausible noise. Compare to the previous methods, our approach generates ASR noise that better fits the real-world error distribution. Experimental results on spoken language translation(SLT) and spoken language understanding (SLU) show that our approach effectively improves the system robustness against the ASR errors and achieves state-of-the-art results on both tasks.

cs.CL↗