arXiv subjects

Linyang Li

Publications and source records attributed to Linyang Li.

At least 73 records · Page 4Linked to original sources

arXiv2022

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction

Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The meaning of a word in Chinese is different in that a word is a compositional unit consisting of multiple characters. Such difference motivates us to investigate whether WWM leads to better context understanding ability for Chinese BERT. To achieve this, we introduce two probing tasks related to grammatical error correction and ask pretrained models to revise or insert tokens in a masked language modeling manner. We construct a dataset including labels for 19,075 tokens in 10,448 sentences. We train three Chinese BERT models with standard character-level masking (CLM), WWM, and a combination of CLM and WWM, respectively. Our major findings are as follows: First, when one character needs to be inserted or replaced, the model trained with CLM performs the best. Second, when more than one character needs to be handled, WWM is the key to better performance. Finally, when being fine-tuned on sentence-level downstream tasks, models trained with different masking strategies perform comparably.

cs.CL↗

arXiv2021

Searching for an Effective Defender: Benchmarking Defense against Adversarial Word Substitution

Recent studies have shown that deep neural networks are vulnerable to intentionally crafted adversarial examples, and various methods have been proposed to defend against adversarial word-substitution attacks for neural NLP models. However, there is a lack of systematic study on comparing different defense approaches under the same attacking setting. In this paper, we seek to fill the gap of systematic studies through comprehensive researches on understanding the behavior of neural text classifiers trained by various defense methods under representative adversarial attacks. In addition, we propose an effective method to further improve the robustness of neural text classifiers against such attacks and achieved the highest accuracy on both clean and adversarial examples on AGNEWS and IMDB datasets by a significant margin.

cs.CL↗

arXiv2021

KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier

Pre-trained models are widely used in fine-tuning downstream tasks with linear classifiers optimized by the cross-entropy loss, which might face robustness and stability problems. These problems can be improved by learning representations that focus on similarities in the same class and contradictions in different classes when making predictions. In this paper, we utilize the K-Nearest Neighbors Classifier in pre-trained model fine-tuning. For this KNN classifier, we introduce a supervised momentum contrastive learning framework to learn the clustered representations of the supervised downstream tasks. Extensive experiments on text classification tasks and robustness tests show that by incorporating KNNs with the traditional fine-tuning process, we can obtain significant improvements on the clean accuracy in both rich-source and few-shot settings and can improve the robustness against adversarial attacks. \footnote{all codes is available at https://github.com/LinyangLee/KNN-BERT}

cs.CL↗

arXiv2021

Floquet band engineering and topological phase transitions in 1T' transition metal dichalcogenides

Using ab initio tight-binding approaches, we investigate Floquet band engineering of the 1T' phase of transition metal dichalcogenides (MX2, M = W, Mo and X = Te, Se, S) monolayers under the irradiation with circularly polarized light. Our first principles calculations demonstrate that light can induce important transitions in the topological phases of this emerging materials family. For example, upon irradiation, Te-based MX2 undergoes a phase transition from quantum spin Hall (QSH) semimetal to time-reversal symmetry broken QSH insulator with a nontrivial band gap of up to 92.5 meV. On the other hand, Se- and S-based MX2 undergoes the topological phase transition from the QSH effect to the quantum anomalous Hall (QAH) effect and into trivial phases with increasing light intensity. From a general perspective, our work brings further insight into non-equilibrium topological systems.

cond-mat.mtrl-sci↗

arXiv2021

Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning

\textbf{P}re-\textbf{T}rained \textbf{M}odel\textbf{s} have been widely applied and recently proved vulnerable under backdoor attacks: the released pre-trained weights can be maliciously poisoned with certain triggers. When the triggers are activated, even the fine-tuned model will predict pre-defined labels, causing a security threat. These backdoors generated by the poisoning methods can be erased by changing hyper-parameters during fine-tuning or detected by finding the triggers. In this paper, we propose a stronger weight-poisoning attack method that introduces a layerwise weight poisoning strategy to plant deeper backdoors; we also introduce a combinatorial trigger that cannot be easily detected. The experiments on text classification tasks show that previous defense methods cannot resist our weight-poisoning method, which indicates that our method can be widely applied and may provide hints for future model robustness studies.

cs.CR↗

arXiv2021

Ferromagnetism with in-plane magnetization, Dirac spin-gapless semiconducting property, and tunable topological states in two-dimensional rare-earth-metal dinitrides

As the bulk single-crystal MoN2/ReN2 with a layered structure was successfully synthesized in experiment, transition-metal dinitrides have attracted considerable attention in recent years. Here, we focus on rare-earth-metal (Rem) elements and propose seven stable Rem dinitride monolayers with a 1T structure, namely 1T-RemN2. These monolayers have a ferromagnetic ground state with in-plane magnetization. Without spin-orbit coupling (SOC) effect, the band structures are spin-polarized with Dirac points at the Fermi level. Remarkably, the 1T-LuN2 monolayer shows an isotropic magnetic anisotropy energy in the xy-plane with in-plane magnetization, indicating easy tunability of the magnetization direction. When rotating the magnetization vector in the xy-plane, our proposed model can accurately describe the variety of the SOC band gap and two topological states (Weyl-like semimetal and Chern insulator states) appear with tunable properties. The Weyl-like semimetal state is a critical point between the two Chern insulator states with opposite sign of the Chern numbers. The large nontrivial band gap (up to 60.3 meV) and the Weyl-like semimetal state are promising for applications in spintronic devices.

cond-mat.mtrl-sci↗

arXiv2021

Certified Robustness to Text Adversarial Attacks by Randomized [MASK]

Recently, few certified defense methods have been developed to provably guarantee the robustness of a text classifier to adversarial synonym substitutions. However, all existing certified defense methods assume that the defenders are informed of how the adversaries generate synonyms, which is not a realistic scenario. In this paper, we propose a certifiably robust defense method by randomly masking a certain proportion of the words in an input text, in which the above unrealistic assumption is no longer necessary. The proposed method can defend against not only word substitution-based attacks, but also character-level perturbations. We can certify the classifications of over 50% texts to be robust to any perturbation of 5 words on AGNEWS, and 2 words on SST2 dataset. The experimental results show that our randomized smoothing method significantly outperforms recently proposed defense methods across multiple datasets.

cs.CL↗

arXiv2021

SENT: Sentence-level Distant Relation Extraction via Negative Training

Distant supervision for relation extraction provides uniform bag labels for each sentence inside the bag, while accurate sentence labels are important for downstream applications that need the exact relation type. Directly using bag labels for sentence-level training will introduce much noise, thus severely degrading performance. In this work, we propose the use of negative training (NT), in which a model is trained using complementary labels regarding that ``the instance does not belong to these complementary labels". Since the probability of selecting a true label as a complementary label is low, NT provides less noisy information. Furthermore, the model trained with NT is able to separate the noisy data from the training data. Based on NT, we propose a sentence-level framework, SENT, for distant relation extraction. SENT not only filters the noisy data to construct a cleaner dataset, but also performs a re-labeling process to transform the noisy data into useful training data, thus further benefiting the model's performance. Experimental results show the significant improvement of the proposed method over previous methods on sentence-level evaluation and de-noise effect.

cs.CL↗

arXiv2021

Room temperature two-dimensional antiferromagnetic Weyl semimetal CrO with giant spin-splitting and spin-momentum locked transport

Giant spin-splitting was recently predicted in collinear antiferromagnetic materials with a specific class of magnetic space group. In this work, we have predicted a two-dimensional (2D) antiferromagnetic Weyl semimetal (WS), CrO with large spin-split band structure, spin-momentum locked transport properties and high Néel temperature. It has two pairs of spin-polarized Weyl points at the Fermi level. By manipulating the position of the Weyl points with strain, four different antiferromagnetic spintronic states can be achieved: WSs with two spin-polarized transport channels (STCs), WSs with single STC, semiconductors with two STCs, and semiconductors with single STC. Based on these properties, a new avenue in spintronics with 2D collinear antiferromagnets is proposed.

cond-mat.mtrl-sci↗

arXiv2020

Generating Adversarial Examples in Chinese Texts Using Sentence-Pieces

Adversarial attacks in texts are mostly substitution-based methods that replace words or characters in the original texts to achieve success attacks. Recent methods use pre-trained language models as the substitutes generator. While in Chinese, such methods are not applicable since words in Chinese require segmentations first. In this paper, we propose a pre-train language model as the substitutes generator using sentence-pieces to craft adversarial examples in Chinese. The substitutions in the generated adversarial examples are not characters or words but \textit{'pieces'}, which are more natural to Chinese readers. Experiments results show that the generated adversarial samples can mislead strong target models and remain fluent and semantically preserved.

cs.CL↗

arXiv2020

Structural phase transition in monolayer gold(I) telluride: From a room-temperature topologicalinsulator to an auxetic semiconductor

Structural phase transitions between semiconductors and topological insulators have rich applications in nanoelectronics but are rarely found in two-dimensional (2D) materials. In this work, by combining ab initio computations and evolutionary structure search, we investigate two stable 2D forms of gold(I) telluride (Au$_{2}$Te) with square symmetry, noted as s(I)- and s(II)-Au$_{2}$Te. s(II)-Au$_{2}$Te is the global minimum structure and is a room-temperature topological insulator. s(I)-Au$_{2}$Te is a direct-gap semiconductor with high carrier mobilities and unusual in-plane negative Poisson's ratio. Both s(I) and s(II) phases have ultra-low Young's modulus, implying high flexibility. By applying a small tensile strain, s(II)-Au$_{2}$Te can be transformed into s(I)-Au$_{2}$Te. Hence, a structural phase transition from a room-temperature topological insulator to an auxetic semiconductor is found in the 2D forms of Au$_{2}$Te, which enables potential applications in phase-change electronic devices. Moreover, we elucidate the mechanism of the phase transition with the help of phonon spectra and group theory analysis.

cond-mat.mtrl-sci↗

arXiv2020

TAVAT: Token-Aware Virtual Adversarial Training for Language Understanding

Gradient-based adversarial training is widely used in improving the robustness of neural networks, while it cannot be easily adapted to natural language processing tasks since the embedding space is discrete. In natural language processing fields, virtual adversarial training is introduced since texts are discrete and cannot be perturbed by gradients directly. Alternatively, virtual adversarial training, which generates perturbations on the embedding space, is introduced in NLP tasks. Despite its success, existing virtual adversarial training methods generate perturbations roughly constrained by Frobenius normalization balls. To craft fine-grained perturbations, we propose a Token-Aware Virtual Adversarial Training method. We introduce a token-level accumulated perturbation vocabulary to initialize the perturbations better and use a token-level normalization ball to constrain these perturbations pertinently. Experiments show that our method improves the performance of pre-trained models such as BERT and ALBERT in various tasks by a considerable margin. The proposed method improves the score of the GLUE benchmark from 78.3 to 80.9 using BERT model and it also enhances the performance of sequence labeling and text classification tasks.

cs.CL↗

arXiv2020

BERT-ATTACK: Adversarial Attack Against BERT Using BERT

Adversarial attacks for discrete data (such as texts) have been proved significantly more challenging than continuous data (such as images) since it is difficult to generate adversarial samples with gradient-based methods. Current successful attack methods for texts usually adopt heuristic replacement strategies on the character or word level, which remains challenging to find the optimal solution in the massive space of possible combinations of replacements while preserving semantic consistency and language fluency. In this paper, we propose \textbf{BERT-Attack}, a high-quality and effective method to generate adversarial samples using pre-trained masked language models exemplified by BERT. We turn BERT against its fine-tuned models and other deep neural models in downstream tasks so that we can successfully mislead the target models to predict incorrectly. Our method outperforms state-of-the-art attack strategies in both success rate and perturb percentage, while the generated adversarial samples are fluent and semantically preserved. Also, the cost of calculation is low, thus possible for large-scale generations. The code is available at https://github.com/LinyangLee/BERT-Attack.

cs.CL↗

arXiv2020

PAI-graphene: a new topological semimetallic two-dimensional carbon allotrope with highly tunable anisotropic Dirac cones

Using evolutionary algorithm for crystal structure prediction, we present a new stable two-dimensional (2D) carbon allotrope composed of polymerized as-indacenes (PAI) in a zigzag pattern, namely PAI-graphene whose energy is lower than most of the reported 2D allotropes of graphene. Crucially, the crystal structure realizes a nonsymmorphic layer group that enforces a nontrivial global topology of the band structure with two Dirac cones lying perfectly at the Fermi level. The absence of electron/hole pockets makes PAI-graphene a pristine crystalline topological semimetal having anisotropic Fermi velocities with a high value of $7.0 \times 10^{5}$ m/s. We show that while the semimetallic property of the allotrope is robust against the application of strain, the positions of the Dirac cone and the Fermi velocities can be modified significantly with strain. Moreover, by combining strain along both the x- and y-directions, two band inversions take place at $Γ$ leading to the annihilation of the Dirac nodes demonstrating the possibility of strain-controlled conversion of a topological semimetal into a semiconductor. Finally we formulate the bulk-boundary correspondence of the topological nodal phase in the form of a generalized Zak-phase argument finding a perfect agreement with the topological edge states computed for different edge-terminations.

cond-mat.mtrl-sci↗

arXiv2020

The magnetic, electronic, and light-induced topological properties in two-dimensional hexagonal FeX2 (X = Cl, Br, I) monolayers

Topological materials are fertile ground for investigating topological phases of matter and topological phase transitions. In particular, the quest for novel topological phases in 2D materials is attracting fast growing attention. Here, using Floquet-Bloch theory, we propose to realize chiral topological phases in 2D hexagonal FeX2 (X=Cl, Br, I) monolayers under irradiation of circularly polarized light. Such 2D FeX2 monolayers are predicted to be dynamical stable, and exhibit both ferromagnetic and semiconducting properties. To capture the full topological physics of the magnetic semiconductor under periodic driving, we adopt ab initio Wannier-based tight-binding methods for the Floquet-Bloch bands, with the light-induced band gap closings and openings being obtained as the light field strength increases. The calculations of slab with open boundaries show the existence of chiral edge states. Interestingly, the topological transitions with branches of chiral edge states changing from zero to one and from one to two by tuning the light amplitude are obtained, showing that the topological floquet phase of high Chern number can be induced in the present Floquet-Bloch systems.

cond-mat.mes-hall↗

arXiv2018

U-Net: Machine Reading Comprehension with Unanswerable Questions

Machine reading comprehension with unanswerable questions is a new challenging task for natural language processing. A key subtask is to reliably predict whether the question is unanswerable. In this paper, we propose a unified model, called U-Net, with three important components: answer pointer, no-answer pointer, and answer verifier. We introduce a universal node and thus process the question and its context passage as a single contiguous sequence of tokens. The universal node encodes the fused information from both the question and passage, and plays an important role to predict whether the question is answerable and also greatly improves the conciseness of the U-Net. Different from the state-of-art pipeline models, U-Net can be learned in an end-to-end fashion. The experimental results on the SQuAD 2.0 dataset show that U-Net can effectively predict the unanswerability of questions and achieves an F1 score of 71.7 on SQuAD 2.0.

cs.CL↗

arXiv2018

Quantum anomalous Hall effect in stable 1T-YN$_2$ monolayer with a large nontrivial band gap and high Chern number

The quantum anomalous Hall (QAH) effect is a topologically nontrivial phase, characterized by a non-zero Chern number defined in the bulk and chiral edge states in the boundary. Using first-principles calculations, we demonstrate the presence of the QAH effect in 1T-YN$_2$ monolayer, which was recently predicted to be a Dirac half metal without spin-orbit coupling (SOC). We show that the inclusion of SOC opens up a large nontrivial band gap of nearly $0.1$ eV in the electronic band structure. This results in the nontrivial bulk topology which is confirmed by the calculation of Berry curvature, anomalous Hall conductance and the presence of chiral edge states. Remarkably, a high Chern number $C = 3$ is found, and there are three corresponding gapless chiral edge states emerging inside the bulk gap. Our results open a new avenue in searching for QAH insulators with high temperature and high Chern numbers, which can have nontrivial practical applications.

cond-mat.mtrl-sci↗

arXiv2018

New Nanoporous Graphyne Monolayer as Nodal Line Semimetal: Double Dirac Points with an Ultrahigh Fermi Velocity

Two-dimensional (2D) carbon materials play an important role in nanomaterials. We propose a new carbon monolayer, named hexagonal-4,4,4-graphyne (H4,4,4-graphyne), which is a nanoporous structure composed of rectangular carbon rings and triple bonds of carbon. Using first-principles calculations, we systematically studied the structure, stability, and band structure of this new material. We found that its energy is much lower than that of some experimental carbon materials and it is stable at least up to 1500 K. In contrast to the single Dirac point band structure of other 2D carbon monolayers, the band structure of H4,4,4-graphyne exhibits double Dirac points along the high symmetry points and the corresponding Fermi velocities (1.04~1.27 * 106 m/s) are asymmetric and higher than that of graphene. The origin of these double Dirac points is traced back to the nodal line states, which can be well explained by a tight-binding model. The H4,4,4-graphyne forms a moiré superstructure when placed on top of a BN substrate, while keeping the double Dirac points. These properties make H4,4,4-graphyne a promising semimetal material for applications in high-speed electronic devices.

cond-mat.mtrl-sci↗