arXiv subjects

Hua Wu

Publications and source records attributed to Hua Wu.

At least 127 records · Page 7Linked to original sources

arXiv2022

GEM-2: Next Generation Molecular Property Prediction Network by Modeling Full-range Many-body Interactions

Molecular property prediction is a fundamental task in the drug and material industries. Physically, the properties of a molecule are determined by its own electronic structure, which is a quantum many-body system and can be exactly described by the Schr"odinger equation. Full-range many-body interactions between electrons have been proven effective in obtaining an accurate solution of the Schr"odinger equation by classical computational chemistry methods, although modeling such interactions consumes an expensive computational cost. Meanwhile, deep learning methods have also demonstrated their competence in molecular property prediction tasks. Inspired by the classical computational chemistry methods, we design a novel method, namely GEM-2, which comprehensively considers full-range many-body interactions in molecules. Multiple tracks are utilized to model the full-range interactions between the many-bodies with different orders, and a novel axial attention mechanism is designed to approximate the full-range interaction modeling with much lower computational cost. Extensive experiments demonstrate the overwhelming superiority of GEM-2 over multiple baseline methods in quantum chemistry and drug discovery tasks. The ablation studies also verify the effectiveness of the full-range many-body interactions.

cs.LG↗

arXiv2022

PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation

To explore the limit of dialogue generation pre-training, we present the models of PLATO-XL with up to 11 billion parameters, trained on both Chinese and English social media conversations. To train such large models, we adopt the architecture of unified transformer with high computation and parameter efficiency. In addition, we carry out multi-party aware pre-training to better distinguish the characteristic information in social media conversations. With such designs, PLATO-XL successfully achieves superior performances as compared to other approaches in both Chinese and English chitchat. We further explore the capacity of PLATO-XL on other conversational tasks, such as knowledge grounded dialogue and task-oriented conversation. The experimental results indicate that PLATO-XL obtains state-of-the-art results across multiple conversational tasks, verifying its potential as a foundation model of conversational AI.

cs.CL↗

arXiv2022

CDConv: A Benchmark for Contradiction Detection in Chinese Conversations

Dialogue contradiction is a critical issue in open-domain dialogue systems. The contextualization nature of conversations makes dialogue contradiction detection rather challenging. In this work, we propose a benchmark for Contradiction Detection in Chinese Conversations, namely CDConv. It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intra-sentence Contradiction, Role Confusion, and History Contradiction. To efficiently construct the CDConv conversations, we devise a series of methods for automatic conversation generation, which simulate common user behaviors that trigger chatbots to make contradictions. We conduct careful manual quality screening of the constructed conversations and show that state-of-the-art Chinese chatbots can be easily goaded into making contradictions. Experiments on CDConv show that properly modeling contextual information is critical for dialogue contradiction detection, but there are still unresolved challenges that require future research.

cs.CL↗

arXiv2022

ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding

Recent years have witnessed the rise and success of pre-training techniques in visually-rich document understanding. However, most existing methods lack the systematic mining and utilization of layout-centered knowledge, leading to sub-optimal performances. In this paper, we propose ERNIE-Layout, a novel document pre-training solution with layout knowledge enhancement in the whole workflow, to learn better representations that combine the features from text, layout, and image. Specifically, we first rearrange input sequences in the serialization stage, and then present a correlative pre-training task, reading order prediction, to learn the proper reading order of documents. To improve the layout awareness of the model, we integrate a spatial-aware disentangled attention into the multi-modal transformer and a replaced regions prediction task into the pre-training phase. Experimental results show that ERNIE-Layout achieves superior performance on various downstream tasks, setting new state-of-the-art on key information extraction, document image classification, and document question answering datasets. The code and models are publicly available at http://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/ernie-layout.

cs.CL↗

arXiv2022

Q-TOD: A Query-driven Task-oriented Dialogue System

Existing pipelined task-oriented dialogue systems usually have difficulties adapting to unseen domains, whereas end-to-end systems are plagued by large-scale knowledge bases in practice. In this paper, we introduce a novel query-driven task-oriented dialogue system, namely Q-TOD. The essential information from the dialogue context is extracted into a query, which is further employed to retrieve relevant knowledge records for response generation. Firstly, as the query is in the form of natural language and not confined to the schema of the knowledge base, the issue of domain adaption is alleviated remarkably in Q-TOD. Secondly, as the query enables the decoupling of knowledge retrieval from the generation, Q-TOD gets rid of the issue of knowledge base scalability. To evaluate the effectiveness of the proposed Q-TOD, we collect query annotations for three publicly available task-oriented dialogue datasets. Comprehensive experiments verify that Q-TOD outperforms strong baselines and establishes a new state-of-the-art performance on these datasets.

cs.CL↗

arXiv2022

ERNIE-ViL 2.0: Multi-view Contrastive Learning for Image-Text Pre-training

Recent Vision-Language Pre-trained (VLP) models based on dual encoder have attracted extensive attention from academia and industry due to their superior performance on various cross-modal tasks and high computational efficiency. They attempt to learn cross-modal representation using contrastive learning on image-text pairs, however, the built inter-modal correlations only rely on a single view for each modality. Actually, an image or a text contains various potential views, just as humans could capture a real-world scene via diverse descriptions or photos. In this paper, we propose ERNIE-ViL 2.0, a Multi-View Contrastive learning framework to build intra-modal and inter-modal correlations between diverse views simultaneously, aiming at learning a more robust cross-modal representation. Specifically, we construct multiple views within each modality to learn the intra-modal correlation for enhancing the single-modal representation. Besides the inherent visual/textual views, we construct sequences of object tags as a special textual view to narrow the cross-modal semantic gap on noisy image-text pairs. Pre-trained with 29M publicly available datasets, ERNIE-ViL 2.0 achieves competitive results on English cross-modal retrieval. Additionally, to generalize our method to Chinese cross-modal tasks, we train ERNIE-ViL 2.0 through scaling up the pre-training datasets to 1.5B Chinese image-text pairs, resulting in significant improvements compared to previous SOTA results on Chinese cross-modal retrieval. We release our pre-trained models in https://github.com/PaddlePaddle/ERNIE.

cs.CV↗

arXiv2022

Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learning

Artificial neural networks (ANNs) are typically confined to accomplishing pre-defined tasks by learning a set of static parameters. In contrast, biological neural networks (BNNs) can adapt to various new tasks by continually updating the neural connections based on the inputs, which is aligned with the paradigm of learning effective learning rules in addition to static parameters, e.g., meta-learning. Among various biologically inspired learning rules, Hebbian plasticity updates the neural network weights using local signals without the guide of an explicit target function, thus enabling an agent to learn automatically without human efforts. However, typical plastic ANNs using a large amount of meta-parameters violate the nature of the genomics bottleneck and potentially deteriorate the generalization capacity. This work proposes a new learning paradigm decomposing those connection-dependent plasticity rules into neuron-dependent rules thus accommodating $Θ(n^2)$ learnable parameters with only $Θ(n)$ meta-parameters. We also thoroughly study the effect of different neural modulation on plasticity. Our algorithms are tested in challenging random 2D maze environments, where the agents have to use their past experiences to shape the neural connections and improve their performances for the future. The results of our experiment validate the following: 1. Plasticity can be adopted to continually update a randomly initialized RNN to surpass pre-trained, more sophisticated recurrent models, especially when coming to long-term memorization. 2. Following the genomics bottleneck, the proposed decomposed plasticity can be comparable to or even more effective than canonical plasticity rules in some instances.

cs.AI↗

arXiv2022

Imaging the quantum melting of Wigner crystal with electronic quasicrystal order

Wigner crystal, as the most fundamental exemplification where the many-body interaction forges the electrons into a solid, experiences an intriguing quantum melting where diverse intermediate phases are predicted to emerge near the quantum critical point. Indications of exotic Wigner orders like bubble phase, liquid-solid phase, and anisotropic Wigner phase have been established by optical or transport measurements. However, the direct visualization of lattice-scale melting order, which is of paramount importance to unequivocally uncover the melting nature, remains challenging and lacking. Noting that Wigner crystals have been achieved in the fractionally filled moire superlattice recently, here, via scanning tunneling microscope, we image the quantum melting of Wigner solid realized by further varying the moire superstructure in monolayer YbCl3/graphene heterostructure. The Wigner solid is constructed on the two-dimensional ensemble of interfacial electron-hole pairs derived from charge transfer. The interplay between certain moire potential and ionic potential leads to the quantum melting of Wigner solid evidenced by the emergence of electron-liquid characteristic, verifying the theoretical predictions. Particularly, akin to the classical quasicrystal made of atoms, a dodecagonal quasicrystal made of electrons, i.e., the Wigner quasicrytal, is visualized at the quantum melting point. In stark contrast to the incompressible Wigner solid, the Wigner quasicrytal hosts considerable liquefied nature unraveled by the interference ripples caused by scattering. By virtue of the two-dimensional charge transfer interface composed of monolayer heavy electron material and graphene, our discovery not only enriches the exploration and understanding of quantum solid-liquid melting, but also paves the way to directly probe the quantum critical order of correlated many-body system.

cond-mat.str-el↗

arXiv2022

Link the World: Improving Open-domain Conversation with Dynamic Spatiotemporal-aware Knowledge

Making chatbots world aware in a conversation like a human is a crucial challenge, where the world may contain dynamic knowledge and spatiotemporal state. Several recent advances have tried to link the dialog system to a static knowledge base or search engine, but they do not contain all the world information needed for conversations. In contrast, we propose a new method to improve the dialogue system using spatiotemporal aware dynamic knowledge. We utilize service information as a way for the dialogue system to link the world. The system actively builds a request according to the dialog context and spatiotemporal state to get service information and then generates world aware responses. To implement this method, we collect DuSinc, an open-domain human-human dialogue dataset, where a participant can access the service to get the information needed for dialogue responses. Through automatic and human evaluations, we found that service information significantly improves the consistency, informativeness, factuality, and engagingness of the dialogue system, making it behave more like a human. Compared to the pre-trained models without spatiotemporal aware dynamic knowledge, the overall session-level score was improved by 60.87\%. The collection dataset and methods will be open-sourced.

cs.CL↗

arXiv2022

Towards Boosting the Open-Domain Chatbot with Human Feedback

Many open-domain dialogue models pre-trained with social media comments can generate coherent replies but have difficulties producing engaging responses when interacting with real users. This phenomenon might mainly result from the deficiency of annotated human-human conversations and the misalignment with human preference. In this paper, we propose a novel and efficient approach Diamante to boost the open-domain chatbot, where two kinds of human feedback (including explicit demonstration and implicit preference) are collected and leveraged. By asking annotators to select or amend the model-generated candidate responses, Diamante efficiently collects the human demonstrated responses and constructs a Chinese chit-chat dataset. To enhance the alignment with human preference, Diamante leverages the implicit preference in the data collection process and introduces the generation-evaluation joint training. Comprehensive experiments indicate that the Diamante dataset and joint training paradigm can significantly boost the performance of Chinese pre-trained dialogue models.

cs.CL↗

arXiv2022

SeSQL: Yet Another Large-scale Session-level Chinese Text-to-SQL Dataset

As the first session-level Chinese dataset, CHASE contains two separate parts, i.e., 2,003 sessions manually constructed from scratch (CHASE-C), and 3,456 sessions translated from English SParC (CHASE-T). We find the two parts are highly discrepant and incompatible as training and evaluation data. In this work, we present SeSQL, yet another large-scale session-level text-to-SQL dataset in Chinese, consisting of 5,028 sessions all manually constructed from scratch. In order to guarantee data quality, we adopt an iterative annotation workflow to facilitate intense and in-time review of previous-round natural language (NL) questions and SQL queries. Moreover, by completing all context-dependent NL questions, we obtain 27,012 context-independent question/SQL pairs, allowing SeSQL to be used as the largest dataset for single-round multi-DB text-to-SQL parsing. We conduct benchmark session-level text-to-SQL parsing experiments on SeSQL by employing three competitive session-level parsers, and present detailed analysis.

cs.CL↗

arXiv2022

Spin-Orbital States and Strong Antiferromagnetism of Layered Eu$_2$SrFe$_2$O$_6$ and Sr$_3$Fe$_2$O$_4$Cl$_2$

The insulating iron compounds Eu$_2$SrFe$_2$O$_6$ and Sr$_3$Fe$_2$O$_4$Cl$_2$ have high-temperature antiferromagnetic (AF) order despite their different layered structures. Here we carry out density functional calculations and Monte Carlo simulations to study their electronic structures and magnetic properties aided with analyses of the crystal field, magnetic anisotropy, and superexchange. We find that both compounds are Mott insulators and in the high-spin (HS) Fe$^{2+}$ state ($S$ = 2) accompanied by the weakened crystal field. Although they have different local coordination and crystal fields, the Fe$^{2+}$ ions have the same level sequence and ground-state configuration $(3z^2-r^2)^2(xz,yz)^2(xy)^1(x^2-y^2)^1$. Then, the multiorbital superexchange produces strong AF couplings, and the $(3z^2-r^2)/(xz,yz)$ mixing via the spin-orbit coupling (SOC) yields a small in-plane orbital moment and anisotropy. Indeed, by tracing a set of different spin-orbital states, our density functional calculations confirm the strong AF couplings and the easy planar magnetization for both compounds. Moreover, using the derived magnetic parameters, our Monte Carlo simulations give the Néel temperature $T_{\rm N}$ = 420 K (372 K) for the former (the latter), which well reproduce the experimental results. Therefore, the present study provides a unified picture for Eu$_2$SrFe$_2$O$_6$ and Sr$_3$Fe$_2$O$_4$Cl$_2$ concerning their electronic and magnetic properties.

cond-mat.str-el↗

arXiv2022

Interaction Driven Topological Phase Transition in Monolayer CrCl$_2$(pyrazine)$_2$

The quadratic band crossing points (QBCPs) at Fermi level in two-dimension have been proposed to be unstable under electron-electron interaction. The possible interaction driven states include quantum anomalous Hall (QAH) state and various nematic ordered states. In this work, motivated by the discovery of ferromagnetic van der Waals layered metal-organic framework CrCl$_2$(pyrazine)$_2$, we theoretically propose that the single layer of CrCl$_2$(pyrazine)$_2$ might realize one or some of these interaction driven states based on the QBCP protected by $C_4$ symmetry. By introducing the short-range density-density type repulsion interactions into this system, we have found the phase diagram depending on different interaction range and strength. The exotic phases include the staggered chiral flux state manifesting QAH effect, the site-nematic insulator and the site-nematic Dirac semimetal state. The QAH state is robust against perturbations breaking the QBCP but it is weakened by increasing temperature. The metal-organic framework is tunable by changing the transition-metal elements, which might improve the gap size and stability of this interaction induced QAH state.

cond-mat.mtrl-sci↗

arXiv2022

An Interpretability Evaluation Benchmark for Pre-trained Language Models

While pre-trained language models (LMs) have brought great improvements in many NLP tasks, there is increasing attention to explore capabilities of LMs and interpret their predictions. However, existing works usually focus only on a certain capability with some downstream tasks. There is a lack of datasets for directly evaluating the masked word prediction performance and the interpretability of pre-trained LMs. To fill in the gap, we propose a novel evaluation benchmark providing with both English and Chinese annotated data. It tests LMs abilities in multiple dimensions, i.e., grammar, semantics, knowledge, reasoning and computation. In addition, it provides carefully annotated token-level rationales that satisfy sufficiency and compactness. It contains perturbed instances for each original instance, so as to use the rationale consistency under perturbations as the metric for faithfulness, a perspective of interpretability. We conduct experiments on several widely-used pre-trained LMs. The results show that they perform very poorly on the dimensions of knowledge and computation. And their plausibility in all dimensions is far from satisfactory, especially when the rationale is short. In addition, the pre-trained LMs we evaluated are not robust on syntax-aware data. We will release this evaluation benchmark at \url{http://xyz}, and hope it can facilitate the research progress of pre-trained LMs.

cs.CL↗

arXiv2022

Bi-SimCut: A Simple Strategy for Boosting Neural Machine Translation

We introduce Bi-SimCut: a simple but effective training strategy to boost neural machine translation (NMT) performance. It consists of two procedures: bidirectional pretraining and unidirectional finetuning. Both procedures utilize SimCut, a simple regularization method that forces the consistency between the output distributions of the original and the cutoff sentence pairs. Without leveraging extra dataset via back-translation or integrating large-scale pretrained model, Bi-SimCut achieves strong translation performance across five translation benchmarks (data sizes range from 160K to 20.2M): BLEU scores of 31.16 for en -> de and 38.37 for de -> en on the IWSLT14 dataset, 30.78 for en -> de and 35.15 for de -> en on the WMT14 dataset, and 27.17 for zh -> en on the WMT17 dataset. SimCut is not a new method, but a version of Cutoff (Shen et al., 2020) simplified and adapted for NMT, and it could be considered as a perturbation-based method. Given the universality and simplicity of SimCut and Bi-SimCut, we believe they can serve as strong baselines for future NMT research.

cs.CL↗

arXiv2022

Unique electronic state in ferromagnetic semiconductor FeCl$_{2}$ monolayer

Two-dimensional (2D) van der Waals (vdW) magnetic materials could be an ideal platform for ultracompact spintronic applications. Among them, FeCl$_{2}$ monolayer in the triangular lattice is subject to a strong debate. Thus, we critically examine its spin-orbital state, electronic structure, and magnetic properties, using a set of delicate first-principles calculations, crystal field level analyses, and Monte Carlo simulations. Our work reveals that FeCl$_{2}$ monolayer is a ferromagnetic (FM) semiconductor in which the electron correlation of the narrow Fe $3d$ bands determines the band gap of about 1.2 eV. Note that only when the spin-orbit coupling (SOC) is properly handled, the unique $d$$^{5\uparrow}$$l$$^\downarrow_{z+}$ electronic ground state is achieved. Then, both the orbital and spin contributions (0.59 $μ_{\rm B}$ plus 3.56 $μ_{\rm B}$) to the total magnetic moment well account for, for the first time, the experimental perpendicular moment of 4.3 $μ_{\rm B}$/Fe. Moreover, we find that a compressive strain further stabilizes the $d$$^{5\uparrow}$$l$$^\downarrow_{z+}$ ground state, and that the enhanced magnetic anisotropy and exchange coupling would boost the Curie temperature ($T_{\rm C}$) from 25 K for the pristine FeCl$_{2}$ monolayer to 69-102 K under 3$\%$-5$\%$ compressive strain. Therefore, FeCl$_{2}$ monolayer is indeed an appealing 2D FM semiconductor.

cond-mat.mtrl-sci↗

arXiv2022

DuQM: A Chinese Dataset of Linguistically Perturbed Natural Questions for Evaluating the Robustness of Question Matching Models

In this paper, we focus on studying robustness evaluation of Chinese question matching. Most of the previous work on analyzing robustness issue focus on just one or a few types of artificial adversarial examples. Instead, we argue that it is necessary to formulate a comprehensive evaluation about the linguistic capabilities of models on natural texts. For this purpose, we create a Chinese dataset namely DuQM which contains natural questions with linguistic perturbations to evaluate the robustness of question matching models. DuQM contains 3 categories and 13 subcategories with 32 linguistic perturbations. The extensive experiments demonstrate that DuQM has a better ability to distinguish different models. Importantly, the detailed breakdown of evaluation by linguistic phenomenon in DuQM helps us easily diagnose the strength and weakness of different models. Additionally, our experiment results show that the effect of artificial adversarial examples does not work on the natural texts.

cs.CL↗

arXiv2022

ERNIE-Search: Bridging Cross-Encoder with Dual-Encoder via Self On-the-fly Distillation for Dense Passage Retrieval

Neural retrievers based on pre-trained language models (PLMs), such as dual-encoders, have achieved promising performance on the task of open-domain question answering (QA). Their effectiveness can further reach new state-of-the-arts by incorporating cross-architecture knowledge distillation. However, most of the existing studies just directly apply conventional distillation methods. They fail to consider the particular situation where the teacher and student have different structures. In this paper, we propose a novel distillation method that significantly advances cross-architecture distillation for dual-encoders. Our method 1) introduces a self on-the-fly distillation method that can effectively distill late interaction (i.e., ColBERT) to vanilla dual-encoder, and 2) incorporates a cascade distillation process to further improve the performance with a cross-encoder teacher. Extensive experiments are conducted to validate that our proposed solution outperforms strong baselines and establish a new state-of-the-art on open-domain QA benchmarks.

cs.CL↗