arXiv subjects

Peng Shi

Publications and source records attributed to Peng Shi.

At least 91 records · Page 5Linked to original sources

arXiv2022

XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing

In-context learning using large language models has recently shown surprising results for semantic parsing tasks such as Text-to-SQL translation. Prompting GPT-3 or Codex using several examples of question-SQL pairs can produce excellent results, comparable to state-of-the-art finetuning-based models. However, existing work primarily focuses on English datasets, and it is unknown whether large language models can serve as competitive semantic parsers for other languages. To bridge this gap, our work focuses on cross-lingual Text-to-SQL semantic parsing for translating non-English utterances into SQL queries based on an English schema. We consider a zero-shot transfer learning setting with the assumption that we do not have any labeled examples in the target language (but have annotated examples in English). This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query to construct prompts. We also include global translation exemplars for a target language to facilitate the translation process for large language models. To systematically evaluate our model, we construct two new benchmark datasets, XSpider and XKaggle-dbqa, which include questions in Chinese, Vietnamese, Farsi, and Hindi. Our experiments show that XRICL effectively leverages large pre-trained language models to outperform existing baselines. Data and code are publicly available at https://github.com/Impavidity/XRICL.

cs.CL↗

arXiv2022

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models

Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases. Since the inputs and outputs of SKG tasks are heterogeneous, they have been studied separately by different communities, which limits systematic and compatible research on SKG. In this paper, we overcome this limitation by proposing the UnifiedSKG framework, which unifies 21 SKG tasks into a text-to-text format, aiming to promote systematic SKG research, instead of being exclusive to a single task, domain, or dataset. We use UnifiedSKG to benchmark T5 with different sizes and show that T5, with simple modifications when necessary, achieves state-of-the-art performance on almost all of the 21 tasks. We further demonstrate that multi-task prefix-tuning improves the performance on most tasks, largely improving the overall performance. UnifiedSKG also facilitates the investigation of zero-shot and few-shot learning, and we show that T0, GPT-3, and Codex struggle in zero-shot and few-shot learning for SKG. We also use UnifiedSKG to conduct a series of controlled experiments on structured knowledge encoding variants across SKG tasks. UnifiedSKG is easily extensible to more tasks, and it is open-sourced at https://github.com/hkunlp/unifiedskg.

cs.CL↗

arXiv2022

Better Language Model with Hypernym Class Prediction

Class-based language models (LMs) have been long devised to address context sparsity in $n$-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and Arxiv. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.

cs.CL↗

arXiv2022

Intrinsic spin-momentum dynamics of surface electromagnetic waves in complex dispersive system

Spin-momentum locking is an intrinsic property of surface electromagnetic fields and its study has led to the discovery of photonic spin lattices and diverse applications. Previously, dispersion was ignored in the spin-momentum locking, giving rise to abnormal phenomena contradictory to the physical realities. Here, we formulate four dispersive spin-momentum equations for surface waves, revealing universally that the transverse spin vector is locked with the momentum. The locking property obeys the right-hand rule in the dielectric but the left-hand rule in the dispersive metal/magnetic materials. In addition to the dispersion, the structural features can affect the spin-momentum locking significantly. Remarkably, an extraordinary longitudinal spin originating from the coupling polarization ellipticity is uncovered even for the purely polarized state. We further demonstrate the spin-momentum locking properties with diverse photonic topological lattices by engineering the rotating symmetry. The findings open up opportunities for designing robust nanodevices with practical importance in chiral quantum optics.

physics.optics↗

arXiv2022

Solution to Waves in Dissipative Media with Reciprocal Attenuation in Time and Space Domains

The study points out that the traditional solutions to wave equation of dissipative wave and motion equation of block for a multi-degree-of-freedom mass spring damper system are the possible solutions, which are not necessarily objective and conflict each other. The disturbance in discrete system like crystals vibration can be expressed in differential form. A new general solution to dissipative wave equation is proposed with the general Fourier transform. The solution reveals that the attenuation of the disturbance can simultaneously occur in time and space domains. Then the general solution is used in case studies to analyze the properties of dissipative waves. It is concluded that the properties of waves formulated with the same equation can be different because of the difference of attenuation mechanism.

physics.class-ph↗

arXiv2022

Semi-global Periodic Event-triggered Output Regulation for Nonlinear Multi-agent Systems

This study focuses on periodic event-triggered (PET) cooperative output regulation problem for a class of nonlinear multi-agent systems. The key feature of PET mechanism is that event-triggered conditions are required to be monitored only periodically. This approach is beneficial for Zeno behavior exclusion and saving of battery energy of onboard sensors. At first, new PET distributed observers are proposed to estimate the leader information. We show that the estimation error converges to zero exponentially with a known convergence rate under asynchronous PET communication. Second, a novel PET output feedback controller is designed for the underlying strict feedback nonlinear multi-agent systems. Based on a state transformation technique and a local PET state observer, the cooperative semi-global output regulation problem can be solved by the proposed new control design technique. Simulation results of multiple Lorenz systems illustrate that the developed control scheme is effective.

eess.SY↗

arXiv2021

Mr. TyDi: A Multi-lingual Benchmark for Dense Retrieval

We present Mr. TyDi, a multi-lingual benchmark dataset for mono-lingual retrieval in eleven typologically diverse languages, designed to evaluate ranking with learned dense representations. The goal of this resource is to spur research in dense retrieval techniques in non-English languages, motivated by recent observations that existing techniques for representation learning perform poorly when applied to out-of-distribution data. As a starting point, we provide zero-shot baselines for this new dataset based on a multi-lingual adaptation of DPR that we call "mDPR". Experiments show that although the effectiveness of mDPR is much lower than BM25, dense representations nevertheless appear to provide valuable relevance signals, improving BM25 results in sparse-dense hybrids. In addition to analyses of our results, we also discuss future challenges and present a research agenda in multi-lingual dense retrieval. Mr. TyDi can be downloaded at https://github.com/castorini/mr.tydi.

cs.CL↗

arXiv2021

Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions

Existing text-to-SQL research only considers complete questions as the input, but lay-users might strive to formulate a complete question. To build a smarter natural language interface to database systems (NLIDB) that also processes incomplete questions, we propose a new task, prefix-to-SQL which takes question prefix from users as the input and predicts the intended SQL. We construct a new benchmark called PAGSAS that contains 124K user question prefixes and the intended SQL for 5 sub-tasks Advising, GeoQuery, Scholar, ATIS, and Spider. Additionally, we propose a new metric SAVE to measure how much effort can be saved by users. Experimental results show that PAGSAS is challenging even for strong baseline models such as T5. As we observe the difficulty of prefix-to-SQL is related to the number of omitted tokens, we incorporate curriculum learning of feeding examples with an increasing number of omitted tokens. This improves scores on various sub-tasks by as much as 9% recall scores on sub-task GeoQuery in PAGSAS.

cs.CL↗

arXiv2021

Hierarchical Character Tagger for Short Text Spelling Error Correction

State-of-the-art approaches to spelling error correction problem include Transformer-based Seq2Seq models, which require large training sets and suffer from slow inference time; and sequence labeling models based on Transformer encoders like BERT, which involve token-level label space and therefore a large pre-defined vocabulary dictionary. In this paper we present a Hierarchical Character Tagger model, or HCTagger, for short text spelling error correction. We use a pre-trained language model at the character level as a text encoder, and then predict character-level edits to transform the original text into its error-free form with a much smaller label space. For decoding, we propose a hierarchical multi-task approach to alleviate the issue of long-tail label distribution without introducing extra model parameters. Experiments on two public misspelling correction datasets demonstrate that HCTagger is an accurate and much faster approach than many existing models.

cs.CL↗

arXiv2021

Cross-Lingual Training with Dense Retrieval for Document Retrieval

Dense retrieval has shown great success in passage ranking in English. However, its effectiveness in document retrieval for non-English languages remains unexplored due to the limitation in training resources. In this work, we explore different transfer techniques for document ranking from English annotations to multiple non-English languages. Our experiments on the test collections in six languages (Chinese, Arabic, French, Hindi, Bengali, Spanish) from diverse language families reveal that zero-shot model-based transfer using mBERT improves the search quality in non-English mono-lingual retrieval. Also, we find that weakly-supervised target language transfer yields competitive performances against the generation-based target language transfer that requires external translators and query generators.

cs.CL↗

arXiv2021

Logic-Consistency Text Generation from Semantic Parses

Text generation from semantic parses is to generate textual descriptions for formal representation inputs such as logic forms and SQL queries. This is challenging due to two reasons: (1) the complex and intensive inner logic with the data scarcity constraint, (2) the lack of automatic evaluation metrics for logic consistency. To address these two challenges, this paper first proposes SNOWBALL, a framework for logic consistent text generation from semantic parses that employs an iterative training procedure by recursively augmenting the training set with quality control. Second, we propose a novel automatic metric, BLEC, for evaluating the logical consistency between the semantic parses and generated texts. The experimental results on two benchmark datasets, Logic2Text and Spider, demonstrate the SNOWBALL framework enhances the logic consistency on both BLEC and human evaluation. Furthermore, our statistical analysis reveals that BLEC is more logically consistent with human evaluation than general-purpose automatic metrics including BLEU, ROUGE and, BLEURT. Our data and code are available at https://github.com/Ciaranshu/relogic.

cs.CL↗

arXiv2021

End-to-End Cross-Domain Text-to-SQL Semantic Parsing with Auxiliary Task

In this work, we focus on two crucial components in the cross-domain text-to-SQL semantic parsing task: schema linking and value filling. To encourage the model to learn better encoding ability, we propose a column selection auxiliary task to empower the encoder with the relevance matching capability by using explicit learning targets. Furthermore, we propose two value filling methods to build the bridge from the existing zero-shot semantic parsers to real-world applications, considering most of the existing parsers ignore the values filling in the synthesized SQL. With experiments on Spider, our proposed framework improves over the baselines on the execution accuracy and exact set match accuracy when database contents are unavailable, and detailed analysis sheds light on future work.

cs.CL↗

arXiv2021

Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2

The semantics of a text is manifested not only by what is read, but also by what is not read. In this article, we will study how the implicit "not read" information such as end-of-paragraph (\eop) and end-of-sequence (\eos) affect the quality of text generation. Specifically, we find that the pre-trained language model GPT2 can generate better continuations by learning to generate the \eop in the fine-tuning stage. Experimental results on English story generation show that \eop can lead to higher BLEU score and lower \eos perplexity. We also conduct experiments on a self-collected Chinese essay dataset with Chinese-GPT2, a character level LM without \eop or \eos during pre-training. Experimental results show that the Chinese GPT2 can generate better essay endings with \eop.

cs.CL↗

arXiv2021

Symmetry-protected photonic chiral spin textures by spin-orbit coupling

Chiral spin textures are researched widely in condensed matter systems and show potential for spintronics and storage applications. Along with extensive condensed-matter studies of chiral spin textures, photonic counterparts of these textures have been observed in various optical systems with broken inversion symmetry. Unfortunately, the resemblances are only phenomenological. This work proposes a theoretical framework based on the variational theorem to show that the formation of photonic chiral spin textures in an optical interface is derived from the system's symmetry and relativity. Analysis of the optical system's rotational symmetry indicates that conservation of the total angular momentum is preserved from the local variations of spin vectors. Specifically, although the integral spin momentum does not carry net energy, the local spin momentum distribution, which determines the local subluminal energy transport and minimization variation of the square of total angular momentum, results in the chiral twisting of the spin vectors. The findings here deepen the understanding of the symmetries, conservative laws and energy transportation in optical system, construct the comparability in the formation mechanisms and geometries of photonic and condensed-matter chiral spin textures, and suggest applications to optical manipulation and chiral photonics.

physics.optics↗

arXiv2021

Photonic spin lattices: symmetry constraints for skyrmion and meron topologies

Symmetry governs many electronic and photonic phenomena in optics and condensed matter physics. Skyrmions and merons are prominent topological structures in magnetic materials, with the topological features determined by the interplay between anisotropy of a material and its magnetization. Here we theoretically show and experimentally demonstrate that the symmetry of the electromagnetic field determines the spin topological properties of the guided modes via spin-orbit coupling and may only result in either hexagonal spin-skyrmion or square spin-meron lattices. We also show that in the absence of spin-orbit coupling these spin topologies are degenerated in dynamic field-skyrmions, unifying description of electromagnetic field topologies. The results provide new understanding of electromagnetic field topology and its transformations as well as new opportunities for applications in quantum optics, spin-optics and topological photonics.

physics.optics↗

arXiv2021

Derivation of Elastic Wave Equation from New Motion Description

In classical mechanics, the motion of an object is described with Newton's three laws of motion, which means that the motion of the material elements composing a continuum can be described with the particle model. However, this viewpoint is not objective, since the existence of transverse wave cannot be predicted by the theory of elasticity based on the particle model. In this paper, the material element of an elastomer is regarded as a rigid body, and the traditional elastic wave equation is derived based on it. In the derivation, the constitutive relations and strain-displacement relations are correspondingly modified. The study reveals that the longitudinal and transverse waves in elastomer correspond to the translational and rotational motion of the material element, respectively. Besides, the reciprocity of shear stress and shear strain is no longer requisite in continuum mechanics, and the local rigid body rotation contributes stress.

physics.class-ph↗

arXiv2020

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train large neural language models with self-supervised learning objectives, such as Masked Language Model (MLM). However, based on a pilot study, we observe three issues of existing general-purpose language models when they are applied to text-to-SQL semantic parsers: fail to detect column mentions in the utterances, fail to infer column mentions from cell values, and fail to compose complex SQL queries. To mitigate these issues, we present a model pre-training framework, Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterances and table schemas by leveraging generation models to generate pre-train data. GAP MODEL is trained on 2M utterance-schema pairs and 30K utterance-schema-SQL triples, whose utterances are produced by generative models. Based on experimental results, neural semantic parsers that leverage GAP MODEL as a representation encoder obtain new state-of-the-art results on both SPIDER and CRITERIA-TO-SQL benchmarks.

cs.CL↗

arXiv2020

Segatron: Segment-Aware Transformer for Language Modeling and Understanding

Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained language models are based on the Transformer architecture. However, it distinguishes sequential tokens only with the token position index. We hypothesize that better contextual representations can be generated from the Transformer with richer positional information. To verify this, we propose a segment-aware Transformer (Segatron), by replacing the original token position encoding with a combined position encoding of paragraph, sentence, and token. We first introduce the segment-aware mechanism to Transformer-XL, which is a popular Transformer-based language model with memory extension and relative position encoding. We find that our method can further improve the Transformer-XL base model and large model, achieving 17.1 perplexity on the WikiText-103 dataset. We further investigate the pre-training masked language modeling task with Segatron. Experimental results show that BERT pre-trained with Segatron (SegaBERT) can outperform BERT with vanilla Transformer on various NLP tasks, and outperforms RoBERTa on zero-shot sentence representation learning.

cs.CL↗