arXiv subjects

Pei Zhou

Publications and source records attributed to Pei Zhou.

50 records · Page 3Linked to original sources

arXiv2021

Commonsense-Focused Dialogues for Response Generation: An Empirical Study

Smooth and effective communication requires the ability to perform latent or explicit commonsense inference. Prior commonsense reasoning benchmarks (such as SocialIQA and CommonsenseQA) mainly focus on the discriminative task of choosing the right answer from a set of candidates, and do not involve interactive language generation as in dialogue. Moreover, existing dialogue datasets do not explicitly focus on exhibiting commonsense as a facet. In this paper, we present an empirical study of commonsense in dialogue response generation. We first auto-extract commonsensical dialogues from existing dialogue datasets by leveraging ConceptNet, a commonsense knowledge graph. Furthermore, building on social contexts/situations in SocialIQA, we collect a new dialogue dataset with 25K dialogues aimed at exhibiting social commonsense in an interactive setting. We evaluate response generation models trained using these datasets and find that models trained on both extracted and our collected data produce responses that consistently exhibit more commonsense than baselines. Finally we propose an approach for automatic evaluation of commonsense that relies on features derived from ConceptNet and pre-trained language and dialog models, and show reasonable correlation with human evaluation of responses' commonsense quality. We are releasing a subset of our collected data, Commonsense-Dialogues, containing about 11K dialogs.

cs.CL↗

arXiv2021

RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms

Pre-trained language models (PTLMs) have achieved impressive performance on commonsense inference benchmarks, but their ability to employ commonsense to make robust inferences, which is crucial for effective communications with humans, is debated. In the pursuit of advancing fluid human-AI communication, we propose a new challenge, RICA: Robust Inference capability based on Commonsense Axioms, that evaluates robust commonsense inference despite textual perturbations. To generate data for this challenge, we develop a systematic and scalable procedure using commonsense knowledge bases and probe PTLMs across two different evaluation settings. Extensive experiments on our generated probe sets with more than 10k statements show that PTLMs perform no better than random guessing on the zero-shot setting, are heavily impacted by statistical biases, and are not robust to perturbation attacks. We also find that fine-tuning on similar statements offer limited gains, as PTLMs still fail to generalize to unseen inferences. Our new large-scale benchmark exposes a significant gap between PTLMs and human-level language understanding and offers a new challenge for PTLMs to demonstrate commonsense.

cs.CL↗

arXiv2021

Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources

Warning: this paper contains content that may be offensive or upsetting. Numerous natural language processing models have tried injecting commonsense by using the ConceptNet knowledge base to improve performance on different tasks. ConceptNet, however, is mostly crowdsourced from humans and may reflect human biases such as "lawyers are dishonest." It is important that these biases are not conflated with the notion of commonsense. We study this missing yet important problem by first defining and quantifying biases in ConceptNet as two types of representational harms: overgeneralization of polarized perceptions and representation disparity. We find that ConceptNet contains severe biases and disparities across four demographic categories. In addition, we analyze two downstream models that use ConceptNet as a source for commonsense knowledge and find the existence of biases in those models as well. We further propose a filtered-based bias-mitigation approach and examine its effectiveness. We show that our mitigation approach can reduce the issues in both resource and models but leads to a performance drop, leaving room for future work to build fairer and stronger commonsense models.

cs.CL↗

arXiv2021

Probing Commonsense Explanation in Dialogue Response Generation

Humans use commonsense reasoning (CSR) implicitly to produce natural and coherent responses in conversations. Aiming to close the gap between current response generation (RG) models and human communication abilities, we want to understand why RG models respond as they do by probing RG model's understanding of commonsense reasoning that elicits proper responses. We formalize the problem by framing commonsense as a latent variable in the RG task and using explanations for responses as textual form of commonsense. We collect 6k annotated explanations justifying responses from four dialogue datasets and ask humans to verify them and propose two probing settings to evaluate RG models' CSR capabilities. Probing results show that models fail to capture the logical relations between commonsense explanations and responses and fine-tuning on in-domain data and increasing model sizes do not lead to understanding of CSR for RG. We hope our study motivates more research in making RG models emulate the human reasoning process in pursuit of smooth human-AI communication.

cs.CL↗

arXiv2021

An RF-source-free microwave photonic radar with an optically injected semiconductor laser for high-resolution detection and imaging

This paper presents a novel microwave photonic (MWP) radar scheme that is capable of optically generating and processing broadband linear frequency-modulated (LFM) microwave signals without using any radio-frequency (RF) sources. In the transmitter, a broadband LFM microwave signal is generated by controlling the period-one (P1) oscillation of an optically injected semiconductor laser. After targets reflection, photonic de-chirping is implemented based on a dual-drive Mach-Zehnder modulator (DMZM), which is followed by a low-speed analog-to-digital converter (ADC) and digital signal processer (DSP) to reconstruct target information. Without the limitations of external RF sources, the proposed radar has an ultra-flexible tunability, and the main operating parameters are adjustable, including central frequency, bandwidth, frequency band, and temporal period. In the experiment, a fully photonics-based Ku-band radar with a bandwidth of 4 GHz is established for high-resolution detection and inverse synthetic aperture radar (ISAR) imaging. Results show that a high range resolution reaching ~1.88 cm, and a two-dimensional (2D) imaging resolution as high as ~1.88 cm x ~2.00 cm are achieved with a sampling rate of 100 MSa/s in the receiver. The flexible tunability of the radar is also experimentally investigated. The proposed radar scheme features low cost, simple structure, and high reconfigurability, which, hopefully, is to be used in future multifunction adaptive and miniaturized radars.

eess.SP↗

arXiv2021

Incorporating Commonsense Knowledge Graph in Pretrained Models for Social Commonsense Tasks

Pretrained language models have excelled at many NLP tasks recently; however, their social intelligence is still unsatisfactory. To enable this, machines need to have a more general understanding of our complicated world and develop the ability to perform commonsense reasoning besides fitting the specific downstream tasks. External commonsense knowledge graphs (KGs), such as ConceptNet, provide rich information about words and their relationships. Thus, towards general commonsense learning, we propose two approaches to \emph{implicitly} and \emph{explicitly} infuse such KGs into pretrained language models. We demonstrate our proposed methods perform well on SocialIQA, a social commonsense reasoning task, in both limited and full training data regimes.

cs.CL↗

arXiv2021

Go Beyond Plain Fine-tuning: Improving Pretrained Models for Social Commonsense

Pretrained language models have demonstrated outstanding performance in many NLP tasks recently. However, their social intelligence, which requires commonsense reasoning about the current situation and mental states of others, is still developing. Towards improving language models' social intelligence, we focus on the Social IQA dataset, a task requiring social and emotional commonsense reasoning. Building on top of the pretrained RoBERTa and GPT2 models, we propose several architecture variations and extensions, as well as leveraging external commonsense corpora, to optimize the model for Social IQA. Our proposed system achieves competitive results as those top-ranking models on the leaderboard. This work demonstrates the strengths of pretrained language models, and provides viable ways to improve their performance for a particular task.

cs.CL↗

arXiv2020

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

Recently, large-scale pre-trained language models have demonstrated impressive performance on several commonsense-reasoning benchmark datasets. However, building machines with commonsense to compose realistically plausible sentences remains challenging. In this paper, we present a constrained text generation task, CommonGen associated with a benchmark dataset, to explicitly test machines for the ability of generative commonsense reasoning. Given a set of common concepts (e.g., {dog, frisbee, catch, throw}); the task is to generate a coherent sentence describing an everyday scenario using these concepts (e.g., "a man throws a frisbee and his dog catches it"). The CommonGen task is challenging because it inherently requires 1) relational reasoning with background commonsense knowledge, and 2) compositional generalization ability to work on unseen concept combinations. Our dataset, constructed through a combination of crowdsourced and existing caption corpora, consists of 79k commonsense descriptions over 35k unique concept-sets. Experiments show that there is a large gap between state-of-the-art text generation models (e.g., T5) and human performance. Furthermore, we demonstrate that the learned generative commonsense reasoning capability can be transferred to improve downstream tasks such as CommonsenseQA by generating additional context.

cs.CL↗

arXiv2019

Retrofitting Contextualized Word Embeddings with Paraphrases

Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is paraphrased. As a result, the downstream model is not robust to paraphrasing and other linguistic variations. To enhance the stability of contextualized word embedding models, we propose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transformation on the input space, which seeks to minimize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classification and language inference tasks.

cs.CL↗

arXiv2019

Examining Gender Bias in Languages with Grammatical Gender

Recent studies have shown that word embeddings exhibit gender bias inherited from the training corpora. However, most studies to date have focused on quantifying and mitigating such bias only in English. These analyses cannot be directly extended to languages that exhibit morphological agreement on gender, such as Spanish and French. In this paper, we propose new metrics for evaluating gender bias in word embeddings of these languages and further demonstrate evidence of gender bias in bilingual embeddings which align these languages with English. Finally, we extend an existing approach to mitigate gender bias in word embeddings under both monolingual and bilingual settings. Experiments on modified Word Embedding Association Test, word similarity, word translation, and word pair translation tasks show that the proposed approaches effectively reduce the gender bias while preserving the utility of the embeddings.

cs.CL↗

arXiv2018

Quantification and Analysis of Scientific Language Variation Across Research Fields

Quantifying differences in terminologies from various academic domains has been a longstanding problem yet to be solved. We propose a computational approach for analyzing linguistic variation among scientific research fields by capturing the semantic change of terms based on a neural language model. The model is trained on a large collection of literature in five computer science research fields, for which we obtain field-specific vector representations for key terms, and global vector representations for other words. Several quantitative approaches are introduced to identify the terms whose semantics have drastically changed, or remain unchanged across different research fields. We also propose a metric to quantify the overall linguistic variation of research fields. After quantitative evaluation on human annotated data and qualitative comparison with other methods, we show that our model can improve cross-disciplinary data collaboration by identifying terms that potentially induce confusion during interdisciplinary studies.

cs.CL↗

arXiv2018

IEEE 802.11ay based mmWave WLANs: Design Challenges and Solutions

Millimeter-wave (mmWave) with large spectrum available is considered as the most promising frequency band for future wireless communications. The IEEE 802.11ad and IEEE 802.11ay operating on 60 GHz mmWave are the two most expected wireless local area network (WLAN) technologies for ultra-high-speed communications. For the IEEE 802.11ay standard still under development, there are plenty of proposals from companies and researchers who are involved with the IEEE 802.11ay task group. In this survey, we conduct a comprehensive review on the medium access control layer (MAC) related issues for the IEEE 802.11ay, some cross-layer between physical layer (PHY) and MAC technologies are also included. We start with MAC related technologies in the IEEE 802.11ad and discuss design challenges on mmWave communications, leading to some MAC related technologies for the IEEE 802.11ay. We then elaborate on important design issues for IEEE 802.11ay. Specifically, we review the channel bonding and aggregation for the IEEE 802.11ay, and point out the major differences between the two technologies. Then, we describe channel access and channel allocation in the IEEE 802.11ay, including spatial sharing and interference mitigation technologies. After that, we present an in-depth survey on beamforming training (BFT), beam tracking, single-user multiple-input-multiple-output (SU-MIMO) beamforming and multi-user multiple-input-multiple-output (MU-MIMO) beamforming. Finally, we discuss some open design issues and future research directions for mmWave WLANs. We hope that this paper provides a good introduction to this exciting research area for future wireless systems.

cs.NI↗

arXiv2017

Throughput and Robustness Guaranteed Beam Tracking for mmWave Wireless Networks

With the increasing demand of ultra-high-speed wireless communications and the existing low frequency band (e.g., sub-6GHz) becomes more and more crowded, millimeter-wave (mmWave) with large spectra available is considered as the most promising frequency band for future wireless communications. Since the mmWave suffers a serious path-loss, beamforming techniques shall be adopted to concentrate the transmit power and receive region on a narrow beam for achieving long distance communications. However, the mobility of users will bring frequent beam handoff, which will decrease the quality of experience (QoE). Therefore, efficient beam tracking mechanism should be carefully researched. However, the existing beam tracking mechanisms concentrate on system throughput maximization without considering beam handoff and link robustness. This paper proposes a throughput and robustness guaranteed beam tracking mechanism for mobile mmWave communication systems which takes account of both system throughput and handoff probability. Simulation results show that the proposed throughput and robustness guaranteed beam tracking mechanism can provide better performance than the other beam tracking mechanisms.

cs.NI↗

arXiv2017

Enhanced Random Access and Beam Training for mmWave Wireless Local Networks with High User Density

As low frequency band becomes more and more crowded, millimeter-wave (mmWave) has attracted significant attention recently. IEEE has released the 802.11ad standard to satisfy the demand of ultra-high-speed communication. It adopts beamforming technology that can generate directional beams to compensate for high path loss. In the Association Beamforming Training (A-BFT) phase of beamforming (BF) training, a station (STA) randomly selects an A-BFT slot to contend for training opportunity. Due to the limited number of A-BFT slots, A-BFT phase suffers high probability of collisions in dense user scenarios, resulting in inefficient training performance. Based on the evaluation of the IEEE 802.11ad standard and 802.11ay draft in dense user scenarios of mmWave wireless networks, we propose an enhanced A-BFT beam training and random access mechanism, including the Separated A-BFT (SA-BFT) and Secondary Backoff A-BFT (SBA-BFT). The SA-BFT can provide more A-BFT slots and divide A-BFT slots into two regions by defining a new `E-A-BFT Length' field compared to the legacy 802.11ad A-BFT, thereby maintaining compatibility when 802.11ay devices are mixed with 802.11ad devices. It can also reduce the collision probability in dense user scenarios greatly. The SBA-BFT performs secondary backoff with very small overhead of transmission opportunities within one A-BFT slot, which not only further reduces collision probability, but also improves the A-BFT slots utilization. Furthermore, we propose a three-dimensional Markov model to analyze the performance of the SBA-BFT. The analytical and simulation results show that both the SA-BFT and the SBA-BFT can significantly improve BF training efficiency, which are beneficial to the optimization design of dense user wireless networks based on the IEEE 802.11ay standard and mmWave technology.

cs.NI↗