Searcharxiv⌕ Search

arXiv subjects

Lili Wang

Publications and source records attributed to Lili Wang.

At least 91 records · Page 5Linked to original sources
arXiv2021

Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks

This paper studies the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks. Prior research has found that only a few attention heads are important in each mono-lingual Natural Language Processing (NLP) task and pruning the remaining heads leads to comparable or improved performance of the model. However, the impact of pruning attention heads is not yet clear in cross-lingual and multi-lingual tasks. Through extensive experiments, we show that (1) pruning a number of attention heads in a multi-lingual Transformer-based model has, in general, positive effects on its performance in cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned can be ranked using gradients and identified with a few trial experiments. Our experiments focus on sequence labeling tasks, with potential applicability on other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and XLM-R, on three tasks across 9 languages each. We also discuss the validity of our findings and their extensibility to truly resource-scarce languages and other task settings.

cs.CL↗
arXiv2021

The observation of in-plane quantum Griffiths singularity in two-dimensional crystalline superconductors

Quantum Griffiths singularity (QGS) reveals the profound influence of quenched disorder on the quantum phase transitions, characterized by the divergence of the dynamical critical exponent at the boundary of the vortex glass-like phase, named as quantum Griffiths phase. However, in the absence of vortices, whether the QGS can exist under parallel magnetic field remains a puzzle. Here we study the magnetic field induced superconductor-metal transition in ultrathin crystalline PdTe2 films grown by molecular beam epitaxy. Remarkably, the QGS emerges under both perpendicular and parallel magnetic field in 4-monolayer PdTe2 films. The direct activated scaling analysis with a new irrelevant correction has been proposed, providing important evidence of QGS. With increasing film thickness to 6 monolayers, the QGS disappears under perpendicular field but persists under parallel field, and this discordance may originate from the differences in microscopic processes. Our work demonstrates the universality of parallel field induced QGS and can stimulate further investigations on novel quantum phase transitions under parallel magnetic field.

cond-mat.supr-con↗
arXiv2021

Embedding Heterogeneous Networks into Hyperbolic Space Without Meta-path

Networks found in the real-world are numerous and varied. A common type of network is the heterogeneous network, where the nodes (and edges) can be of different types. Accordingly, there have been efforts at learning representations of these heterogeneous networks in low-dimensional space. However, most of the existing heterogeneous network embedding methods suffer from the following two drawbacks: (1) The target space is usually Euclidean. Conversely, many recent works have shown that complex networks may have hyperbolic latent anatomy, which is non-Euclidean. (2) These methods usually rely on meta-paths, which require domain-specific prior knowledge for meta-path selection. Additionally, different down-streaming tasks on the same network might require different meta-paths in order to generate task-specific embeddings. In this paper, we propose a novel self-guided random walk method that does not require meta-path for embedding heterogeneous networks into hyperbolic space. We conduct thorough experiments for the tasks of network reconstruction and link prediction on two public datasets, showing that our model outperforms a variety of well-known baselines across all tasks.

cs.SI↗
arXiv2021

Mitigating Political Bias in Language Models Through Reinforced Calibration

Current large-scale language models can be politically biased as a result of the data they are trained on, potentially causing serious problems when they are deployed in real-world settings. In this paper, we describe metrics for measuring political bias in GPT-2 generation and propose a reinforcement learning (RL) framework for mitigating political biases in generated text. By using rewards from word embeddings or a classifier, our RL framework guides debiased generation without having access to the training data or requiring the model to be retrained. In empirical experiments on three attributes sensitive to political bias (gender, location, and topic), our methods reduced bias according to both our metrics and human evaluation, while maintaining readability and semantic coherence.

cs.CL↗
arXiv2021

Political Depolarization of News Articles Using Attribute-aware Word Embeddings

Political polarization in the US is on the rise. This polarization negatively affects the public sphere by contributing to the creation of ideological echo chambers. In this paper, we focus on addressing one of the factors that contributes to this polarity, polarized media. We introduce a framework for depolarizing news articles. Given an article on a certain topic with a particular ideological slant (eg., liberal or conservative), the framework first detects polar language in the article and then generates a new article with the polar language replaced with neutral expressions. To detect polar words, we train a multi-attribute-aware word embedding model that is aware of ideology and topics on 360k full-length media articles. Then, for text generation, we propose a new algorithm called Text Annealing Depolarization Algorithm (TADA). TADA retrieves neutral expressions from the word embedding model that not only decrease ideological polarity but also preserve the original argument of the text, while maintaining grammatical correctness. We evaluate our framework by comparing the depolarized output of our model in two modes, fully-automatic and semi-automatic, on 99 stories spanning 11 topics. Based on feedback from 161 human testers, our framework successfully depolarized 90.1% of paragraphs in semi-automatic mode and 78.3% of paragraphs in fully-automatic mode. Furthermore, 81.2% of the testers agree that the non-polar content information is well-preserved and 79% agree that depolarization does not harm semantic correctness when they compare the original text and the depolarized text. Our work shows that data-driven methods can help to locate political polarity and aid in the depolarization of articles.

cs.CL↗
arXiv2020

Origin of the Electronic Structure in Single-Layer FeSe/SrTiO3 Films

The accurate theoretical description of the underlying electronic structures is essential for understanding the superconducting mechanism of iron-based superconductors. Compared to bulk FeSe, the superconducting single-layer FeSe/SrTiO3 films exhibit a distinct electronic structure consisting of only electron Fermi pockets, due to the formation of a new band gap at the Brillouin zone (BZ) corners and an indirect band gap between the BZ center and corners. Although intensive studies have been carried out, the origin of such a distinct electronic structure and its connection to bulk FeSe remain unclear. Here we report a systematic study on the temperature evolution of the electronic structure in single-layer FeSe/SrTiO3 films by angle-resolved photoemission spectroscopy. A temperature-induced electronic phase transition was clearly observed at 200 K, above which the electronic structure of single-layer FeSe/SrTiO3 films restored to that of bulk FeSe, characterized by the closing of the new band gap and the vanishing of the indirect band gap. Moreover, the interfacial charge transfer effect induced band shift of ~ 60 meV was determined for the first time. These observations not only show the first direct evidence that the electronic structure of single-layer FeSe/SrTiO3 films originates from bulk FeSe through a combined effect of an electronic phase transition and an interfacial charge transfer, but also provide a quantitative basis for theoretical models in describing the electronic structure and understanding the superconducting mechanism in single-layer FeSe/SrTiO3 films.

cond-mat.supr-con↗
arXiv2020

An Empirical Survey of Unsupervised Text Representation Methods on Twitter Data

The field of NLP has seen unprecedented achievements in recent years. Most notably, with the advent of large-scale pre-trained Transformer-based language models, such as BERT, there has been a noticeable improvement in text representation. It is, however, unclear whether these improvements translate to noisy user-generated text, such as tweets. In this paper, we present an experimental survey of a wide range of well-known text representation techniques for the task of text clustering on noisy Twitter data. Our results indicate that the more advanced models do not necessarily work best on tweets and that more exploration in this area is needed.

cs.CL↗
arXiv2020

Improvements and Extensions on Metaphor Detection

Metaphors are ubiquitous in human language. The metaphor detection task (MD) aims at detecting and interpreting metaphors from written language, which is crucial in natural language understanding (NLU) research. In this paper, we introduce a pre-trained Transformer-based model into MD. Our model outperforms the previous state-of-the-art models by large margins in our evaluations, with relative improvements on the F-1 score from 5.33% to 28.39%. Second, we extend MD to a classification task about the metaphoricity of an entire piece of text to make MD applicable in more general NLU scenes. Finally, we clean up the improper or outdated annotations in one of the MD benchmark datasets and re-benchmark it with our Transformer-based model. This approach could be applied to other existing MD datasets as well, since the metaphoricity annotations in these benchmark datasets may be outdated. Future research efforts are also necessary to build an up-to-date and well-annotated dataset consisting of longer and more complex texts.

cs.CL↗
arXiv2020

Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation

Data augmentation is proven to be effective in many NLU tasks, especially for those suffering from data scarcity. In this paper, we present a powerful and easy to deploy text augmentation framework, Data Boost, which augments data through reinforcement learning guided conditional generation. We evaluate Data Boost on three diverse text classification tasks under five different classifier architectures. The result shows that Data Boost can boost the performance of classifiers especially in low-resource data scenarios. For instance, Data Boost improves F1 for the three tasks by 8.7% on average when given only 10% of the whole data for training. We also compare Data Boost with six prior text augmentation methods. Through human evaluations (N=178), we confirm that Data Boost augmentation has comparable quality as the original data with respect to readability and class consistency.

cs.CL↗
arXiv2020

Embedding Node Structural Role Identity into Hyperbolic Space

Recently, there has been an interest in embedding networks in hyperbolic space, since hyperbolic space has been shown to work well in capturing graph/network structure as it can naturally reflect some properties of complex networks. However, the work on network embedding in hyperbolic space has been focused on microscopic node embedding. In this work, we are the first to present a framework to embed the structural roles of nodes into hyperbolic space. Our framework extends struct2vec, a well-known structural role preserving embedding method, by moving it to a hyperboloid model. We evaluated our method on four real-world and one synthetic network. Our results show that hyperbolic space is more effective than euclidean space in learning latent representations for the structural role of nodes.

cs.SI↗
arXiv2020

Towards Improved Model Design for Authorship Identification: A Survey on Writing Style Understanding

Authorship identification tasks, which rely heavily on linguistic styles, have always been an important part of Natural Language Understanding (NLU) research. While other tasks based on linguistic style understanding benefit from deep learning methods, these methods have not behaved as well as traditional machine learning methods in many authorship-based tasks. With these tasks becoming more and more challenging, however, traditional machine learning methods based on handcrafted feature sets are already approaching their performance limits. Thus, in order to inspire future applications of deep learning methods in authorship-based tasks in ways that benefit the extraction of stylistic features, we survey authorship-based tasks and other tasks related to writing style understanding. We first describe our survey results on the current state of research in both sets of tasks and summarize existing achievements and problems in authorship-related tasks. We then describe outstanding methods in style-related tasks in general and analyze how they are used in combination in the top-performing models. We are optimistic about the applicability of these models to authorship-based tasks and hope our survey will help advance research in this field.

cs.CL↗
arXiv2020

Emoji Prediction: Extensions and Benchmarking

Emojis are a succinct form of language which can express concrete meanings, emotions, and intentions. Emojis also carry signals that can be used to better understand communicative intent. They have become a ubiquitous part of our daily lives, making them an important part of understanding user-generated content. The emoji prediction task aims at predicting the proper set of emojis associated with a piece of text. Through emoji prediction, models can learn rich representations of the communicative intent of the written text. While existing research on the emoji prediction task focus on a small subset of emoji types closely related to certain emotions, this setting oversimplifies the task and wastes the expressive power of emojis. In this paper, we extend the existing setting of the emoji prediction task to include a richer set of emojis and to allow multi-label classification on the task. We propose novel models for multi-class and multi-label emoji prediction based on Transformer networks. We also construct multiple emoji prediction datasets from Twitter using heuristics. The BERT models achieve state-of-the-art performances on all our datasets under all the settings, with relative improvements of 27.21% to 236.36% in accuracy, 2.01% to 88.28% in top-5 accuracy and 65.19% to 346.79% in F-1 score, compared to the prior state-of-the-art. Our results demonstrate the efficacy of deep Transformer-based models on the emoji prediction task. We also release our datasets at https://github.com/hikari-NYU/Emoji_Prediction_Datasets_MMS for future researchers.

cs.CL↗
arXiv2020

Type-II Ising superconductivity and anomalous metallic state in macro-size ambient-stable ultrathin crystalline films

Recent emergence of two-dimensional (2D) crystalline superconductors has provided a promising platform to investigate novel quantum physics and potential applications. To reveal essential quantum phenomena therein, ultralow temperature transport investigation on high quality ultrathin superconducting films is critically required, although it has been quite challenging experimentally. Here we report a systematic transport study on the ultrathin crystalline PdTe2 films grown by molecular beam epitaxy (MBE). Interestingly, a new type of Ising superconductivity in 2D centrosymmetric materials is revealed by the detection of large in-plane critical field more than 7 times Pauli limit. Remarkably, in perpendicular magnetic field, we provide solid evidence of anomalous metallic state characterized by the resistance saturation at low temperatures with high quality filters. The robust superconductivity with intriguing quantum phenomena in the macro-size ambient-stable ultrathin PdTe2 films remains almost the same for 20 months, showing great potentials in electronic and spintronic applications.

cond-mat.supr-con↗
arXiv2020

Salienteye: Maximizing Engagement While Maintaining Artistic Style on Instagram Using Deep Neural Networks

Instagram has become a great venue for amateur and professional photographers alike to showcase their work. It has, in other words, democratized photography. Generally, photographers take thousands of photos in a session, from which they pick a few to showcase their work on Instagram. Photographers trying to build a reputation on Instagram have to strike a balance between maximizing their followers' engagement with their photos, while also maintaining their artistic style. We used transfer learning to adapt Xception, which is a model for object recognition trained on the ImageNet dataset, to the task of engagement prediction and utilized Gram matrices generated from VGG19, another object recognition model trained on ImageNet, for the task of style similarity measurement on photos posted on Instagram. Our models can be trained on individual Instagram accounts to create personalized engagement prediction and style similarity models. Once trained on their accounts, users can have new photos sorted based on predicted engagement and style similarity to their previous work, thus enabling them to upload photos that not only have the potential to maximize engagement from their followers but also maintain their style of photography. We trained and validated our models on several Instagram accounts, showing it to be adept at both tasks, also outperforming several baseline models and human annotators.

cs.CV↗
arXiv2020

Towards defining reference materials for extracellular vesicle size, concentration, refractive index and epitope abundance

Accurate characterization of extracellular vesicles (EVs) is critical to explore their diagnostic and therapeutic applications. As the EV research field has developed, so too have the techniques used to characterize them. The development of reference materials is required for the standardization of these techniques. This work, initiated from the ISEV 2017 Biomarker Workshop in Birmingham, UK, and with further discussion during the ISEV 2019 Standardization Workshop in Ghent, Belgium, sets out to elucidate which reference materials are required and which are currently available to standardize commonly used analysis platforms for characterizing EV size, concentration, refractive index, and epitope expression. Due to their predominant use, a particular focus is placed on the optical methods nanoparticle tracking analysis and flow cytometry.

q-bio.QM↗
arXiv2020

A Distributed Observer for a Continuous-Time Linear System with time-varying network

A simply structured distributed observer is described for estimating the state of a continuous-time, jointly observable, input-free, linear system whose sensed outputs are distributed across a time-varying network. It is explained how to design a gain $g$ in the observer so that their state estimation errors all converge exponentially fast to zero at a fixed, but arbitrarily chosen rate provided the network's graph is strongly connected for all time. A linear inequality for $g$ is provided when the network's graph is switching according to a switching signal with a dwell time or an average dwell time, respectively. It has also been shown the existence of $g$ when the stochastic matrix of the network's graph is chosen to be doubly stochastic under arbitrarily switching signals. This is accomplished by exploiting several well-known properties of invariant subspaces and properties of perturbed systems.

eess.SY↗
arXiv2019

A curvature notion for planar graphs stable under planar duality

Woess \cite{Woess98} introduced a curvature notion on the set of edges of a planar graph, called $Ψ$-curvature in our paper, which is stable under the planar duality. We study geometric and combinatorial properties for the class of infinite planar graphs with non-negative $Ψ$-curvature. By using the discharging method, we prove that for such an infinite graph the number of vertices (resp. faces) of degree $k,$ except $k=3,4$ or $6,$ is finite. As a main result, we prove that for an infinite planar graph with non-negative $Ψ$-curvature the sum of the number of vertices of degree at least $8$ and the number of faces of degree at least $8$ is at most one.

math.CO↗