arXiv subjects

Ke Xu

Publications and source records attributed to Ke Xu.

At least 343 records · Page 19Linked to original sources

arXiv2021

Photoluminescence switching in a two-dimensional atomic crystal

Two-dimensional materials are an emerging class of new materials with a wide range of electrical and optical properties and potential applications. Single-layer structures of semiconducting transition metal dichalcogenides are gaining increasing attention for use in field-effect transistors. Here, we report a photoluminescence switching effect based on single-layer WSe2 transistors. Dual gates are used to tune the photoluminescence intensity. In particular, a side-gate is utilized to control the location of ions within a solid polymer electrolyte to form an electric double layer at the interface of electrolyte and WSe2 and induce a vertical electric field. Additionally, a back-gate is used to apply a 2nd vertical electric field. An on-off ratio of the light emission up to 90 was observed under constant pump light intensity. In addition, a blue shift of the photoluminescence line up to 36 meV was observed. We attribute this blue shift to the decrease of exciton binding energy due to the change of nonlinear in-plane dielectric constant and use it to determine the 3rd order off-diagonal susceptibility \c{hi}^((3) )=3.50*10^(-19)m2/V2.

cond-mat.mtrl-sci↗

arXiv2021

Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression

Recent studies on compression of pretrained language models (e.g., BERT) usually use preserved accuracy as the metric for evaluation. In this paper, we propose two new metrics, label loyalty and probability loyalty that measure how closely a compressed model (i.e., student) mimics the original model (i.e., teacher). We also explore the effect of compression with regard to robustness under adversarial attacks. We benchmark quantization, pruning, knowledge distillation and progressive module replacing with loyalty and robustness. By combining multiple compression techniques, we provide a practical strategy to achieve better accuracy, loyalty and robustness.

cs.CL↗

arXiv2021

Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting

In this paper, we generalize text infilling (e.g., masked language models) by proposing Sequence Span Rewriting (SSR) as a self-supervised sequence-to-sequence (seq2seq) pre-training objective. SSR provides more fine-grained learning signals for text representations by supervising the model to rewrite imperfect spans to ground truth, and it is more consistent than text infilling with many downstream seq2seq tasks that rewrite a source sentences into a target sentence. Our experiments with T5 models on various seq2seq tasks show that SSR can substantially improve seq2seq pre-training. Moreover, we observe SSR is especially helpful to improve pre-training a small-size seq2seq model with a powerful imperfect span generator, which indicates a new perspective of transferring knowledge from a large model to a smaller model for seq2seq pre-training.

cs.CL↗

arXiv2021

Realtime Robust Malicious Traffic Detection via Frequency Domain Analysis

Machine learning (ML) based malicious traffic detection is an emerging security paradigm, particularly for zero-day attack detection, which is complementary to existing rule based detection. However, the existing ML based detection has low detection accuracy and low throughput incurred by inefficient traffic features extraction. Thus, they cannot detect attacks in realtime especially in high throughput networks. Particularly, these detection systems similar to the existing rule based detection can be easily evaded by sophisticated attacks. To this end, we propose Whisper, a realtime ML based malicious traffic detection system that achieves both high accuracy and high throughput by utilizing frequency domain features. It utilizes sequential features represented by the frequency domain features to achieve bounded information loss, which ensures high detection accuracy, and meanwhile constrains the scale of features to achieve high detection throughput. Particularly, attackers cannot easily interfere with the frequency domain features and thus Whisper is robust against various evasion attacks. Our experiments with 42 types of attacks demonstrate that, compared with the state-of-theart systems, Whisper can accurately detect various sophisticated and stealthy attacks, achieving at most 18.36% improvement, while achieving two orders of magnitude throughput. Even under various evasion attacks, Whisper is still able to maintain around 90% detection accuracy.

cs.CR↗

arXiv2021

Structural Optimization Makes Graph Classification Simpler and Better

In deep neural networks, better results can often be obtained by increasing the complexity of previously developed basic models. However, it is unclear whether there is a way to boost performance by decreasing the complexity of such models. Here, based on an optimization method, we investigate the feasibility of improving graph classification performance while simplifying the model learning process. Inspired by progress in structural information assessment, we optimize the given data sample from graphs to encoding trees. In particular, we minimize the structural entropy of the transformed encoding tree to decode the key structure underlying a graph. This transformation is denoted as structural optimization. Furthermore, we propose a novel feature combination scheme, termed hierarchical reporting, for encoding trees. In this scheme, features are transferred from leaf nodes to root nodes by following the hierarchical structures of encoding trees. We then present an implementation of the scheme in a tree kernel and a convolutional network to perform graph classification. The tree kernel follows label propagation in the Weisfeiler-Lehman (WL) subtree kernel, but it has a lower runtime complexity $O(n)$. The convolutional network is a special implementation of our tree kernel in the deep learning field and is called Encoding Tree Learning (ETL). We empirically validate our tree kernel and convolutional network with several graph classification benchmarks and demonstrate that our methods achieve better performance and lower computational consumption than competing approaches.

cs.LG↗

arXiv2021

Location-aware Single Image Reflection Removal

This paper proposes a novel location-aware deep-learning-based single image reflection removal method. Our network has a reflection detection module to regress a probabilistic reflection confidence map, taking multi-scale Laplacian features as inputs. This probabilistic map tells if a region is reflection-dominated or transmission-dominated, and it is used as a cue for the network to control the feature flow when predicting the reflection and transmission layers. We design our network as a recurrent network to progressively refine reflection removal results at each iteration. The novelty is that we leverage Laplacian kernel parameters to emphasize the boundaries of strong reflections. It is beneficial to strong reflection detection and substantially improves the quality of reflection removal results. Extensive experiments verify the superior performance of the proposed method over state-of-the-art approaches. Our code and the pre-trained model can be found at https://github.com/zdlarr/Location-aware-SIRR.

cs.CV↗

arXiv2021

Strengthening and Weakening by Dislocations in Monolayer MoS2

Dislocations govern the properties of any crystals. Yet, how dislocation of pentagonheptagon (5|7) in grain boundaries (GBs) affects the mechanical properties of two-dimensional MoS2 crystals remains poorly known. Using atomistic simulations and continuum disclination dipole model, we show that, depending on the tilt angle and 5|7 dislocation arrangement, MoS2 GB strength can be enhanced or reduced with tilt angle. For zigzag-tilt GBs primarily composed of Mo5|7+S5|7 dislocations, GB strength monotonically increases as the square of tilt angle. For armchair-tilt GBs with Mo5|7 or S5|7 dislocations, however, the trend of GB strength breaks down as 5|7 dislocations are non-evenly spaced. Moreover, mechanical failure initiates at the bond shared by 5|7 rings, in contrast to graphene where failure occurs at the bond shared by 6|7 rings. This work provides new insights into mechanical design of synthetic transition metal dichalcogenide crystals via dislocation engineering.

cond-mat.mtrl-sci↗

arXiv2021

Mechanical Ductile Detwinning in CH3NH3PbI3 Perovskite

Twin boundaries (TBs) were identified to show conflicting positive/negative effects on the physical properties of CH3NH3PbI3 perovskite, but their roles on the mechanical properties are pending. Herein, tensile characteristics of a variety of TB-dominated bicrystalline CH3NH3PbI3 perovskites are explored using molecular simulations. TB-contained CH3NH3PbI3 are classified into four types from their tensile ductile detwinning characteristics. Type I is characterized by smooth loading flow stressstrain responses, originating from relatively uniform stress distribution induced gradual amorphization at TB region. Types II and III are represented by sudden drop of loading stresses but then distinct ductile flow stress-strain curves, resulting from limited and large-area amorphizations of TB-involved structures, respectively. However, Type IV is highlighted by double apparent peaks in the loading curve followed by ductile flow response, coming from stress-concentration of localization-to-globalization at TB structure, as well as amorphization. This study provides critical insights into mechanics of CH3NH3PbI3 perovskites, and offers that TB engineering is a promising strategy to design mechanically robust hybrid organic-inorganic perovskites-based device systems

cond-mat.mtrl-sci↗

arXiv2021

AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning

In multi-party collaborative learning, the parameter server sends a global model to each data holder for local training and then aggregates committed models globally to achieve privacy protection. However, both the dragger issue of synchronous collaborative learning and the staleness issue of asynchronous collaborative learning make collaborative learning inefficient in real-world heterogeneous environments. We propose a novel and efficient collaborative learning framework named AdaptCL, which generates an adaptive sub-model dynamically from the global base model for each data holder, without any prior information about worker capability. All workers (data holders) achieve approximately identical update time as the fastest worker by equipping them with capability-adapted pruned models. Thus the training process can be dramatically accelerated. Besides, we tailor the efficient pruned rate learning algorithm and pruning approach for AdaptCL. Meanwhile, AdaptCL provides a mechanism for handling the trade-off between accuracy and time overhead and can be combined with other techniques to accelerate training further. Empirical results show that AdaptCL introduces little computing and communication overhead. AdaptCL achieves time savings of more than 41\% on average and improves accuracy in a low heterogeneous environment. In a highly heterogeneous environment, AdaptCL achieves a training speedup of 6.2x with a slight loss of accuracy.

cs.LG↗

arXiv2021

Learning to Sample Replacements for ELECTRA Pre-Training

ELECTRA pretrains a discriminator to detect replaced tokens, where the replacements are sampled from a generator trained with masked language modeling. Despite the compelling performance, ELECTRA suffers from the following two issues. First, there is no direct feedback loop from discriminator to generator, which renders replacement sampling inefficient. Second, the generator's prediction tends to be over-confident along with training, making replacements biased to correct tokens. In this paper, we propose two methods to improve replacement sampling for ELECTRA pre-training. Specifically, we augment sampling with a hardness prediction mechanism, so that the generator can encourage the discriminator to learn what it has not acquired. We also prove that efficient sampling reduces the training variance of the discriminator. Moreover, we propose to use a focal loss for the generator in order to relieve oversampling of correct tokens as replacements. Experimental results show that our method improves ELECTRA pre-training on various downstream tasks.

cs.CL↗

arXiv2021

Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge

Cant is important for understanding advertising, comedies and dog-whistle politics. However, computational research on cant is hindered by a lack of available datasets. In this paper, we propose a large and diverse Chinese dataset for creating and understanding cant from a computational linguistics perspective. We formulate a task for cant understanding and provide both quantitative and qualitative analysis for tested word embedding similarity and pretrained language models. Experiments suggest that such a task requires deep language understanding, common sense, and world knowledge and thus can be a good testbed for pretrained language models and help models perform better on other tasks. The code is available at https://github.com/JetRunner/dogwhistle. The data and leaderboard are available at https://competitions.codalab.org/competitions/30451.

cs.CL↗

arXiv2021

CoSQA: 20,000+ Web Queries for Code Search and Question Answering

Finding codes given natural language query isb eneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce the CoSQA dataset.It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance query-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%, and incorporating CoCLR brings a further improvement of 10.5%.

cs.CL↗

arXiv2021

Improving BERT with Syntax-aware Local Attention

Pre-trained Transformer-based neural language models, such as BERT, have achieved remarkable results on varieties of NLP tasks. Recent works have shown that attention-based models can benefit from more focused attention over local regions. Most of them restrict the attention scope within a linear span, or confine to certain tasks such as machine translation and question answering. In this paper, we propose a syntax-aware local attention, where the attention scopes are restrained based on the distances in the syntactic structure. The proposed syntax-aware local attention can be integrated with pretrained language models, such as BERT, to render the model to focus on syntactically relevant words. We conduct experiments on various single-sentence benchmarks, including sentence classification and sequence labeling tasks. Experimental results show consistent gains over BERT on all benchmark datasets. The extensive studies verify that our model achieves better performance owing to more focused attention over syntactically relevant words.

cs.CL↗

arXiv2021

$W^{1,p}$-metrics and conformal metrics with $L^{n/2}$-bounded scalar curvature

A $W^{1,p}$-metric on an $n$-dimensional closed Riemannian manifold naturally induces a distance function, provided $p$ is sufficiently close to $n$. If a sequence of metrics $g_k$ converges in $W^{1,p}$ to a limit metric $g$, then the corresponding distance functions $d_{g_k}$ subconverge to a limit distance function $d$, which satisfies $d\le d_g$. As an application, we show that the above convergence result applies to a sequence of conformal metrics with $L^{n/2}$-bounded scalar curvatures, under certain geometric assumptions. In particular, in this special setting, the limit distance function $d$ actually coincides with $d_{g}$.

math.DG↗

arXiv2021

Voice2Mesh: Cross-Modal 3D Face Model Generation from Voices

This work focuses on the analysis that whether 3D face models can be learned from only the speech inputs of speakers. Previous works for cross-modal face synthesis study image generation from voices. However, image synthesis includes variations such as hairstyles, backgrounds, and facial textures, that are arguably irrelevant to voice or without direct studies to show correlations. We instead investigate the ability to reconstruct 3D faces to concentrate on only geometry, which is more physiologically grounded. We propose both the supervised learning and unsupervised learning frameworks. Especially we demonstrate how unsupervised learning is possible in the absence of a direct voice-to-3D-face dataset under limited availability of 3D face scans when the model is equipped with knowledge distillation. To evaluate the performance, we also propose several metrics to measure the geometric fitness of two 3D faces based on points, lines, and regions. We find that 3D face shapes can be reconstructed from voices. Experimental results suggest that 3D faces can be reconstructed from voices, and our method can improve the performance over the baseline. The best performance gains (15% - 20%) on ear-to-ear distance ratio metric (ER) coincides with the intuition that one can roughly envision whether a speaker's face is overall wider or thinner only from a person's voice. See our project page for codes and data.

cs.GR↗

arXiv2021

Excited state fluid mechanics and mathematical principles of separation and transition

Transition and separation are difficult but important problems in the field of fluid mechanics. Hitherto, separation and transition problems have not been described accurately in mathematical terms, leading to design errors and prediction problems in fluid machine engineering. The nonlinear uncertainty involved in separation and transition makes it difficult to accurately analyze these phenomena using experimental methods. Thus, new ideas and methods are required for the mathematical prediction of fluid separation and transition. In this article, after an axiomatic treatment of fluid mechanics, the concept of an excited state is derived by generating a fluctuation velocity, and it is revealed that fluid separation and transition are special forms of this excited state. This allows us to clarify the state conditions of fluid separation and transition. Mathematical analysis of the Navier--Stokes equations leads to a general excited state theorem suitable for flowfields. Finally, the conditions of separation and transition are derived, and the corresponding general laws are established. The results presented in this article provide a foundation for future research on the mechanism of turbulence and the solution of engineering problems.

physics.flu-dyn↗

arXiv2021

High-speed silicon microring modulator at 2-um waveband

We demonstrated a silicon integrated microring modulator working at the 2-um waveband with an L-shaped PN junction. 15-GHz 3-dB electro-optic bandwidth and <1 Vcm modulation efficiency for 45-Gbps NRZ-OOK signaling is achieved at 1960 nm.

physics.app-ph↗

arXiv2021

Self-Attention Attribution: Interpreting Information Interactions Inside Transformer

The great success of Transformer-based models benefits from the powerful multi-head self-attention mechanism, which learns token dependencies and encodes contextual information from the input. Prior work strives to attribute model decisions to individual input features with different saliency measures, but they fail to explain how these input features interact with each other to reach predictions. In this paper, we propose a self-attention attribution method to interpret the information interactions inside Transformer. We take BERT as an example to conduct extensive studies. Firstly, we apply self-attention attribution to identify the important attention heads, while others can be pruned with marginal performance degradation. Furthermore, we extract the most salient dependencies in each layer to construct an attribution tree, which reveals the hierarchical interactions inside Transformer. Finally, we show that the attribution results can be used as adversarial patterns to implement non-targeted attacks towards BERT.

cs.CL↗