arXiv subjects

Peng Xu

Publications and source records attributed to Peng Xu.

At least 289 records · Page 16Linked to original sources

arXiv2022

How to Understand Masked Autoencoders

"Masked Autoencoders (MAE) Are Scalable Vision Learners" revolutionizes the self-supervised learning method in that it not only achieves the state-of-the-art for image pre-training, but is also a milestone that bridges the gap between visual and linguistic masked autoencoding (BERT-style) pre-trainings. However, to our knowledge, to date there are no theoretical perspectives to explain the powerful expressivity of MAE. In this paper, we, for the first time, propose a unified theoretical framework that provides a mathematical understanding for MAE. Specifically, we explain the patch-based attention approaches of MAE using an integral kernel under a non-overlapping domain decomposition setting. To help the research community to further comprehend the main reasons of the great success of MAE, based on our framework, we pose five questions and answer them with mathematical rigor using insights from operator theory.

cs.CV↗

arXiv2022

Quantum crosstalk analysis for simultaneous gate operations on superconducting qubits

Maintaining or even improving gate performance with growing numbers of parallel controlled qubits is a vital requirement for fault-tolerant quantum computing. For superconducting quantum processors, though isolated one- or two-qubit gates have been demonstrated with high-fidelity, implementing these gates in parallel commonly shows worse performance. Generally, this degradation is attributed to various crosstalks between qubits, such as quantum crosstalk due to residual inter-qubit coupling. An understanding of the exact nature of these crosstalks is critical to figuring out respective mitigation schemes and improved qubit architecture designs with low crosstalk. Here we give a theoretical analysis of quantum crosstalk impact on simultaneous gate operations in a qubit architecture, where fixed-frequency transmon qubits are coupled via a tunable bus, and sub-100-ns controlled-Z (CZ) gates can be realized by applying a baseband flux pulse on the bus. Our analysis shows that for microwave-driven single-qubit gates, the dressing from the qubit-qubit coupling can cause non-negligible cross-driving errors when qubits operate near frequency collision regions. During CZ gate operations, although unwanted near-neighbor interactions are nominally turned off, sub-MHz parasitic next-near-neighbor interactions involving spectator qubits can still exist, causing considerable leakage or control error when one operates qubit systems around these parasitic resonance points. To ensure high-fidelity simultaneous operations, there could raise a request to figure out a better way to balance the gate error from target qubit systems themselves and the error from non-participating spectator qubits. Overall, our analysis suggests that towards useful quantum processors, the qubit architecture should be examined carefully in the context of high-fidelity simultaneous gate operations in a scalable qubit lattice.

quant-ph↗

arXiv2022

$r$-Hyperideals and Generalizations of $r$-Hyperideals in Krasner Hyperrings

In this study, we examine some properties of $r$-hyperideals in the commutative Krasner hyperrings. Some properties of $pr$-hyperideals are also studied. The relation between prime hyperideals and $r$-hyperideals is investigated. We show that the image and the inverse image of an $r$-hyperideal is also an $r$-hyperideal. We also introduce a generalization of r-hyperideals and we prove some properties of them.

math.GM↗

arXiv2022

Deep Learning for Free-Hand Sketch: A Survey

Free-hand sketches are highly illustrative, and have been widely used by humans to depict objects or stories from ancient times to the present. The recent prevalence of touchscreen devices has made sketch creation a much easier task than ever and consequently made sketch-oriented applications increasingly popular. The progress of deep learning has immensely benefited free-hand sketch research and applications. This paper presents a comprehensive survey of the deep learning techniques oriented at free-hand sketch data, and the applications that they enable. The main contents of this survey include: (i) A discussion of the intrinsic traits and unique challenges of free-hand sketch, to highlight the essential differences between sketch data and other data modalities, e.g., natural photos. (ii) A review of the developments of free-hand sketch research in the deep learning era, by surveying existing datasets, research topics, and the state-of-the-art methods through a detailed taxonomy and experimental evaluation. (iii) Promotion of future work via a discussion of bottlenecks, open problems, and potential research directions for the community.

cs.CV↗

arXiv2022

Android-COCO: Android Malware Detection with Graph Neural Network for Byte- and Native-Code

With the popularity of Android growing exponentially, the amount of malware has significantly exploded. It is arguably one of the most viral problems on mobile platforms. Recently, various approaches have been introduced to detect Android malware, the majority of these are either based on the Manifest File features or the structural information, such as control flow graph and API calls. Among those methods, nearly all of them only consider the Java byte-code as the target to detect malicious behaviors. However, Recent research and our own statistics show that native payloads are commonly used in both benign and malicious apps. Current state-of-the-art Android static analysis tools avoid handling native method invocation. None of those tools have the capability to capture the inter-language behaviors. In this work, we explore an ensemble mechanism, which presents how the combination of byte-code and native-code analysis of Android applications can be efficiently used to cope with the advanced sophistication of Android malware. We, therefore, present a multi-layer approach that utilizes deep learning, natural language processing (NLP), as well as graph embedding techniques to handle the threats of Android malware, both from the Java byte-code and native code. After that, we design an ensemble algorithm to get the final result of malware detection system. To be specific, the first layer of our detection approach operates on the byte-code of application and the native code level, whereas the second layer focuses on the ensemble algorithm. Large-scale experiments on 100,113 samples (35,113 malware and 65,000 benign) show that only byte-code sub-system yields 99.8% accuracy and native-code sub-system yields an accuracy of 96.6%, whereas the Android-COCO method attains an accuracy of 99.86% which outperforms various related works.

cs.CR↗

arXiv2022

hybrid-Falcon: Hybrid Pattern Malware Detection and Categorization with Network Traffic and Program Code

Nowadays, Android is the most dominant operating system in the mobile ecosystem, with billions of people using its apps daily. As expected, this trend did not go unnoticed by miscreants, and Android became the favorite platform for discovering new victims through malicious apps. Moreover, these apps have become so sophisticated that they can bypass anti-malware measures to protect the users. Therefore, it is safe to admit that traditional anti-malware techniques have become cumbersome, sparking the urge to develop an efficient way to detect Android malware. This paper presents hybrid-Flacon, a hybrid pattern Android malware detection and categorization framework. It combines dynamic and static features of Android malware, which are from network traffic and code graph structure. In hybrid-Flacon, we treat network traffic as a dynamic feature and process it as a 2D image sequence. Meanwhile, hybrid-Flacon handles each network flow in the packet as a 2D image and uses a bidirectional LSTM network to process those 2D-image sequences to obtain vectors representing network packets. We use the program code graph for a static feature and introduce natural language processing (NLP) inspired techniques on function call graph (FCG). We design a graph neural network-based approach to convert the whole graph structure of Android apps to vectors. Finally, We utilize those converted vectors, both network and program code features, and concatenate them to detect and categorize the malware. Our results reveal that hybrid-Flacon yields better results as we get 97.16% accuracy on average for malware detection and 88.32% accuracy for malware categorization. Additionally, we release a dataset AndroNetMnist, which converts the network traffic to a 2D-image sequence and helps to accomplish malware detection on a 2D-image sequence.

cs.CR↗

arXiv2022

Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset

Automatic speech recognition (ASR) on low resource languages improves the access of linguistic minorities to technological advantages provided by artificial intelligence (AI). In this paper, we address the problem of data scarcity for the Hong Kong Cantonese language by creating a new Cantonese dataset. Our dataset, Multi-Domain Cantonese Corpus (MDCC), consists of 73.6 hours of clean read speech paired with transcripts, collected from Cantonese audiobooks from Hong Kong. It comprises philosophy, politics, education, culture, lifestyle and family domains, covering a wide range of topics. We also review all existing Cantonese datasets and analyze them according to their speech type, data source, total size and availability. We further conduct experiments with Fairseq S2T Transformer, a state-of-the-art ASR model, on the biggest existing dataset, Common Voice zh-HK, and our proposed MDCC, and the results show the effectiveness of our dataset. In addition, we create a powerful and robust Cantonese ASR model by applying multi-dataset learning on MDCC and Common Voice zh-HK.

cs.CL↗

arXiv2021

Generating Empathetic Responses by Looking Ahead the User's Sentiment

An important aspect of human conversation difficult for machines is conversing with empathy, which is to understand the user's emotion and respond appropriately. Recent neural conversation models that attempted to generate empathetic responses either focused on conditioning the output to a given emotion, or incorporating the current user emotional state. However, these approaches do not factor in how the user would feel towards the generated response. Hence, in this paper, we propose Sentiment Look-ahead, which is a novel perspective for empathy that models the future user emotional state. In short, Sentiment Look-ahead is a reward function under a reinforcement learning framework that provides a higher reward to the generative model when the generated utterance improves the user's sentiment. We implement and evaluate three different possible implementations of sentiment look-ahead and empirically show that our proposed approach can generate significantly more empathetic, relevant, and fluent responses than other competitive baselines such as multitask learning.

cs.CL↗

arXiv2021

Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering

The current state-of-the-art generative models for open-domain question answering (ODQA) have focused on generating direct answers from unstructured textual information. However, a large amount of world's knowledge is stored in structured databases, and need to be accessed using query languages such as SQL. Furthermore, query languages can answer questions that require complex reasoning, as well as offering full explainability. In this paper, we propose a hybrid framework that takes both textual and tabular evidence as input and generates either direct answers or SQL queries depending on which form could better answer the question. The generated SQL queries can then be executed on the associated databases to obtain the final answers. To the best of our knowledge, this is the first paper that applies Text2SQL to ODQA tasks. Empirically, we demonstrate that on several ODQA datasets, the hybrid methods consistently outperforms the baseline models that only take homogeneous input by a large margin. Specifically we achieve state-of-the-art performance on OpenSQuAD dataset using a T5-base model. In a detailed analysis, we demonstrate that the being able to generate structural SQL queries can always bring gains, especially for those questions that requires complex reasoning.

cs.CL↗

arXiv2021

Hierarchical Neural Data Synthesis for Semantic Parsing

Semantic parsing datasets are expensive to collect. Moreover, even the questions pertinent to a given domain, which are the input of a semantic parsing system, might not be readily available, especially in cross-domain semantic parsing. This makes data augmentation even more challenging. Existing methods to synthesize new data use hand-crafted or induced rules, requiring substantial engineering effort and linguistic expertise to achieve good coverage and precision, which limits the scalability. In this work, we propose a purely neural approach of data augmentation for semantic parsing that completely removes the need for grammar engineering while achieving higher semantic parsing accuracy. Furthermore, our method can synthesize in the zero-shot setting, where only a new domain schema is available without any input-output examples of the new domain. On the Spider cross-domain text-to-SQL semantic parsing benchmark, we achieve the state-of-the-art performance on the development set (77.2% accuracy) using our zero-shot augmentation.

cs.CL↗

arXiv2021

Contrastive Document Representation Learning with Graph Attention Networks

Recent progress in pretrained Transformer-based language models has shown great success in learning contextual representation of text. However, due to the quadratic self-attention complexity, most of the pretrained Transformers models can only handle relatively short text. It is still a challenge when it comes to modeling very long documents. In this work, we propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. This graph attention network allows us to leverage the high-level semantic structure of the document. In addition, based on our graph document model, we design a simple contrastive learning strategy to pretrain our models on a large amount of unlabeled corpus. Empirically, we demonstrate the effectiveness of our approaches in document classification and document retrieval tasks.

cs.CL↗

arXiv2021

Attention-guided Generative Models for Extractive Question Answering

We propose a novel method for applying Transformer models to extractive question answering (QA) tasks. Recently, pretrained generative sequence-to-sequence (seq2seq) models have achieved great success in question answering. Contributing to the success of these models are internal attention mechanisms such as cross-attention. We propose a simple strategy to obtain an extractive answer span from the generative model by leveraging the decoder cross-attention patterns. Viewing cross-attention as an architectural prior, we apply joint training to further improve QA performance. Empirical results show that on open-domain question answering datasets like NaturalQuestions and TriviaQA, our method approaches state-of-the-art performance on both generative and extractive inference, all while using much fewer parameters. Furthermore, this strategy allows us to perform hallucination-free inference while conferring significant improvements to the model's ability to rerank relevant passages.

cs.CL↗

arXiv2021

Multiplicative Position-aware Transformer Models for Language Understanding

Transformer models, which leverage architectural improvements like self-attention, perform remarkably well on Natural Language Processing (NLP) tasks. The self-attention mechanism is position agnostic. In order to capture positional ordering information, various flavors of absolute and relative position embeddings have been proposed. However, there is no systematic analysis on their contributions and a comprehensive comparison of these methods is missing in the literature. In this paper, we review major existing position embedding methods and compare their accuracy on downstream NLP tasks, using our own implementations. We also propose a novel multiplicative embedding method which leads to superior accuracy when compared to existing methods. Finally, we show that our proposed embedding method, served as a drop-in replacement of the default absolute position embedding, can improve the RoBERTa-base and RoBERTa-large models on SQuAD1.1 and SQuAD2.0 datasets.

cs.CL↗

arXiv2021

A Parallel Tempering Approach for Efficient Exploration of the Verification Tradespace in Engineered Systems

Verification is a critical process in the development of engineered systems. Through verification, engineers gain confidence in the correct functionality of the system before it is deployed into operation. Traditionally, verification strategies are fixed at the beginning of the system's development and verification activities are executed as the development progresses. Such an approach appears to give inferior results as the selection of the verification activities does not leverage information gained through the system's development process. In contrast, a set-based design approach to verification, where verification activities are dynamically selected as the system's development progresses, has been shown to provide superior results. However, its application under realistic engineering scenarios remains unproven due to the large size of the verification tradespace. In this work, we propose a parallel tempering approach (PTA) to efficiently explore the verification tradespace. First, we formulate exploration of the verification tradespace as a tree search problem. Second, we design a parallel tempering (PT) algorithm by simulating several replicas of the verification process at different temperatures to obtain a near-optimal result. Third, We apply the PT algorithm to all possible verification states to dynamically identify near-optimal results. The effectiveness of the proposed PTA is evaluated on a partial model of a notional satellite optical instrument.

cs.SE↗

arXiv2021

High fidelity entanglement of neutral atoms via a Rydberg-mediated single-modulated-pulse controlled-PHASE gate

Neutral atom platform has become an attractive choice to study the science of quantum information and quantum simulation, where intense efforts have been devoted to the entangling processes between individual atoms. For the development of this area, two-qubit controlled-PHASE gate via Rydberg blockade is one of the most essential elements. Recent theoretical studies have suggested the advantages of introducing non-trivial waveform modulation into the gate protocol, which is anticipated to improve its performance towards the next stage. We report our recent experimental results in realizing a two-qubit controlled-PHASE($C_Z$) gate via off-resonant modulated driving(ORMD) embedded in two-photon transition for Rb atoms. It relies upon a single modulated driving pulse with a carefully calculated smooth waveform to gain the appropriate phase accumulations required by the two-qubit gate. Combining this $C_Z$ gate with global microwave pulses, two-atom entanglement is generated with the raw fidelity of 0.945(6). Accounting for state preparation and measurement (SPAM) errors, we extract the entanglement operation fidelity to be 0.980(7). Our work features completing the $C_Z$ gate operation within a single pulse to avoid shelved Rydberg population, thus demonstrate another promising route for realizing high-fidelity two-qubit gate for neutral atom platform.

quant-ph↗

arXiv2021

Generalized parametric resonance in a spin-1 Bose-Einstein condensate

We propose a generalized Mathieu equation (GME) which describes well the dynamics for two different models in spin-1 Bose-Einstein condensates. The stability chart of this GME differs significantly from that of Mathieu's equation and the unstable dynamics under this GME is called generalized parametric resonance. A typical region of $ε\gtrsim 1$ and $δ\approx 0.25$ can be used to distinguish these two equations. The GME we propose not only explains the experimental results of Hoang et al. [Nat. Commun. 7, 11233 (2016)] in nematic space with a small driving strength, but predicts the behavior in the regime of large driving strength. In addition, the model in spin space we propose, whose dynamics also obeys this GME, can be well-tuned such that it is easily implemented in experiments.

cond-mat.quant-gas↗

arXiv2021

Defect-free arbitrary-geometry assembly of mixed-species atom arrays

Optically trapped mixed-species single atom arrays with arbitrary geometries are an attractive and promising platform for various applications, because tunable quantum systems with multiple components provide extra degrees of freedom for experimental control. Here, we report the first demonstration of two-dimensional $6\times4$ dual-species atom assembly with a filling fraction of 0.88 (0.89) for $^{85}$Rb ($^{87}$Rb) atoms. This mixed-species atomic synthetic is achieved via rearranging initially randomly distributed atoms using a sorting algorithm (heuristic heteronuclear algorithm) which is proposed for bottom-up atom assembly with both user-defined geometries and two-species atom number ratios. Our fully tunable hybrid-atom system of scalable advantages is a good starting point for high-fidelity quantum logic, many-body quantum simulation and forming defect-free single molecule arrays.

physics.atom-ph↗

arXiv2021

CAiRE in DialDoc21: Data Augmentation for Information-Seeking Dialogue System

Information-seeking dialogue systems, including knowledge identification and response generation, aim to respond to users with fluent, coherent, and informative responses based on users' needs, which. To tackle this challenge, we utilize data augmentation methods and several training techniques with the pre-trained language models to learn a general pattern of the task and thus achieve promising performance. In DialDoc21 competition, our system achieved 74.95 F1 score and 60.74 Exact Match score in subtask 1, and 37.72 SacreBLEU score in subtask 2. Empirical analysis is provided to explain the effectiveness of our approaches.

cs.CL↗