arXiv subjects

Zhe Zhao

Publications and source records attributed to Zhe Zhao.

At least 145 records · Page 8Linked to original sources

arXiv2020

CLUE: A Chinese Language Understanding Evaluation Benchmark

The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.CLUEbenchmarks.com

cs.CL↗

arXiv2020

Small Towers Make Big Differences

Multi-task learning aims at solving multiple machine learning tasks at the same time. A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal. In this paper, we provide some insights on understanding the trade-off between Pareto efficiency and generalization as a result of parameterization in multi-task deep learning models. As a multi-objective optimization problem, enough parameterization is needed for handling task conflicts in a constrained solution space; however, from a multi-task generalization perspective, over-parameterization undermines the benefit of learning a shared representation which helps harder tasks or tasks with limited training examples. A delicate balance between multi-task generalization and multi-objective optimization is therefore needed for finding a better trade-off between efficiency and generalization. To this end, we propose a method of under-parameterized self-auxiliaries for multi-task models to achieve the best of both worlds. It is task-agnostic and works with other multi-task learning algorithms. Empirical results show that small towers of under-parameterized self-auxiliaries can make big differences in improving Pareto efficiency in various multi-task applications.

cs.LG↗

arXiv2020

HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections

Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose \textsc{HyperGrid}, a new approach for highly effective multi-task learning. The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We apply our proposed \textsc{HyperGrid} on the current state-of-the-art T5 model, demonstrating strong performance across the GLUE and SuperGLUE benchmarks when using only a single multi-task model. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.

cs.CL↗

arXiv2020

FastBERT: a Self-distilling BERT with Adaptive Inference Time

Pre-trained language models like BERT have proven to be highly performant. However, they are often computationally expensive in many practical scenarios, for such heavy models can hardly be readily implemented with limited resources. To improve their efficiency with an assured model performance, we propose a novel speed-tunable FastBERT with adaptive inference time. The speed at inference can be flexibly adjusted under varying demands, while redundant calculation of samples is avoided. Moreover, this model adopts a unique self-distillation mechanism at fine-tuning, further enabling a greater computational efficacy with minimal loss in performance. Our model achieves promising results in twelve English and Chinese datasets. It is able to speed up by a wide range from 1 to 12 times than BERT if given different speedup thresholds to make a speed-performance tradeoff.

cs.CL↗

arXiv2020

Taking Care of The Discretization Problem: A Comprehensive Study of the Discretization Problem and A Black-Box Adversarial Attack in Discrete Integer Domain

Numerous methods for crafting adversarial examples were proposed recently with high success rate. Since most existing machine learning based classifiers normalize images into some continuous, real vector, domain firstly, attacks often craft adversarial examples in such domain. However, "adversarial" examples may become benign after denormalizing them back into the discrete integer domain, known as the discretization problem. This problem was mentioned in some work, but has received relatively little attention. In this work, we first conduct a comprehensive study of existing methods and tools for crafting. We theoretically analyze 34 representative methods and empirically study 20 representative open source tools for crafting adversarial images. Our study reveals that the discretization problem is far more serious than originally thought. This suggests that the discretization problem should be taken into account seriously when crafting adversarial examples and measuring attack success rate. As a first step towards addressing this problem in black-box scenario, we propose a black-box method which reduces the adversarial example searching problem to a derivative-free optimization problem. Our method is able to craft adversarial images by derivative-free search in the discrete integer domain. Experimental results show that our method is comparable to recent white-box methods (e.g., FGSM, BIM and C\&W) and achieves significantly higher success rate in terms of adversarial examples in the discrete integer domain than recent black-box methods (e.g., ZOO, NES-PGD and Bandits). Moreover, our method is able to handle models that is non-differentiable and successfully break the winner of NIPS 2017 competition on defense with 95\% success rate. Our results suggest that discrete optimization algorithms open up a promising area of research into effective black-box attacks.

cs.LG↗

arXiv2020

Who is Real Bob? Adversarial Attacks on Speaker Recognition Systems

Speaker recognition (SR) is widely used in our daily life as a biometric authentication or identification mechanism. The popularity of SR brings in serious security concerns, as demonstrated by recent adversarial attacks. However, the impacts of such threats in the practical black-box setting are still open, since current attacks consider the white-box setting only. In this paper, we conduct the first comprehensive and systematic study of the adversarial attacks on SR systems (SRSs) to understand their security weakness in the practical blackbox setting. For this purpose, we propose an adversarial attack, named FAKEBOB, to craft adversarial samples. Specifically, we formulate the adversarial sample generation as an optimization problem, incorporated with the confidence of adversarial samples and maximal distortion to balance between the strength and imperceptibility of adversarial voices. One key contribution is to propose a novel algorithm to estimate the score threshold, a feature in SRSs, and use it in the optimization problem to solve the optimization problem. We demonstrate that FAKEBOB achieves 99% targeted attack success rate on both open-source and commercial systems. We further demonstrate that FAKEBOB is also effective on both open-source and commercial systems when playing over the air in the physical world. Moreover, we have conducted a human study which reveals that it is hard for human to differentiate the speakers of the original and adversarial voices. Last but not least, we show that four promising defense methods for adversarial attack from the speech recognition domain become ineffective on SRSs against FAKEBOB, which calls for more effective defense methods. We highlight that our study peeks into the security implications of adversarial attacks on SRSs, and realistically fosters to improve the security robustness of SRSs.

eess.AS↗

arXiv2019

K-BERT: Enabling Language Representation with Knowledge Graph

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by equipped with a KG without pre-training by-self because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

cs.CL↗

arXiv2019

UER: An Open-Source Toolkit for Pre-training Models

Existing works, including ELMO and BERT, have revealed the importance of pre-training for NLP tasks. While there does not exist a single pre-training model that works best in all cases, it is of necessity to develop a framework that is able to deploy various pre-training models efficiently. For this purpose, we propose an assemble-on-demand pre-training toolkit, namely Universal Encoder Representations (UER). UER is loosely coupled, and encapsulated with rich modules. By assembling modules on demand, users can either reproduce a state-of-the-art pre-training model or develop a pre-training model that remains unexplored. With UER, we have built a model zoo, which contains pre-trained models based on different corpora, encoders, and targets (objectives). With proper pre-trained models, we could achieve new state-of-the-art results on a range of downstream datasets.

cs.CL↗

arXiv2019

Dynamic Spatiotemporal Beams that Combine Two Independent and Controllable Orbital-Angular-Momenta Using Multiple Optical-Frequency-Comb Lines

Novel forms of beam generation and propagation based on structured light and orbital angular momentum (OAM) have gained significant interest over the past several years. Indeed, dynamic OAM can manifest at a given propagation distance in different forms , including: (1) a simple Gaussian-like beam "dot" "revolves" around an offset central axis in time, and (2) a Laguerre-Gaussian (LG) beam with a helically "twisting" phase front that "rotates" around its own central null in time. In this paper, we numerically generate dynamic spatiotemporal beams that combine these two forms of orbital-angular-momenta by coherently adding multiple frequency comb lines such that each carries a superposition of multiple LG(l,p) modes containing one l value but multiple p values. The generated beams can have different non-zero rotating l values with high modal purities that exhibit both "rotation" and "revolution" in time at a given propagation distance. In our simulation results, we were able to control and vary several parameters, including the: (i) rotating l value from +1 to +3 with modal purities of ~96%, (ii) revolving speed of 0.2-0.6 THz, (iii) beam waist of 0.15-0.5 mm, and (iv) revolving radius of 0.75-1.5 mm.

physics.optics↗

arXiv2019

Fairness in Recommendation Ranking through Pairwise Comparisons

Recommender systems are one of the most pervasive applications of machine learning in industry, with many services using them to match users to products or information. As such it is important to ask: what are the possible fairness risks, how can we quantify them, and how should we address them? In this paper we offer a set of novel metrics for evaluating algorithmic fairness concerns in recommender systems. In particular we show how measuring fairness based on pairwise comparisons from randomized experiments provides a tractable means to reason about fairness in rankings from recommender systems. Building on this metric, we offer a new regularizer to encourage improving this metric during model training and thus improve fairness in the resulting rankings. We apply this pairwise regularization to a large-scale, production recommender system and show that we are able to significantly improve the system's pairwise fairness.

cs.CY↗

arXiv2018

Analogical Reasoning on Chinese Morphological and Semantic Relations

Analogical reasoning is effective in capturing linguistic regularities. This paper proposes an analogical reasoning task on Chinese. After delving into Chinese lexical knowledge, we sketch 68 implicit morphological relations and 28 explicit semantic relations. A big and balanced dataset CA8 is then built for this task, including 17813 questions. Furthermore, we systematically explore the influences of vector representations, context features, and corpora on analogical reasoning. With the experiments, CA8 is proved to be a reliable benchmark for evaluating Chinese word embeddings.

cs.CL↗

arXiv2017

80-Gbit/s 100-m Free-Space Optical Data Transmission Link via a Flying UAV Using Multiplexing of Orbital-Angular-Momentum Beams

We explore the use of orbital-angular-momentum (OAM)-multiplexing to increase the capacity of free-space data transmission to moving platforms, with an added potential benefit of decreasing the probability of data intercept. Specifically, we experimentally demonstrate and characterize the performance of an OAM-multiplexed, free-space optical (FSO) communications link between a ground station and a moving unmanned-aerial-vehicle (UAV). We achieve a total capacity of 80 Gbit/s up to 100-m-roundtrip link by multiplexing 2 OAM beams, each carrying a 40-Gbit/s quadrature-phase-shift-keying (QPSK) signal. Moreover, we investigate for static, hovering, and moving conditions the effects of channel impairments, including: tracking errors, propeller-induced airflows, power loss, intermodal crosstalk, and system bit error rate (BER). We find the following: (a) when the UAV hovers in the air, the power on the desired mode fluctuates by 2.1 dB, while the crosstalk to the other mode is -19 dB below the power on the desired mode; and (b) when the UAV moves in the air, the power fluctuation on the desired mode increases to 4.3 dB and the crosstalk to the other mode increases to -10 dB. Furthermore, the channel crosstalk decreases with an increase in OAM mode spacing.

physics.optics↗

arXiv2017

Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations

How can we learn a classifier that is "fair" for a protected or sensitive group, when we do not know if the input to the classifier belongs to the protected group? How can we train such a classifier when data on the protected group is difficult to attain? In many settings, finding out the sensitive input attribute can be prohibitively expensive even during model training, and sometimes impossible during model serving. For example, in recommender systems, if we want to predict if a user will click on a given recommendation, we often do not know many attributes of the user, e.g., race or age, and many attributes of the content are hard to determine, e.g., the language or topic. Thus, it is not feasible to use a different classifier calibrated based on knowledge of the sensitive attribute. Here, we use an adversarial training procedure to remove information about the sensitive attribute from the latent representation learned by a neural network. In particular, we study how the choice of data for the adversarial training effects the resulting fairness properties. We find two interesting results: a small amount of data is needed to train these adversarial models, and the data distribution empirically drives the adversary's notion of fairness.

cs.LG↗

arXiv2017

Orthogonally Polarized Kerr Frequency Combs

Kerr optical frequency combs with multi-gigahertz spacing have previously been demonstrated in chip-scale microresonators, with potential applications in coherent communication, spectroscopy, arbitrary waveform generation, and radio frequency photonic oscillators. In general, the harmonics of a frequency comb are identically polarized in a single microresonator. In this work, we report that one comb in one polarization is generated by an orthogonally polarized soliton comb and two low-noise, orthogonally polarized combs interact with each other and exist simultaneously in a single microresonator. The second comb generation is attributed to the strong cross-phase modulation with the orthogonally polarized soliton comb and the high peak power of the intracavity soliton pulse. Experimental results show that a second frequency comb is excited even when a continuous wave light as a "seed"-with power as low as 0.1 mW-is input, while its own power level is below the threshold of comb generation. Moreover, the second comb has a concave envelope, which is different from the sech2 envelope of the soliton comb. This is due to the frequency mismatch between the harmonics and the resonant frequency. We also find that the repetition rates of these two combs coincide, although two orthogonal resonant modes are characterized by different free spectral ranges.

physics.optics↗

arXiv2017

Using a Complex Optical Orbital-Angular-Momentum Spectrum to Measure Object Parameters: A Spatial Domain Approach

Light beams can be characterized by their complex spatial profiles in both intensity and phase. Analogous to time signals, which can be decomposed into multiple orthogonal frequency functions, a light beam can also be decomposed into a set of spatial modes that are taken from an orthogonal basis. Such a decomposition can provide a tool for spatial spectrum analysis, which may allow the stable, accurate and robust extraction of physical object information that may not be readily achievable using traditional approaches. As an example, we measure the opening angle of an object using the complex spectrum of orbital angular momentum (OAM) modes as the basis, achieving a more than 15 dB signal-to-noise ratio. We find that the dip (i.e., notch) positions of the OAM intensity spectrum are dependent on an object's opening angle but independent of the object opening's angular orientation, whereas the slope of the OAM phase spectrum is dependent on the object opening's orientation but independent on the opening angle.

physics.optics↗

arXiv2016

Spatial Phase and Amplitude Structuring of Beams Using a Combination of Multiple Orthogonal Spatial Functions with Complex Coefficients

Analogous to time signals that can be composed of multiple frequency functions, we use uniquely structured orthogonal spatial modes to create different beam shapes. We tailor the spatial structure by judiciously choosing a weighted combination of multiple modal states within an orthogonal basis set, and we can tunably create beam phase and intensity "shapes" that are not otherwise readily achievable. As an example shape, we use a series of orbital-angular-momentum (OAM) functions with adjustable complex weights to create a reconfigurable spatial region of higher localized power as compared to traditional beam combining. We simulate a structured beam created by coherently combining several orthogonal OAM beams with different complex weights, and we achieve a >10X localized power density enhancement with 19 beams. Additionally, we can create unique shapes by passing a single beam through a specially designed phase and intensity mask that contains the combination of multiple OAM functions each with complex weights. Using this approach, we experimentally demonstrate a ~2.5X localized power density increase when utilizing 9 functions.

physics.optics↗

arXiv2016

Learning Document Embeddings by Predicting N-grams for Sentiment Classification of Long Movie Reviews

Despite the loss of semantic information, bag-of-ngram based methods still achieve state-of-the-art results for tasks such as sentiment classification of long movie reviews. Many document embeddings methods have been proposed to capture semantics, but they still can't outperform bag-of-ngram based methods on this task. In this paper, we modify the architecture of the recently proposed Paragraph Vector, allowing it to learn document vectors by predicting not only words, but n-gram features as well. Our model is able to capture both semantics and word order in documents while keeping the expressive power of learned vectors. Experimental results on IMDB movie review dataset shows that our model outperforms previous deep learning models and bag-of-ngram based models due to the above advantages. More robust results are also obtained when our model is combined with other models. The source code of our model will be also published together with this paper.

cs.CL↗

arXiv2016

Orbital Angular Momentum-based Space Division Multiplexing for High-capacity Underwater Optical Communications

To increase system capacity of underwater optical communications, we employ the spatial domain to simultaneously transmit multiple orthogonal spatial beams, each carrying an independent data channel. In this paper, we multiplex and transmit four green orbital angular momentum (OAM) beams through a single aperture. Moreover, we investigate the degrading effects of scattering/turbidity, water current, and thermal gradient-induced turbulence, and we find that thermal gradients cause the most distortions and turbidity causes the most loss. We show systems results using two different data generation techniques, one at 1064 nm for 10-Gbit/s/beam and one at 520 nm for 1-Gbit/s/beam, we use both techniques since present data-modulation technologies are faster for infrared (IR) than for green. For the higher-rate link, data is modulated in the IR, and OAM imprinting is performed in the green using a specially-designed metasurface phase mask. For the lower rates, a green laser diode is directly modulated. Finally, we show that inter-channel crosstalk induced by thermal gradients can be mitigated using multi-channel equalisation processing.

physics.optics↗