arXiv subjects

Cong Zhou

Publications and source records attributed to Cong Zhou.

52 records · Page 3Linked to original sources

arXiv2022

GUDN: A novel guide network with label reinforcement strategy for extreme multi-label text classification

In natural language processing, extreme multi-label text classification is an emerging but essential task. The problem of extreme multi-label text classification (XMTC) is to recall some of the most relevant labels for a text from an extremely large label set. Large-scale pre-trained models have brought a new trend to this problem. Though the large-scale pre-trained models have made significant achievements on this problem, the valuable fine-tuned methods have yet to be studied. Though label semantics have been introduced in XMTC, the vast semantic gap between texts and labels has yet to gain enough attention. This paper builds a new guide network (GUDN) to help fine-tune the pre-trained model to instruct classification later. Furthermore, GUDN uses raw label semantics combined with a helpful label reinforcement strategy to effectively explore the latent space between texts and labels, narrowing the semantic gap, which can further improve predicted accuracy. Experimental results demonstrate that GUDN outperforms state-of-the-art methods on Eurlex-4k and has competitive results on other popular datasets. In an additional experiment, we investigated the input lengths' influence on the Transformer-based model's accuracy. Our source code is released at https://t.hk.uy/aFSH.

cs.CL↗

arXiv2022

Effidit: Your AI Writing Assistant

In this technical report, we introduce Effidit (Efficient and Intelligent Editing), a digital writing assistant that facilitates users to write higher-quality text more efficiently by using artificial intelligence (AI) technologies. Previous writing assistants typically provide the function of error checking (to detect and correct spelling and grammatical errors) and limited text-rewriting functionality. With the emergence of large-scale neural language models, some systems support automatically completing a sentence or a paragraph. In Effidit, we significantly expand the capacities of a writing assistant by providing functions in five categories: text completion, error checking, text polishing, keywords to sentences (K2S), and cloud input methods (cloud IME). In the text completion category, Effidit supports generation-based sentence completion, retrieval-based sentence completion, and phrase completion. In contrast, many other writing assistants so far only provide one or two of the three functions. For text polishing, we have three functions: (context-aware) phrase polishing, sentence paraphrasing, and sentence expansion, whereas many other writing assistants often support one or two functions in this category. The main contents of this report include major modules of Effidit, methods for implementing these modules, and evaluation results of some key methods.

cs.CL↗

arXiv2022

One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code

People perceive the world with multiple senses (e.g., through hearing sounds, reading words and seeing objects). However, most existing AI systems only process an individual modality. This paper presents an approach that excels at handling multiple modalities of information with a single model. In our "{SkillNet}" model, different parts of the parameters are specialized for processing different modalities. Unlike traditional dense models that always activate all the model parameters, our model sparsely activates parts of the parameters whose skills are relevant to the task. Such model design enables SkillNet to learn skills in a more interpretable way. We develop our model for five modalities including text, image, sound, video and code. Results show that, SkillNet performs comparably to five modality-specific fine-tuned models. Moreover, our model supports self-supervised pretraining with the same sparsely activated way, resulting in better initialized parameters for different modalities. We find that pretraining significantly improves the performance of SkillNet on five modalities, on par with or even better than baselines with modality-specific pretraining. On the task of Chinese text-to-image retrieval, our final system achieves higher accuracy than existing leading systems including Wukong{ViT-B} and Wenlan 2.0 while using less number of activated parameters.

cs.CL↗

arXiv2022

SkillNet-NLU: A Sparsely Activated Model for General-Purpose Natural Language Understanding

Prevailing deep models are single-purpose and overspecialize at individual tasks. However, when being extended to new tasks, they typically forget previously learned skills and learn from scratch. We address this issue by introducing SkillNet-NLU, a general-purpose model that stitches together existing skills to learn new tasks more effectively. The key feature of our approach is that it is sparsely activated guided by predefined skills. Different from traditional dense models that always activate all the model parameters, SkillNet-NLU only activates parts of the model parameters whose skills are relevant to the target task. When learning for a new task, our approach precisely activates required skills and also provides an option to add new skills. We evaluate on natural language understandings tasks and have the following findings. First, with only one model checkpoint, SkillNet-NLU performs better than task-specific fine-tuning and two multi-task learning baselines (i.e., dense model and Mixture-of-Experts model) on six tasks. Second, sparsely activated pre-training further improves the overall performance. Third, SkillNet-NLU significantly outperforms baseline systems when being extended to new tasks.

cs.CL↗

arXiv2022

Pretraining Chinese BERT for Detecting Word Insertion and Deletion Errors

Chinese BERT models achieve remarkable progress in dealing with grammatical errors of word substitution. However, they fail to handle word insertion and deletion because BERT assumes the existence of a word at each position. To address this, we present a simple and effective Chinese pretrained model. The basic idea is to enable the model to determine whether a word exists at a particular position. We achieve this by introducing a special token \texttt{[null]}, the prediction of which stands for the non-existence of a word. In the training stage, we design pretraining tasks such that the model learns to predict \texttt{[null]} and real words jointly given the surrounding context. In the inference stage, the model readily detects whether a word should be inserted or deleted with the standard masked language modeling function. We further create an evaluation dataset to foster research on word insertion and deletion. It includes human-annotated corrections for 7,726 erroneous sentences. Results show that existing Chinese BERT performs poorly on detecting insertion and deletion errors. Our approach significantly improves the F1 scores from 24.1\% to 78.1\% for word insertion and from 26.5\% to 68.5\% for word deletion, respectively.

cs.CL↗

arXiv2022

"Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction

Whole word masking (WWM), which masks all subwords corresponding to a word at once, makes a better English BERT model. For the Chinese language, however, there is no subword because each token is an atomic character. The meaning of a word in Chinese is different in that a word is a compositional unit consisting of multiple characters. Such difference motivates us to investigate whether WWM leads to better context understanding ability for Chinese BERT. To achieve this, we introduce two probing tasks related to grammatical error correction and ask pretrained models to revise or insert tokens in a masked language modeling manner. We construct a dataset including labels for 19,075 tokens in 10,448 sentences. We train three Chinese BERT models with standard character-level masking (CLM), WWM, and a combination of CLM and WWM, respectively. Our major findings are as follows: First, when one character needs to be inserted or replaced, the model trained with CLM performs the best. Second, when more than one character needs to be handled, WWM is the key to better performance. Finally, when being fine-tuned on sentence-level downstream tasks, models trained with different masking strategies perform comparably.

cs.CL↗

arXiv2022

Pretraining without Wordpieces: Learning Over a Vocabulary of Millions of Words

The standard BERT adopts subword-based tokenization, which may break a word into two or more wordpieces (e.g., converting "lossless" to "loss" and "less"). This will bring inconvenience in following situations: (1) what is the best way to obtain the contextual vector of a word that is divided into multiple wordpieces? (2) how to predict a word via cloze test without knowing the number of wordpieces in advance? In this work, we explore the possibility of developing BERT-style pretrained model over a vocabulary of words instead of wordpieces. We call such word-level BERT model as WordBERT. We train models with different vocabulary sizes, initialization configurations and languages. Results show that, compared to standard wordpiece-based BERT, WordBERT makes significant improvements on cloze test and machine reading comprehension. On many other natural language understanding tasks, including POS tagging, chunking and NER, WordBERT consistently performs better than BERT. Model analysis indicates that the major advantage of WordBERT over BERT lies in the understanding for low-frequency words and rare words. Furthermore, since the pipeline is language-independent, we train WordBERT for Chinese language and obtain significant gains on five natural language understanding datasets. Lastly, the analyse on inference speed illustrates WordBERT has comparable time cost to BERT in natural language understanding tasks.

cs.CL↗

arXiv2022

Deep Performer: Score-to-Audio Music Performance Synthesis

Music performance synthesis aims to synthesize a musical score into a natural performance. In this paper, we borrow recent advances in text-to-speech synthesis and present the Deep Performer -- a novel system for score-to-audio music performance synthesis. Unlike speech, music often contains polyphony and long notes. Hence, we propose two new techniques for handling polyphonic inputs and providing a fine-grained conditioning in a transformer encoder-decoder model. To train our proposed system, we present a new violin dataset consisting of paired recordings and scores along with estimated alignments between them. We show that our proposed model can synthesize music with clear polyphony and harmonic structures. In a listening test, we achieve competitive quality against the baseline model, a conditional generative audio model, in terms of pitch accuracy, timbre and noise level. Moreover, our proposed model significantly outperforms the baseline on an existing piano dataset in overall quality.

cs.SD↗

arXiv2021

Semianalytical Approach for Sky Localization of Gravitational Waves

Rapid sky localization of gravitational wave sources is crucial to enable prompt electromagnetic follow-ups. In this article, we present a novel semianalytical approach for sky localization of gravitational waves from compact binary coalescences. We use the Bayesian framework with an analytical approximation to the prior distributions for a given astrophysical model. We derive a semianalytical solution to the posterior distribution of source directions. This method only requires one-fold numerical integral that marginalizes over the merger time, compared to the five-fold numerical integration otherwise needed in the Bayesian localization method. The performance of the method is demonstrated using a set of binary neutron stars (BNS) injections on Gaussian noise using LIGO-Virgo's design and O2 sensitivity. We find the median of 90% confidence area in O2 sensitivity to be $\mathcal{O}(10^2) ~\mathrm{deg}^2$, comparable to that of the existing LIGO-Virgo online localization method Bayestar and parameter estimation toolkit LALInference. In the end, we apply this method to localize the BNS event GW170817 and find the 50% (90%) confidence region of 11 $\mathrm{deg}^2$ (50 $\mathrm{deg}^2$). The detected optical counterpart of GW170817 resides within our 50% confidence area.

gr-qc↗

arXiv2020

Model-independent test of the parity symmetry of gravity with gravitational waves

Gravitational wave (GW) data can be used to test the parity symmetry of gravity by investigating the difference between left-hand and right-hand circular polarization modes. In this article, we develop a method to decompose the circular polarizations of GWs produced during the inspiralling stage of compact binaries, with the help of stationary phase approximation. The foremost advantage is that this method is simple, clean, independent of GW waveform, and is applicable to the existing detector network. Applying it to the mock data, we test the parity symmetry of gravity by constraining the velocity birefringence of GWs. If a nearly edge-on binary neutron-stars with observed electromagnetic counterparts at 40 Mpc is detected by the second-generation detector network, one could derive the model-independent test on the parity symmetry in gravity: the lower limit of the energy scale of parity violation can be constrained within $\mathcal{O}(10^4{\rm eV})$.

gr-qc↗

arXiv2020

On the Space of $C^1$ Regular Curves on Sphere with Constrained Curvature

Let $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$ denote the set of $C^1$ regular curves in the $2$-sphere $\mathbb{S}^2$ that start and end at given points with the corresponding Frenet frames $\boldsymbol{P}$ and $\boldsymbol{Q}$, whose tangent vectors are Lipschitz continuous, and their a.e. existing geodesic curvatures have essentially bounds in $(κ_1, κ_2)$, $-\infty<κ_1<κ_2<\infty$. In this article, firstly we study the geometric property of the curves in $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$. We introduce the concepts of the lower and upper curvatures at any point of a $C^1$ regular curve and prove that a $C^1$ regular curve is in $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$ if and only if the infimum of its lower curvature and the supremum of its upper curvature are constrained in $(κ_1,κ_2)$. Secondly we prove that the $C^0$ and $C^1$ topologies on $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$ are the same. Further, we show that a curve in $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$ can be determined by the solutions of differential equation $Φ'(t) = Φ(t)Λ(t)$ with $Φ(t)\in \textrm{SO}_3(\mathbb{R})$ with special constraints to $Λ(t)\in\mathfrak{so}_3(\mathbb{R})$ and give a complete metric on $\mathcal{P}_{κ_1}^{κ_2}(\boldsymbol{P}, \boldsymbol{Q})$ such that it becomes a (trivial) Banach manifold.

math.DG↗

arXiv2020

Source coding of audio signals with a generative model

We consider source coding of audio signals with the help of a generative model. We use a construction where a waveform is first quantized, yielding a finite bitrate representation. The waveform is then reconstructed by random sampling from a model conditioned on the quantized waveform. The proposed coding scheme is theoretically analyzed. Using SampleRNN as the generative model, we demonstrate that the proposed coding structure provides performance competitive with state-of-the-art source coding tools for specific categories of audio signals.

eess.AS↗

arXiv2019

Towards a simple, comprehensive model of regular earthquakes and slow slip events, part I: one-dimensional model

We have developed a model that describes the major characteristics of a rupture, ranging from regular earthquakes (EQs) to slow slip events (SSEs), including episodic tremor and slip (ETS). Previous model predictions, while accurate, are based on a highly idealized initial stress distribution and a simple velocity-dependent expression for friction. The full scope of the model has, therefore, not been fully demonstrated. Further developments, presented here, include more physically realistic treatments of both the initial conditions and friction. Model predictions are: (1) The type of a seismic event, i.e. regular EQ or SSE, is determined by the fault strength, the shear to normal stress ratio, and the gradient in the ratio. Quantitative values for these crucial parameters are also obtained here; (2) Rupture velocities for regular EQs range from a fraction of the shear wave velocity up to the supershear velocity. The maximum slip velocity for regular EQs is typically on the order of 1 m/s. For SSEs, the slip velocity ranges widely from a few cm/year up to 0.1 m/s. The magnitude of the stress drop may vary in the range from 1% to 10% of the initial shear stress for regular EQs and from 0.1% to 10% for the SSEs; (3) The rupture can expend as a crack-like mode or as a self-healing pulse mode. The type of rupture mode is determined by the stress ratio and its gradient. If stress heterogeneity has a step-like shape, the rupture is always crack-like. If the heterogeneity is localized, the rupture is crack-like for sufficiently large values of the stress and its gradient and is pulse-like for smaller values; (4) After an instability develops, rupture dynamics do not depend on the relative values of the rate-state a and b parameters, i.e., it does not matter if frictional sliding is velocity-weakening or velocity-strengthening. Models based on the rate-state are not consistent with this result.

physics.geo-ph↗

arXiv2018

High-quality speech coding with SampleRNN

We provide a speech coding scheme employing a generative model based on SampleRNN that, while operating at significantly lower bitrates, matches or surpasses the perceptual quality of state-of-the-art classic wide-band codecs. Moreover, it is demonstrated that the proposed scheme can provide a meaningful rate-distortion trade-off without retraining. We evaluate the proposed scheme in a series of listening tests and discuss limitations of the approach.

eess.AS↗

arXiv2018

On the Homology of the Space of Curves Immersed in The Sphere with Curvature Constrained to a Prescribed Interval

While the topology of the space of all smooth immersed curves on the $2$-sphere $\mathbb{S}^2$ that start and end at given points in given directions is well known, it is an open problem to understand the homotopy type of its subspaces consisting of the curves whose geodesic curvatures are constrained to a prescribed proper open interval. In this article we prove that, under certain circumstances for endpoints and end directions, these subspaces are not homotopically equivalent to the whole space. Moreover, we give an explicit construction of exotic generators for some homotopy and cohomology groups. It turns out that the dimensions of these generators depend on endpoints and end directions. A version of the h-principle is used to prove these results.

math.GT↗

arXiv2018

Voice Conversion with Conditional SampleRNN

Here we present a novel approach to conditioning the SampleRNN generative model for voice conversion (VC). Conventional methods for VC modify the perceived speaker identity by converting between source and target acoustic features. Our approach focuses on preserving voice content and depends on the generative network to learn voice style. We first train a multi-speaker SampleRNN model conditioned on linguistic features, pitch contour, and speaker identity using a multi-speaker speech corpus. Voice-converted speech is generated using linguistic features and pitch contour extracted from the source speaker, and the target speaker identity. We demonstrate that our system is capable of many-to-many voice conversion without requiring parallel data, enabling broad applications. Subjective evaluation demonstrates that our approach outperforms conventional VC methods.

cs.SD↗