arXiv subjects

Xing Wu

Publications and source records attributed to Xing Wu.

At least 73 records · Page 4Linked to original sources

arXiv2023

CoT-MAE v2: Contextual Masked Auto-Encoder with Multi-view Modeling for Passage Retrieval

Growing techniques have been emerging to improve the performance of passage retrieval. As an effective representation bottleneck pretraining technique, the contextual masked auto-encoder utilizes contextual embedding to assist in the reconstruction of passages. However, it only uses a single auto-encoding pre-task for dense representation pre-training. This study brings multi-view modeling to the contextual masked auto-encoder. Firstly, multi-view representation utilizes both dense and sparse vectors as multi-view representations, aiming to capture sentence semantics from different aspects. Moreover, multiview decoding paradigm utilizes both autoencoding and auto-regressive decoders in representation bottleneck pre-training, aiming to provide both reconstructive and generative signals for better contextual representation pretraining. We refer to this multi-view pretraining method as CoT-MAE v2. Through extensive experiments, we show that CoT-MAE v2 is effective and robust on large-scale passage retrieval benchmarks and out-of-domain zero-shot benchmarks.

cs.CL↗

arXiv2023

Ill-posedness for a generalized Camassa-Holm equation with higher-order nonlinearity in the critical Besov space

In this paper, we prove that the Cauchy problem for a generalized Camassa-Holm equation with higher-order nonlinearity is ill-posed in the critical Besov space $B^1_{\infty,1}(\R)$. It is shown in (J. Differ. Equ., 327:127-144,2022) that the Camassa-Holm equation is ill-posed in $B^1_{\infty,1}(\R)$, here we turn our attention to a higher-order nonlinear generalization of Camassa-Holm equation proposed by Hakkaev and Kirchev (Commun Partial Differ Equ 30:761-781,2005). With newly constructed initial data, we get the norm inflation in the critical space $B^1_{\infty,1}(\R)$ which leads to ill-posedness.

math.AP↗

arXiv2023

DistilCSE: Effective Knowledge Distillation For Contrastive Sentence Embeddings

Large-scale contrastive learning models can learn very informative sentence embeddings, but are hard to serve online due to the huge model size. Therefore, they often play the role of "teacher", transferring abilities to small "student" models through knowledge distillation. However, knowledge distillation inevitably brings some drop in embedding effect. To tackle that, we propose an effective knowledge distillation framework for contrastive sentence embeddings, termed DistilCSE. It first applies knowledge distillation on a large amount of unlabeled data, and then fine-tunes student models through contrastive learning on limited labeled data. To achieve better distillation results, we further propose Contrastive Knowledge Distillation (CKD). CKD uses InfoNCE as the loss function in knowledge distillation, enhancing the objective consistency among teacher model training, knowledge distillation, and student model fine-tuning. Extensive experiments show that student models trained with the proposed DistilCSE and CKD suffer from little or even no performance decrease and consistently outperform the corresponding counterparts of the same parameter size. Impressively, our 110M student model outperforms the latest state-of-the-art model, i.e., Sentence-T5 (11B), with only 1% parameters and 0.25% unlabeled data.

cs.AI↗

arXiv2022

ConTextual Masked Auto-Encoder for Dense Passage Retrieval

Dense passage retrieval aims to retrieve the relevant passages of a query from a large corpus based on dense representations (i.e., vectors) of the query and the passages. Recent studies have explored improving pre-trained language models to boost dense retrieval performance. This paper proposes CoT-MAE (ConTextual Masked Auto-Encoder), a simple yet effective generative pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric encoder-decoder architecture that learns to compress the sentence semantics into a dense vector through self-supervised and context-supervised masked auto-encoding. Precisely, self-supervised masked auto-encoding learns to model the semantics of the tokens inside a text span, and context-supervised masked auto-encoding learns to model the semantical correlation between the text spans. We conduct experiments on large-scale passage retrieval benchmarks and show considerable improvements over strong baselines, demonstrating the high efficiency of CoT-MAE. Our code is available at https://github.com/caskcsg/ir/tree/main/cotmae.

cs.CL↗

arXiv2022

High-sensitivity low-noise photodetector using large-area silicon photomultiplier

The application of silicon photomultiplier (SiPM) technology for weak-light detection at a single photon level has expanded thanks to its better photon detection efficiency in comparison to a conventional photomultiplier tube (PMT). SiPMs with large detection area have recently become commercially available, enabling applications where the photon flux is low both temporarily and spatially. On the other hand, several drawbacks exist in the usage of SiPMs such as a higher dark count rate, many readout channels, slow response time, and optical crosstalk; therefore, users need to carefully consider the trade-offs. This work presents a SiPM-embedded compact large-area photon detection module. Various techniques are adopted to overcome the disadvantages of SiPMs so that it can be generally utilized as an upgrade from a PMT. A simple cooling component and recently developed optical crosstalk suppression method are adopted to reduce the noise which is more serious for larger-area SiPMs. A dedicated readout circuit increases the response frequency and reduces the number of readout channels. We favorably compare this design with a conventional PMT and obtain both higher photon detection efficiency and larger-area acceptance.

physics.ins-det↗

arXiv2022

Electrostatic focusing of cold and heavy molecules for the ACME electron EDM search

The current best upper limit for electron electric dipole moment (EDM), $|d_e|<1.1\times10^{-29}\,$e$\cdot$cm ($90$% confidence), was set by the ACME collaboration in 2018. The ACME experiment uses a spin-precession measurement in a cold beam of ThO molecules to detect $d_e$. An improvement in statistical uncertainty would be possible with more efficient use of molecules from the cryogenic buffer gas beam source. Here, we demonstrate electrostatic focusing of the ThO beam with a hexapole lens. This results in a factor of $16$ enhancement in the molecular flux detectable downstream, in a beamline similar to that built for the next generation of ACME. We also demonstrate an upgraded rotational cooling scheme that increases the ground state population by $3.5$ times compared to no cooling, consistent with expectations and a factor of $1.4$ larger than previously in ACME. When combined with other demonstrated improvements, we project over an order of magnitude improvement in statistical sensitivity for the next generation ACME electron EDM search.

physics.atom-ph↗

arXiv2022

InfoCSE: Information-aggregated Contrastive Learning of Sentence Embeddings

Contrastive learning has been extensively studied in sentence embedding learning, which assumes that the embeddings of different views of the same sentence are closer. The constraint brought by this assumption is weak, and a good sentence representation should also be able to reconstruct the original sentence fragments. Therefore, this paper proposes an information-aggregated contrastive learning framework for learning unsupervised sentence embeddings, termed InfoCSE. InfoCSE forces the representation of [CLS] positions to aggregate denser sentence information by introducing an additional Masked language model task and a well-designed network. We evaluate the proposed InfoCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that InfoCSE outperforms SimCSE by an average Spearman correlation of 2.60% on BERT-base, and 1.77% on BERT-large, achieving state-of-the-art results among unsupervised sentence representation learning methods. Our code are available at https://github.com/caskcsg/sentemb/tree/main/InfoCSE.

cs.CL↗

arXiv2022

RaP: Redundancy-aware Video-language Pre-training for Text-Video Retrieval

Video language pre-training methods have mainly adopted sparse sampling techniques to alleviate the temporal redundancy of videos. Though effective, sparse sampling still suffers inter-modal redundancy: visual redundancy and textual redundancy. Compared with highly generalized text, sparsely sampled frames usually contain text-independent portions, called visual redundancy. Sparse sampling is also likely to miss important frames corresponding to some text portions, resulting in textual redundancy. Inter-modal redundancy leads to a mismatch of video and text information, hindering the model from better learning the shared semantics across modalities. To alleviate it, we propose Redundancy-aware Video-language Pre-training. We design a redundancy measurement of video patches and text tokens by calculating the cross-modal minimum dis-similarity. Then, we penalize the highredundant video patches and text tokens through a proposed redundancy-aware contrastive learning. We evaluate our method on four benchmark datasets, MSRVTT, MSVD, DiDeMo, and LSMDC, achieving a significant improvement over the previous stateof-the-art results. Our code are available at https://github.com/caskcsg/VLP/tree/main/RaP.

cs.CV↗

arXiv2022

Norm inflation and ill-posedness for the Fornberg-Whitham equation

In this paper, we prove that the Cauchy problem for the Fornberg-Whitham equation is not locally well-posed in $B^s_{p,r}(\R)$ with $(s,p,r)\in (1,1+\frac1p)\times[2,\infty)\times [1,\infty]$ or $(s,p,r)\in \{1\}\times[2,\infty)\times [1,2]$ by showing norm inflation phenomena of the solution for some special initial data.

math.AP↗

arXiv2022

Smoothed Contrastive Learning for Unsupervised Sentence Embedding

Contrastive learning has been gradually applied to learn high-quality unsupervised sentence embedding. Among the previous un-supervised methods, the latest state-of-the-art method, as far as we know, is unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE uses the InfoNCE1loss function in the training stage by pulling semantically similar sentences together and pushing apart dis-similar ones.Theoretically, we expect to use larger batches in unsup-SimCSE to get more adequate comparisons among samples and avoid overfitting. However, increasing the batch size does not always lead to improvements, but instead even lead to performance degradation when the batch size exceeds a threshold. Through statistical observation, we find that this is probably due to the introduction of low-confidence negative pairs after in-creasing the batch size. To alleviate this problem, we introduce a simple smoothing strategy upon the InfoNCE loss function, termedGaussian Smoothing InfoNCE (GS-InfoNCE).Specifically, we add random Gaussian noise vectors as negative samples, which act asa smoothing of the negative sample space.Though being simple, the proposed smooth-ing strategy brings substantial improvements to unsup-SimCSE. We evaluate GS-InfoNCEon the standard semantic text similarity (STS)task. GS-InfoNCE outperforms the state-of-the-art unsup-SimCSE by an average Spear-man correlation of 1.38%, 0.72%, 1.17% and0.28% on the base of BERT-base, BERT-large,RoBERTa-base and RoBERTa-large, respectively.

cs.CL↗

arXiv2022

ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding

Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as a minimal data augmentation method, and passes the same input sentence to a pre-trained Transformer encoder (with dropout turned on) twice to obtain the two corresponding embeddings to build a positive pair. As the length information of a sentence will generally be encoded into the sentence embeddings due to the usage of position embedding in Transformer, each positive pair in unsup-SimCSE actually contains the same length information. And thus unsup-SimCSE trained with these positive pairs is probably biased, which would tend to consider that sentences of the same or similar length are more similar in semantics. Through statistical observations, we find that unsup-SimCSE does have such a problem. To alleviate it, we apply a simple repetition operation to modify the input sentence, and then pass the input sentence and its modified counterpart to the pre-trained Transformer encoder, respectively, to get the positive pair. Additionally, we draw inspiration from the community of computer vision and introduce a momentum contrast, enlarging the number of negative pairs without additional calculations. The proposed two modifications are applied on positive and negative pairs separately, and build a new sentence embedding method, termed Enhanced Unsup-SimCSE (ESimCSE). We evaluate the proposed ESimCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that ESimCSE outperforms the state-of-the-art unsup-SimCSE by an average Spearman correlation of 2.02% on BERT-base.

cs.CL↗

arXiv2022

Stacked Autoencoder Based Multi-Omics Data Integration for Cancer Survival Prediction

Cancer survival prediction is important for developing personalized treatments and inducing disease-causing mechanisms. Multi-omics data integration is attracting widespread interest in cancer research for providing information for understanding cancer progression at multiple genetic levels. Many works, however, are limited because of the high dimensionality and heterogeneity of multi-omics data. In this paper, we propose a novel method to integrate multi-omics data for cancer survival prediction, called Stacked AutoEncoder-based Survival Prediction Neural Network (SAEsurv-net). In the cancer survival prediction for TCGA cases, SAEsurv-net addresses the curse of dimensionality with a two-stage dimensionality reduction strategy and handles multi-omics heterogeneity with a stacked autoencoder model. The two-stage dimensionality reduction strategy achieves a balance between computation complexity and information exploiting. The stacked autoencoder model removes most heterogeneities such as data's type and size in the first group of autoencoders, and integrates multiple omics data in the second autoencoder. The experiments show that SAEsurv-net outperforms models based on a single type of data as well as other state-of-the-art methods.

q-bio.GN↗

arXiv2022

Measurement of the H$^3Δ_1$ Radiative Lifetime in ThO

The best limit on the electron electric dipole moment (eEDM) comes from the ACME II experiment [Nature \textbf{562} (2018), 355-360] which probes physics beyond the Standard Model at energy scales well above 1 TeV. ACME II measured the eEDM by monitoring electron spin precession in a cold beam of the metastable H$^3Δ_1$ state of thorium monoxide (ThO) molecules, with an observation time $τ\approx 1$ ms for each molecule. We report here a new measurement of the lifetime of the ThO (H$^3Δ_1$) state, $τ_H = 4.2\pm 0.5$ ms. Using an apparatus within which $τ\approx τ_H$ will enable a substantial reduction in uncertainty of an eEDM measurement.

physics.atom-ph↗

arXiv2022

Text Smoothing: Enhance Various Data Augmentation Methods on Text Classification Tasks

Before entering the neural network, a token is generally converted to the corresponding one-hot representation, which is a discrete distribution of the vocabulary. Smoothed representation is the probability of candidate tokens obtained from a pre-trained masked language model, which can be seen as a more informative substitution to the one-hot representation. We propose an efficient data augmentation method, termed text smoothing, by converting a sentence from its one-hot representation to a controllable smoothed representation. We evaluate text smoothing on different benchmarks in a low-resource regime. Experimental results show that text smoothing outperforms various mainstream data augmentation methods by a substantial margin. Moreover, text smoothing can be combined with those data augmentation methods to achieve better performance.

cs.CL↗

arXiv2022

Ill_posedness for a two_component Novikov system in Besov space

In this paper, we consider the Cauchy problem for a two-component Novikov system on the line. By specially constructed initial data $(ρ_0, u_0)$ in $B_{p, \infty}^{s-1}(\mathbb{R})\times B_{p, \infty}^s(\mathbb{R})$ with $s>\max\{2+\frac{1}{p}, \frac{5}{2}\}$ and $1\leq p \leq \infty$, we show that any energy bounded solution starting from $(ρ_0, u_0)$ does not converge back to $(ρ_0, u_0)$ in the metric of $B_{p, \infty}^{s-1}(\mathbb{R})\times B_{p, \infty}^s(\mathbb{R})$ as time goes to zero, thus results in discontinuity of the data-to-solution map and ill-posedness.

math.AP↗

arXiv2021

Suppression of the optical crosstalk in a multi-channel silicon photomultiplier array

We propose and study a method of optical crosstalk suppression for silicon photomultipliers (SiPMs) using optical filters. We demonstrate that attaching absorptive visible bandpass filters to the SiPM can substantially reduce the optical crosstalk. Measurements suggest that the absorption of near infrared light is important to achieve this suppression. The proposed technique can be easily applied to suppress the optical crosstalk in SiPMs in cases where filtering near infrared light is compatible with the application.

physics.ins-det↗

arXiv2021

Non-uniform continuity on initial data for the two-component b-family system in Besov space

In this paper, we consider the Cauchy problem of a two-component b-family system, which includes the two-component Camassa-Holm system and the two-component Degasperis-Procesi system. It is shown that the solution map of the two-component b-family system is not uniformly continuous on the initial data in Besov spaces $B_{p, r}^{s-1}(\mathbb{R})\times B_{p, r}^s(\mathbb{R})$ with $s>\max\{1+\frac{1}{p}, \frac{3}{2}\}$, $1\leq p, r< \infty$. Our result covers and extends the previous non-uniform continuity in Sobolev spaces $H^{s-1}(\mathbb{R})\times H^s(\mathbb{R})$ for $s>\frac{5}{2}$ (Nonlinear Anal., 2014) to Besov spaces.

math.AP↗

arXiv2021

Non-uniform dependence on initial data for the generalized Camassa-Holm-Novikov equation in Besov space

Considered in this paper is the generalized Camassa-Holm-Novikov equation with high order nonlinearity, which unifies the Camassa-Holm and Novikov equations as special cases. We show that the solution map of generalized Camassa-Holm-Novikov equation is not uniformly continuous on the initial data in Besov spaces $B_{p, r}^s(\mathbb{R})$ with $s>\max\{1+\frac{1}{p}, \frac{3}{2}\}$, $1\leq p, r< \infty$ as well as in critical space $B_{2, 1}^{\frac{3}{2}}(\mathbb{R}).$ Our result covers and improves the previous work given by Li et al. \cite{Li 2020, 1Li 2020, Li 2021}(J. Differ. Equ. 269 (2020) 8686-8700; J. Math. Fluid Mech. 22 (2020) 4:50; J. Math. Fluid Mech., (2021) 23:36).

math.AP↗