arXiv subjects

Xu Li

Publications and source records attributed to Xu Li.

At least 91 records · Page 5Linked to original sources

arXiv2024

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

Large Vision-Language Model (LVLM) systems have demonstrated impressive vision-language reasoning capabilities but suffer from pervasive and severe hallucination issues, posing significant risks in critical domains such as healthcare and autonomous systems. Despite previous efforts to mitigate hallucinations, a persistent issue remains: visual defect from vision-language misalignment, creating a bottleneck in visual processing capacity. To address this challenge, we develop Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs (CATCH), based on the Information Bottleneck theory. CATCH introduces Complementary Visual Decoupling (CVD) for visual information separation, Non-Visual Screening (NVS) for hallucination detection, and Adaptive Token-level Contrastive Decoding (ATCD) for hallucination mitigation. CATCH addresses issues related to visual defects that cause diminished fine-grained feature perception and cumulative hallucinations in open-ended scenarios. It is applicable to various visual question-answering tasks without requiring any specific data or prior knowledge, and generalizes robustly to new tasks without additional training, opening new possibilities for advancing LVLM in various challenging applications.

cs.CV↗

arXiv2024

SIGGesture: Generalized Co-Speech Gesture Synthesis via Semantic Injection with Large-Scale Pre-Training Diffusion Models

The automated synthesis of high-quality 3D gestures from speech is of significant value in virtual humans and gaming. Previous methods focus on synthesizing gestures that are synchronized with speech rhythm, yet they frequently overlook the inclusion of semantic gestures. These are sparse and follow a long-tailed distribution across the gesture sequence, making them difficult to learn in an end-to-end manner. Moreover, generating gestures, rhythmically aligned with speech, faces a significant issue that cannot be generalized to in-the-wild speeches. To address these issues, we introduce SIGGesture, a novel diffusion-based approach for synthesizing realistic gestures that are of both high quality and semantically pertinent. Specifically, we firstly build a strong diffusion-based foundation model for rhythmical gesture synthesis by pre-training it on a collected large-scale dataset with pseudo labels. Secondly, we leverage the powerful generalization capabilities of Large Language Models (LLMs) to generate proper semantic gestures for the various speech content. Finally, we propose a semantic injection module to infuse semantic information into the synthesized results during diffusion reverse process. Extensive experiments demonstrate that the proposed SIGGesture significantly outperforms existing baselines and shows excellent generalization and controllability.

cs.HC↗

arXiv2024

Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility

Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative model for this task. As other models of its kind, MaskSR attains high quality but, as we show, intelligibility can be substantially improved. We do so by boosting the speech encoder component of MaskSR with predictions of semantic representations of the target speech, using a pre-trained self-supervised teacher model. Then, a masked language model is conditioned on the learned semantic features to predict acoustic tokens that encode low level spectral details of the target speech. We show that, with the same MaskSR model capacity and inference time, the proposed model, MaskSR2, significantly reduces the word error rate, a typical metric for intelligibility. MaskSR2 also achieves competitive word error rate among other models, while providing superior quality. An ablation study shows the effectiveness of various semantic representations.

cs.SD↗

arXiv2024

Dynamic Resolution Guidance for Facial Expression Recognition

Facial expression recognition (FER) is vital for human-computer interaction and emotion analysis, yet recognizing expressions in low-resolution images remains challenging. This paper introduces a practical method called Dynamic Resolution Guidance for Facial Expression Recognition (DRGFER) to effectively recognize facial expressions in images with varying resolutions without compromising FER model accuracy. Our framework comprises two main components: the Resolution Recognition Network (RRN) and the Multi-Resolution Adaptation Facial Expression Recognition Network (MRAFER). The RRN determines image resolution, outputs a binary vector, and the MRAFER assigns images to suitable facial expression recognition networks based on resolution. We evaluated DRGFER on widely-used datasets RAFDB and FERPlus, demonstrating that our method retains optimal model performance at each resolution and outperforms alternative resolution approaches. The proposed framework exhibits robustness against resolution variations and facial expressions, offering a promising solution for real-world applications.

cs.CV↗

arXiv2024

DFML: Decentralized Federated Mutual Learning

In the realm of real-world devices, centralized servers in Federated Learning (FL) present challenges including communication bottlenecks and susceptibility to a single point of failure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions or assuming the availability of public data. To address these issues, we propose a Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogeneous models, and avoids reliance on public data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cyclically varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 dataset and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.

cs.LG↗

arXiv2024

EA-VTR: Event-Aware Video-Text Retrieval

Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and the widely adopted video-level cross-modal contrastive learning also struggles to capture detailed and complex video-text event alignment. To address these challenges, we make improvements from both data and model perspectives. In terms of pre-training data, we focus on supplementing the missing specific event content and event temporal transitions with the proposed event augmentation strategies. Based on the event-augmented data, we construct a novel Event-Aware Video-Text Retrieval model, ie, EA-VTR, which achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events. Our method not only significantly outperforms existing approaches on multiple datasets for Text-to-Video Retrieval and Video Action Recognition tasks, but also demonstrates superior event content perceive ability on Multi-event Video-Text Retrieval and Video Moment Retrieval tasks, as well as outstanding event temporal logic understanding ability on Test of Time task.

cs.CV↗

arXiv2024

Improving the Adversarial Robustness for Speaker Verification by Self-Supervised Learning

Previous works have shown that automatic speaker verification (ASV) is seriously vulnerable to malicious spoofing attacks, such as replay, synthetic speech, and recently emerged adversarial attacks. Great efforts have been dedicated to defending ASV against replay and synthetic speech; however, only a few approaches have been explored to deal with adversarial attacks. All the existing approaches to tackle adversarial attacks for ASV require the knowledge for adversarial samples generation, but it is impractical for defenders to know the exact attack algorithms that are applied by the in-the-wild attackers. This work is among the first to perform adversarial defense for ASV without knowing the specific attack algorithms. Inspired by self-supervised learning models (SSLMs) that possess the merits of alleviating the superficial noise in the inputs and reconstructing clean samples from the interrupted ones, this work regards adversarial perturbations as one kind of noise and conducts adversarial defense for ASV by SSLMs. Specifically, we propose to perform adversarial defense from two perspectives: 1) adversarial perturbation purification and 2) adversarial perturbation detection. Experimental results show that our detection module effectively shields the ASV by detecting adversarial samples with an accuracy of around 80%. Moreover, since there is no common metric for evaluating the adversarial defense performance for ASV, this work also formalizes evaluation metrics for adversarial defense considering both purification and detection based approaches into account. We sincerely encourage future works to benchmark their approaches based on the proposed evaluation framework.

cs.SD↗

arXiv2024

MaskSR: Masked Language Model for Full-band Speech Restoration

Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. Although several deep learning paradigms have been studied for this task, the power of the recently emerging language models has not been fully explored. In this paper, we propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth. MaskSR works with discrete acoustic tokens extracted using a pre-trained neural codec. During training, MaskSR is optimized to predict randomly masked tokens extracted from the high quality target speech, conditioned on the corrupted speech with various distortions. During inference, MaskSR reconstructs the target speech tokens with efficient iterative sampling. Extensive experiments show that MaskSR obtains competitive results on both the full-band speech restoration task and also on sub-tasks compared with a wide range of models.

cs.SD↗

arXiv2024

CSST Strong Lensing Preparation: a Framework for Detecting Strong Lenses in the Multi-color Imaging Survey by the China Survey Space Telescope (CSST)

Strong gravitational lensing is a powerful tool for investigating dark matter and dark energy properties. With the advent of large-scale sky surveys, we can discover strong lensing systems on an unprecedented scale, which requires efficient tools to extract them from billions of astronomical objects. The existing mainstream lens-finding tools are based on machine learning algorithms and applied to cut-out-centered galaxies. However, according to the design and survey strategy of optical surveys by CSST, preparing cutouts with multiple bands requires considerable efforts. To overcome these challenges, we have developed a framework based on a hierarchical visual Transformer with a sliding window technique to search for strong lensing systems within entire images. Moreover, given that multi-color images of strong lensing systems can provide insights into their physical characteristics, our framework is specifically crafted to identify strong lensing systems in images with any number of channels. As evaluated using CSST mock data based on an Semi-Analytic Model named CosmoDC2, our framework achieves precision and recall rates of 0.98 and 0.90, respectively. To evaluate the effectiveness of our method in real observations, we have applied it to a subset of images from the DESI Legacy Imaging Surveys and media images from Euclid Early Release Observations. 61 new strong lensing system candidates are discovered by our method. However, we also identified false positives arising primarily from the simplified galaxy morphology assumptions within the simulation. This underscores the practical limitations of our approach while simultaneously highlighting potential avenues for future improvements.

astro-ph.IM↗

arXiv2024

AFL-Net: Integrating Audio, Facial, and Lip Modalities with a Two-step Cross-attention for Robust Speaker Diarization in the Wild

Speaker diarization in real-world videos presents significant challenges due to varying acoustic conditions, diverse scenes, the presence of off-screen speakers, etc. This paper builds upon a previous study (AVR-Net) and introduces a novel multi-modal speaker diarization system, AFL-Net. The proposed AFL-Net incorporates dynamic lip movement as an additional modality to enhance the identity distinction. Besides, unlike AVR-Net which extracts high-level representations from each modality independently, AFL-Net employs a two-step cross-attention mechanism to sufficiently fuse different modalities, resulting in more comprehensive information to enhance the performance. Moreover, we also incorporated a masking strategy during training, where the face and lip modalities are randomly obscured. This strategy enhances the impact of the audio modality on the system outputs. Experimental results demonstrate that AFL-Net outperforms state-of-the-art baselines, such as the AVR-Net and DyViSE.

cs.MM↗

arXiv2024

Ultrafast switching of sliding ferroelectricity and dynamical magnetic field in van der Waals bilayer induced by light

Sliding ferroelectricity is a unique type of polarity recently observed in a properly stacked van der Waals bilayer. However, electric-field control of sliding ferroelectricity is hard and could induce large coercive electric fields and serious leakage currents which corrode the ferroelectricity and electronic properties, which are essential for modern two-dimensional electronics and optoelectronics. Here, we proposed laser-pulse deterministic control of sliding ferroelectricity in bilayer h-BN by first principles and molecular dynamics simulation with machine-learned force fields. The laser pulses excite shear modes which exhibit certain directional movements of lateral sliding between bilayers. The vibration of excited modes under laser pulses is predicted to overcome the energy barrier and achieve the switching of sliding ferroelectricity. Furthermore, it is found that three possible sliding transitions - between AB (BA) and BA (AB) stacking - can lead to the occurrence of dynamical magnetic fields along three different directions. Remarkably, the magnetic fields are generated by the simple linear motion of nonmagnetic species, without any need for more exotic (circular, spiral) pathways. Such predictions of deterministic control of sliding ferroelectricity and multi-states of dynamical magnetic field thus expand the potential applications of sliding ferroelectricity in memory and electronic devices.

cond-mat.mtrl-sci↗

arXiv2024

Positivity restrictions on the mixing of dimension-eight SMEFT operators

We discuss the structure of the mixing among dimension-eight operators in the SMEFT relying on the positivity of two-to-two forward scattering amplitudes. We uncover tens of new non-trivial zeros as well as hundreds of terms with definite sign in (a particular basis of) the corresponding anomalous dimension matrix. We highlight that our results are not immediately apparent from the Feynman diagrammatic perspective, nor from on-shell amplitude methods. As a byproduct of this work, we provide positivity bounds not previously derived in the literature, as well as explicit values of certain elements of the anomalous dimension matrix that serve for cross-check of our results.

hep-ph↗

arXiv2024

Parametric Feature Transfer: One-shot Federated Learning with Foundation Models

In one-shot federated learning (FL), clients collaboratively train a global model in a single round of communication. Existing approaches for one-shot FL enhance communication efficiency at the expense of diminished accuracy. This paper introduces FedPFT (Federated Learning with Parametric Feature Transfer), a methodology that harnesses the transferability of foundation models to enhance both accuracy and communication efficiency in one-shot FL. The approach involves transferring per-client parametric models (specifically, Gaussian mixtures) of features extracted from foundation models. Subsequently, each parametric model is employed to generate synthetic features for training a classifier head. Experimental results on eight datasets demonstrate that FedPFT enhances the communication-accuracy frontier in both centralized and decentralized FL scenarios, as well as across diverse data-heterogeneity settings such as covariate shift and task shift, with improvements of up to 20.6%. Additionally, FedPFT adheres to the data minimization principle of FL, as clients do not send real features. We demonstrate that sending real features is vulnerable to potent reconstruction attacks. Moreover, we show that FedPFT is amenable to formal privacy guarantees via differential privacy, demonstrating favourable privacy-accuracy tradeoffs.

cs.LG↗

arXiv2024

Analysis of a $\boldsymbol{P}_1\oplus \boldsymbol{RT}_0$ finite element method for linear elasticity with Dirichlet and mixed boundary conditions

In this paper, we investigate a low-order robust numerical method for the linear elasticity problem. The method is based on a Bernardi--Raugel-like $\boldsymbol{H}(\mathrm{div})$-conforming method proposed first for the Stokes flows in [Li and Rui, IMA J. Numer. Anal. {42} (2022) 3711--3734].Therein the lowest-order $\boldsymbol{H}(\mathrm{div})$-conforming Raviart--Thomas space ($\boldsymbol{RT}_0$) was added to the classical conforming $\boldsymbol{P}_1\times P_0$ pair to meet the inf-sup condition, while preserving the divergence constraint and some important features of conforming methods. Due to the inf-sup stability of {the} $\boldsymbol{P}_1\oplus \boldsymbol{RT}_0\times P_0$ pair, a locking-free elasticity discretization {with respect to} {the Lamé constant $λ$} can be naturally obtained. Moreover, our scheme is gradient-robust for the pure and homogeneous displacement boundary problem, that is, the discrete $\boldsymbol{H}^1$-norm of the displacement is $\mathcal{O}(λ^{-1})$ when the external body force is a gradient field. We also consider the mixed displacement and stress boundary problem, whose $\boldsymbol{P}_1\oplus \boldsymbol{RT}_0$ discretization should be carefully designed due to a consistency error arising from the $\boldsymbol{RT}_0$ part. We propose both symmetric and nonsymmetric schemes to approximate the mixed boundary case. The optimal error estimates are derived for the energy norm and/or $\boldsymbol{L}^2$-norm. Numerical experiments demonstrate the accuracy and robustness of our schemes.

math.NA↗

arXiv2024

Neural Concatenative Singing Voice Conversion: Rethinking Concatenation-Based Approach for One-Shot Singing Voice Conversion

Any-to-any singing voice conversion (SVC) is confronted with the challenge of ``timbre leakage'' issue caused by inadequate disentanglement between the content and the speaker timbre. To address this issue, this study introduces NeuCoSVC, a novel neural concatenative SVC framework. It consists of a self-supervised learning (SSL) representation extractor, a neural harmonic signal generator, and a waveform synthesizer. The SSL extractor condenses audio into fixed-dimensional SSL features, while the harmonic signal generator leverages linear time-varying filters to produce both raw and filtered harmonic signals for pitch information. The synthesizer reconstructs waveforms using SSL features, harmonic signals, and loudness information. During inference, voice conversion is performed by substituting source SSL features with their nearest counterparts from a matching pool which comprises SSL features extracted from the reference audio, while preserving raw harmonic signals and loudness from the source audio. By directly utilizing SSL features from the reference audio, the proposed framework effectively resolves the ``timbre leakage" issue caused by previous disentanglement-based approaches. Experimental results demonstrate that the proposed NeuCoSVC system outperforms the disentanglement-based speaker embedding approach in one-shot SVC across intra-language, cross-language, and cross-domain evaluations.

cs.SD↗

arXiv2023

AI Techniques for Uncovering Resolved Planetary Nebula Candidates from Wide-field VPHAS+ Survey Data

AI and deep learning techniques are beginning to play an increasing role in astronomy as a necessary tool to deal with the data avalanche. Here we describe an application for finding resolved Planetary Nebulae (PNe) in crowded, wide-field, narrow-band H-alpha survey imagery in the Galactic plane. PNe are important to study late stage of stellar evolution of low to intermediate-mass stars. However, the confirmed ~3800 Galactic PNe fall far short of the numbers expected. Traditional visual searching for resolved PNe is time-consuming due to the large data size and areal coverage of modern astronomical surveys, especially those taken in narrow-band filters highlighting emission nebulae. To test and facilitate more objective, reproducible, efficient and reliable trawls for PNe candidates we have developed a new, deep learning algorithm. In this paper, we applied the algorithm to several H-alpha digital surveys (e.g. IPHAS and VPHAS+). The training and validation dataset was built with true PNe from the HASH database. After transfer learning, it was then applied to the VPHAS+ survey. We examined 979 out of 2284 survey fields with each survey field covering 1 * 1 deg^2. With a sample of 454 PNe from the IPHAS as our validation set, our algorithm correctly identified 444 of these objects (97.8%), with only 16 explicable 'false' positives. Our model returned ~20,000 detections, including 2637 known PNe and many other kinds of catalogued non-PNe such as HII regions. A total of 815 new high-quality PNe candidates were found, 31 of which were selected as top-quality targets for subsequent optical spectroscopic follow-up. Representative preliminary confirmatory spectroscopy results are presented here to demonstrate the effectiveness of our techniques with full details to be given in paper-II.

astro-ph.IM↗

arXiv2023

Recovering Linear Causal Models with Latent Variables via Cholesky Factorization of Covariance Matrix

Discovering the causal relationship via recovering the directed acyclic graph (DAG) structure from the observed data is a well-known challenging combinatorial problem. When there are latent variables, the problem becomes even more difficult. In this paper, we first propose a DAG structure recovering algorithm, which is based on the Cholesky factorization of the covariance matrix of the observed data. The algorithm is fast and easy to implement and has theoretical grantees for exact recovery. On synthetic and real-world datasets, the algorithm is significantly faster than previous methods and achieves the state-of-the-art performance. Furthermore, under the equal error variances assumption, we incorporate an optimization procedure into the Cholesky factorization based algorithm to handle the DAG recovering problem with latent variables. Numerical simulations show that the modified "Cholesky + optimization" algorithm is able to recover the ground truth graph in most cases and outperforms existing algorithms.

stat.ML↗

arXiv2023

HumTrans: A Novel Open-Source Dataset for Humming Melody Transcription and Beyond

This paper introduces the HumTrans dataset, which is publicly available and primarily designed for humming melody transcription. The dataset can also serve as a foundation for downstream tasks such as humming melody based music generation. It consists of 500 musical compositions of different genres and languages, with each composition divided into multiple segments. In total, the dataset comprises 1000 music segments. To collect this humming dataset, we employed 10 college students, all of whom are either music majors or proficient in playing at least one musical instrument. Each of them hummed every segment twice using the web recording interface provided by our designed website. The humming recordings were sampled at a frequency of 44,100 Hz. During the humming session, the main interface provides a musical score for students to reference, with the melody audio playing simultaneously to aid in capturing both melody and rhythm. The dataset encompasses approximately 56.22 hours of audio, making it the largest known humming dataset to date. The dataset will be released on Hugging Face, and we will provide a GitHub repository containing baseline results and evaluation codes.

cs.SD↗