arXiv subjects

Yi Lu

Publications and source records attributed to Yi Lu.

At least 145 records · Page 8Linked to original sources

arXiv2024

MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation

Foley audio, critical for enhancing the immersive experience in multimedia content, faces significant challenges in the AI-generated content (AIGC) landscape. Despite advancements in AIGC technologies for text and image generation, the foley audio dubbing remains rudimentary due to difficulties in cross-modal scene matching and content correlation. Current text-to-audio technology, which relies on detailed and acoustically relevant textual descriptions, falls short in practical video dubbing applications. Existing datasets like AudioSet, AudioCaps, Clotho, Sound-of-Story, and WavCaps do not fully meet the requirements for real-world foley audio dubbing task. To address this, we introduce the Multi-modal Image and Narrative Text Dubbing Dataset (MINT), designed to enhance mainstream dubbing tasks such as literary story audiobooks dubbing, image/silent video dubbing. Besides, to address the limitations of existing TTA technology in understanding and planning complex prompts, a Foley Audio Content Planning, Generation, and Alignment (CPGA) framework is proposed, which includes a content planning module leveraging large language models for complex multi-modal prompts comprehension. Additionally, the training process is optimized using Proximal Policy Optimization based reinforcement learning, significantly improving the alignment and auditory realism of generated foley audio. Experimental results demonstrate that our approach significantly advances the field of foley audio dubbing, providing robust solutions for the challenges of multi-modal dubbing. Even when utilizing the relatively lightweight GPT-2 model, our framework outperforms open-source multimodal large models such as LLaVA, DeepSeek-VL, and Moondream2. The dataset is available at https://github.com/borisfrb/MINT .

eess.AS↗

arXiv2024

Codecfake: An Initial Dataset for Detecting LLM-based Deepfake Audio

With the proliferation of Large Language Model (LLM) based deepfake audio, there is an urgent need for effective detection methods. Previous deepfake audio generation methods typically involve a multi-step generation process, with the final step using a vocoder to predict the waveform from handcrafted features. However, LLM-based audio is directly generated from discrete neural codecs in an end-to-end generation process, skipping the final step of vocoder processing. This poses a significant challenge for current audio deepfake detection (ADD) models based on vocoder artifacts. To effectively detect LLM-based deepfake audio, we focus on the core of the generation process, the conversion from neural codec to waveform. We propose Codecfake dataset, which is generated by seven representative neural codec methods. Experiment results show that codec-trained ADD models exhibit a 41.406% reduction in average equal error rate compared to vocoder-trained ADD models on the Codecfake test set.

cs.SD↗

arXiv2024

Genuine-Focused Learning using Mask AutoEncoder for Generalized Fake Audio Detection

The generalization of Fake Audio Detection (FAD) is critical due to the emergence of new spoofing techniques. Traditional FAD methods often focus solely on distinguishing between genuine and known spoofed audio. We propose a Genuine-Focused Learning (GFL) framework guided, aiming for highly generalized FAD, called GFL-FAD. This method incorporates a Counterfactual Reasoning Enhanced Representation (CRER) based on audio reconstruction using the Mask AutoEncoder (MAE) architecture to accurately model genuine audio features. To reduce the influence of spoofed audio during training, we introduce a genuine audio reconstruction loss, maintaining the focus on learning genuine data features. In addition, content-related bottleneck (BN) features are extracted from the MAE to supplement the knowledge of the original audio. These BN features are adaptively fused with CRER to further improve robustness. Our method achieves state-of-the-art performance with an EER of 0.25% on ASVspoof2019 LA.

cs.SD↗

arXiv2024

PPPR: Portable Plug-in Prompt Refiner for Text to Audio Generation

Text-to-Audio (TTA) aims to generate audio that corresponds to the given text description, playing a crucial role in media production. The text descriptions in TTA datasets lack rich variations and diversity, resulting in a drop in TTA model performance when faced with complex text. To address this issue, we propose a method called Portable Plug-in Prompt Refiner, which utilizes rich knowledge about textual descriptions inherent in large language models to effectively enhance the robustness of TTA acoustic models without altering the acoustic training set. Furthermore, a Chain-of-Thought that mimics human verification is introduced to enhance the accuracy of audio descriptions, thereby improving the accuracy of generated content in practical applications. The experiments show that our method achieves a state-of-the-art Inception Score (IS) of 8.72, surpassing AudioGen, AudioLDM and Tango.

cs.SD↗

arXiv2024

Generalized Fake Audio Detection via Deep Stable Learning

Although current fake audio detection approaches have achieved remarkable success on specific datasets, they often fail when evaluated with datasets from different distributions. Previous studies typically address distribution shift by focusing on using extra data or applying extra loss restrictions during training. However, these methods either require a substantial amount of data or complicate the training process. In this work, we propose a stable learning-based training scheme that involves a Sample Weight Learning (SWL) module, addressing distribution shift by decorrelating all selected features via learning weights from training samples. The proposed portable plug-in-like SWL is easy to apply to multiple base models and generalizes them without using extra data during training. Experiments conducted on the ASVspoof datasets clearly demonstrate the effectiveness of SWL in generalizing different models across three evaluation datasets from different distributions.

cs.SD↗

arXiv2024

Dynamical correlation functions from complex time evolution

We present an approach to tame the growth of entanglement during time evolution by tensor network methods. It combines time evolution in the complex plane with a perturbative and controlled reconstruction of correlation functions on the real-time axis. We benchmark our approach on the single impurity Anderson model. Compared to purely real-time evolution, the complex time evolution significantly reduces the required bond dimension to obtain the spectral function. Notably, our approach yields self-energy results with high precision at low frequencies, comparable to numerical renormalization group (NRG) results, and it successfully captures the exponentially small Kondo energy scale.

cond-mat.str-el↗

arXiv2024

Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories

Image aesthetic evaluation is a highly prominent research domain in the field of computer vision. In recent years, there has been a proliferation of datasets and corresponding evaluation methodologies for assessing the aesthetic quality of photographic works, leading to the establishment of a relatively mature research environment. However, in contrast to the extensive research in photographic aesthetics, the field of aesthetic evaluation for paintings and Drawings has seen limited attention until the introduction of the BAID dataset in March 2023. This dataset solely comprises overall scores for high-quality artistic images. Our research marks the pioneering introduction of a multi-attribute, multi-category dataset specifically tailored to the field of painting: Aesthetics of Paintings and Drawings Dataset (APDD). The construction of APDD received active participation from 28 professional artists worldwide, along with dozens of students specializing in the field of art. This dataset encompasses 24 distinct artistic categories and 10 different aesthetic attributes. Each image in APDD has been evaluated by six professionally trained experts in the field of art, including assessments for both total aesthetic scores and aesthetic attribute scores. The final APDD dataset comprises a total of 4985 images, with an annotation count exceeding 31100 entries. Concurrently, we propose an innovative approach: Art Assessment Network for Specific Painting Styles (AANSPS), designed for the assessment of aesthetic attributes in mixed-attribute art datasets. Through this research, our goal is to catalyze advancements in the field of aesthetic evaluation for paintings and drawings, while enriching the available resources and methodologies for its further development and application.

cs.CV↗

arXiv2024

The effective potential of composite operator in the first order region of QCD phase transition

We propose a method to determine the effective potential of QCD from the gap equation, by introducing the homotopy method between the solutions of the equation of motion. Via this method, the effective potential beyond the bare vertex approximation is obtained, which then generalizes the Cornwall, Jackiw and Tomboulis (CJT) effective potential for the bilocal composite operators. Moreover, the extended effective potential is set to be a function of self energy instead of the propagator, which is the key point for the potential to be bounded from below. We then investigate the extended effective potential in the cases of phase transition of the QCD vacuum with a small current quark mass, and the first-order phase transition of QCD at finite temperature and high baryon chemical potential. In the former case, the effective potential shows as an inflection point at the critical mass where the multiple solutions of the Dyson-Schwinger equation (DSE) vanishes, which is consistent with that obtained by solving the DSE directly. For the latter case, the in-medium properties, such as the latent heat and the difference of trace anomaly, of QCD is obtained.

hep-ph↗

arXiv2024

Probing the topological phase transition in the Su-Schrieffer-Heeger model using Rydberg-atom synthetic dimensions

We simulate the the Su-Schrieffer-Heeger (SSH) model using Rydberg-atom synthetic dimensions constructed, in a single atom, from a ladder of six neighboring $n\:^3S_1$ Rydberg states in which adjacent states are coupled with two-photon transitions using microwave fields. Alternating strong/weak tunneling rates, controlled by adjusting the microwave amplitudes, are varied to map out the topological phase transition as a function of the ratio of the tunneling rates. For each ratio, quench dynamics experiments, in which the system is initially prepared in one of the bulk Rydberg states and then subjected to the microwave fields, are performed to measure the population evolution of the system. From the dynamics measurements, we extract the mean chiral displacement and verify that its long-time average value converges towards the system winding number. The topological phase transition is also examined by probing the energy spectrum of the system in steady state and observing the disappearance of the zero-energy edge states. The results show that even a system with as few as six levels can demonstrate the essential characteristics of the SSH Hamiltonian.

quant-ph↗

arXiv2024

TeleAware Robot: Designing Awareness-augmented Telepresence Robot for Remote Collaborative Locomotion

Telepresence robots can be used to support users to navigate an environment remotely and share the visiting experience with their social partners. Although such systems allow users to see and hear the remote environment and communicate with their partners via live video feed, this does not provide enough awareness of the environment and their remote partner's activities. In this paper, we introduce an awareness framework for collaborative locomotion in scenarios of onsite and remote users visiting a place together. From an observational study of small groups of people visiting exhibitions, we derived four design goals for enhancing the environmental and social awareness between social partners, and developed a set of awareness-enhancing techniques to add to a standard telepresence robot - named TeleAware robot. Through a controlled experiment simulating a guided exhibition visiting task, TeleAware robot showed the ability to lower the workload, facilitate closer social proximity, and improve mutual awareness and social presence compared with the standard one. We discuss the impact of mobility and roles of local and remote users, and provide insights for the future design of awareness-enhancing telepresence robot systems that facilitate collaborative locomotion.

cs.HC↗

arXiv2024

Electronic correlations and partial gap in the bilayer nickelate La$_{3}$Ni$_{2}$O$_{7}$

The discovery of superconductivity with a critical temperature of about 80~K in La$_{3}$Ni$_{2}$O$_{7}$ single crystals under pressure has received enormous attention. La$_{3}$Ni$_{2}$O$_{7}$ is not superconducting under ambient pressure but exhibits a transition at $T^{\ast} \simeq 115$~K. Understanding the electronic correlations and charge dynamics is an important step towards the origin of superconductivity and other instabilities. Here, our optical study shows that La$_{3}$Ni$_{2}$O$_{7}$ features strong electronic correlations which significantly reduce the electron's kinetic energy and place this system in the proximity of the Mott phase. The low-frequency optical conductivity reveals two Drude components arising from multiple bands at the Fermi level. The transition at $T^{\ast}$ removes the Drude component exhibiting non-Fermi liquid behavior, whereas the one with Fermi-liquid behavior is barely affected. These observations in combination with theoretical results suggest that the Fermi surface dominated by the Ni-$d_{3z^{2}-r^{2}}$ orbital is removed due to the transition at $T^{\ast}$. Our experimental results provide pivotal information for understanding the transition at $T^{\ast}$ and superconductivity in La$_{3}$Ni$_{2}$O$_{7}$.

cond-mat.supr-con↗

arXiv2024

Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models

Large language models (LLMs) have shown promising abilities of in-context learning (ICL), adapting swiftly to new tasks with only few-shot demonstrations. However, current few-shot methods heavily depend on high-quality, query-specific demos, which are often lacking. When faced with out-of-demonstration (OOD) queries, methods that rely on hand-crafted demos or external retrievers might fail. To bridge the gap between limited demos and OOD queries, we propose Self-Demos, a novel prompting method that elicits the inherent generalizability in LLMs by query-aware demo generation. The generated demos strategically interpolate between existing demos and the given query, transforming the query from OOD to ID. To evaluate the effectiveness of our approach, we manually constructed OOD-Toolset, a dataset in the tool-using scenario with over 300 real-world APIs and 1000 instances, each consisting of three tool-use cases as demos and an OOD query. Thorough experiments on our dataset and two public math benchmarks have shown that our method can outperform state-of-the-art baselines in the OOD setting. Moreover, we conduct a range of analyses to validate Self-Demos's generalization and provide more insights.

cs.CL↗

arXiv2024

LongHeads: Multi-Head Attention is Secretly a Long Context Processor

Large language models (LLMs) have achieved impressive performance in numerous domains but often struggle to process lengthy inputs effectively and efficiently due to limited length generalization and attention's quadratic computational demands. Many sought to mitigate this by restricting the attention window within the pre-trained length. However, these methods introduce new issues such as ignoring the middle context and requiring additional training. To address these problems, we propose LongHeads, a training-free framework that enhances LLM's long context ability by unlocking multi-head attention's untapped potential. Instead of allowing each head to attend to the full sentence, which struggles with generalizing to longer sequences due to out-of-distribution (OOD) issues, we allow each head to process in-distribution length by selecting and attending to important context chunks. To this end, we propose a chunk selection strategy that relies on the inherent correlation between the query and the key representations, efficiently distributing context chunks to different heads. In this way, each head ensures it can effectively process attended tokens within the trained length, while different heads in different layers can collectively process longer contexts. LongHeads works efficiently in linear time, fits seamlessly with many LLMs that use relative positional encoding. LongHeads achieves 100% accuracy at the 128k length on passkey retrieval task, verifying LongHeads's efficacy in extending the usable context window for existing models. We release our code at https://github.com/LuLuLuyi/LongHeads .

cs.CL↗

arXiv2024

A novel quark pairing in sQGP induced by the non-Abelian feature of the interaction

We solve the coupled Dyson-Schwinger equations for quark propagator and quark gluon vertex in the Nambu-Gorkov basis which is widely applied to study the color superconductivity. After considering the non-Abelian feature in the off-diagonal part of quark gluon vertex, we acquire a quark pairing gap in chiral limit above the chiral phase transition temperature $T_c$. The gap persists up to $2-3\,T_c$ and vanishes at higher temperature. Such a quark pairing characterizes the strongly coupled quark gluon plasma phase as a new phase and distinct from the phase with quasi quarks and gluons. Its new features can be disclosed in the heavy ion collision experiments.

hep-ph↗

arXiv2024

CSST large-scale structure analysis pipeline: I. constructing reference mock galaxy redshift surveys

In this paper, we set out to construct a set of reference mock galaxy redshift surveys (MGRSs) for the future Chinese Space-station Survey Telescope (CSST) observation, where subsequent survey selection effects can be added and evaluated. This set of MGRSs is generated using the dark matter subhalos extracted from a high-resolution Jiutian $N$-body simulation of the standard $Λ$CDM cosmogony with $Ω_m=0.3111$, $Ω_Λ=0.6889$, and $σ_8=0.8102$. The simulation has a boxsize of $1~h^{-1} {\rm Gpc}$, and consists of $6144^3$ particles with mass resolution $3.723 \times 10^{8} h^{-1} M_\odot $. In order to take into account the effect of redshift evolution, we first use all 128 snapshots in the Jiutian simulation to generate a light-cone halo/subhalo catalog. Next, galaxy luminosities are assigned to the main and subhalo populations using the subhalo abundance matching (SHAM) method with the DESI $z$-band luminosity functions at different redshifts. Multi-band photometries, as well as images, are then assigned to each mock galaxy using a 3-dimensional parameter space nearest neighbor sampling of the DESI LS observational galaxies and groups. Finally, the CSST and DESI LS survey geometry and magnitude limit cuts are applied to generate the required MGRSs. As we have checked, this set of MGRSs can generally reproduce the observed galaxy luminosity/mass functions within 0.1 dex for galaxies with $L > 10^8 L_\odot$ (or $M_* > 10^{8.5} M_\odot$) and within 1-$σ$ level for galaxies with $L < 10^8L_\odot$ (or $M_* < 10^{8.5} M_\odot$). Together with the CSST slitless spectra and redshifts for our DESI LS seed galaxies that are under construction, we will set out to test various slitless observational selection effects in subsequent probes.

astro-ph.GA↗

arXiv2024

LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration

Large language models (LLMs) have demonstrated impressive performance in understanding language and executing complex reasoning tasks. However, LLMs with long context windows have been notorious for their expensive training costs and high inference latency. Even the most advanced models such as GPT-4 and Claude2 often make mistakes when processing inputs of over $100k$ tokens, a phenomenon also known as \textit{lost in the middle}. In this paper, we propose \textsc{LongAgent}, a method based on multi-agent collaboration, which scales LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority in long-text processing compared to GPT-4. In \textsc{LongAgent}, a leader is responsible for understanding user intent and directing team members to acquire information from documents. Due to members' hallucinations, it is non-trivial for a leader to obtain accurate information from the responses of dozens to hundreds of members. To address this, we develop an \textit{inter-member communication} mechanism to resolve response conflicts caused by hallucinations through information sharing. Our experimental results indicate that \textsc{LongAgent} offers a promising alternative for long-text processing. The agent team instantiated with LLaMA-7B achieves significant improvements in tasks such as 128k-long text retrieval, multi-hop question answering, compared to GPT-4.

cs.CL↗

arXiv2024

Unrestricted Error-Type Codebook Generation for Error Correction Code in DNA Storage Inspired by NLP

Recently, DNA storage has surfaced as a promising alternative for data storage, presenting notable benefits in terms of storage capacity, cost-effectiveness in maintenance, and the capability for parallel replication. Mathematically, the DNA storage process can be conceptualized as an insertion, deletion, and substitution (IDS) channel. Due to the mathematical complexity associated with the Levenshtein distance, creating a code that corrects for IDS remains a challenging task. In this paper, we propose a bottom-up generation approach to grow the required codebook based on the computation of Edit Computational Graph (ECG) which differs from the algebraic constructions by incorporating the Derivative-Free Optimization (DFO) method. Specifically, this approach is regardless of the type of errors. Compared the results with the work for 1-substitution-1-deletion and 2-deletion, the redundancy is reduced by about 30-bit and 60-bit, respectively. As far as we know, our method is the first IDS-correcting code designed using classical Natural Language Process (NLP) techniques, marking a turning point in the field of error correction code research. Based on the codebook generated by our method, there may be significant breakthroughs in the complexity of encoding and decoding algorithms.

cs.IT↗

arXiv2024

Electronic and magnetic excitations in La$_3$Ni$_2$O$_7$

The striking discovery of high-temperature superconductivity (HTSC) of 80 K in a bilayer nickelate La$_3$Ni$_2$O$_7$ under a moderately high pressure of about 14 GPa ignited a new wave of studying HTSC in nickelates. The properties of the parental phase at ambient pressure may contain key information on basic interactions therein and bosons that may mediate pairing giving birth to superconductivity. Moreover, the bilayer structure of La$_3$Ni$_2$O$_7$ may suggest a distinct minimal model in comparison to cuprate superconductors. Here using X-ray absorption spectroscopy and resonant inelastic X-ray scattering, we studied La$_3$Ni$_2$O$_7$ at ambient pressure, and found that Ni 3$d_{x^2-y^2}$, Ni 3$d_{z^2}$, and ligand oxygen 2$p$ orbitals dominate the low-energy physics with a small charge-transfer energy. Remarkably, well-defined optical-like magnetic excitations were found to soften into a quasi-static spin-density-wave ordering, evidencing the strong electronic correlations and rich magnetic properties. Based on a Heisenberg spin model, we found that the inter-layer effective magnetic superexchange interaction is much larger than the intra-layer ones, and proposed two viable magnetic structures. Our results set the foundation for further exploration of La$_3$Ni$_2$O$_7$ superconductor.

cond-mat.supr-con↗