arXiv subjects

Zhiqiang Wang

Publications and source records attributed to Zhiqiang Wang.

At least 73 records · Page 4Linked to original sources

arXiv2024

SINGER: Vivid Audio-driven Singing Video Generation with Multi-scale Spectral Diffusion Model

Recent advancements in generative models have significantly enhanced talking face video generation, yet singing video generation remains underexplored. The differences between human talking and singing limit the performance of existing talking face video generation models when applied to singing. The fundamental differences between talking and singing-specifically in audio characteristics and behavioral expressions-limit the effectiveness of existing models. We observe that the differences between singing and talking audios manifest in terms of frequency and amplitude. To address this, we have designed a multi-scale spectral module to help the model learn singing patterns in the spectral domain. Additionally, we develop a spectral-filtering module that aids the model in learning the human behaviors associated with singing audio. These two modules are integrated into the diffusion model to enhance singing video generation performance, resulting in our proposed model, SINGER. Furthermore, the lack of high-quality real-world singing face videos has hindered the development of the singing video generation community. To address this gap, we have collected an in-the-wild audio-visual singing dataset to facilitate research in this area. Our experiments demonstrate that SINGER is capable of generating vivid singing videos and outperforms state-of-the-art methods in both objective and subjective evaluations.

cs.CV↗

arXiv2024

Problematic Tokens: Tokenizer Bias in Large Language Models

Recent advancements in large language models(LLMs), such as GPT-4 and GPT-4o, have shown exceptional performance, especially in languages with abundant resources like English, thanks to extensive datasets that ensure robust training. Conversely, these models exhibit limitations when processing under-resourced languages such as Chinese and Korean, where issues including hallucinatory responses remain prevalent. This paper traces the roots of these disparities to the tokenization process inherent to these models. Specifically, it explores how the tokenizers vocabulary, often used to speed up the tokenization process and reduce tokens but constructed independently of the actual model training data, inadequately represents non-English languages. This misrepresentation results in the propagation of under-trained or untrained tokens, which perpetuate biases and pose serious concerns related to data security and ethical standards. We aim to dissect the tokenization mechanics of GPT-4o, illustrating how its simplified token-handling methods amplify these risks and offer strategic solutions to mitigate associated security and ethical issues. Through this study, we emphasize the critical need to rethink tokenization frameworks to foster more equitable and secure AI technologies. The code and data are available at https://github.com/yeyimilk/LLMGPT4o

cs.CL↗

arXiv2024

Integrating Graph Neural Networks and Many-Body Expansion Theory for Potential Energy Surfaces

Rational design of next-generation functional materials relied on quantitative predictions of their electronic structures beyond single building blocks. First-principles quantum mechanical (QM) modeling became infeasible as the size of a material grew beyond hundreds of atoms. In this study, we developed a new computational tool integrating fragment-based graph neural networks (FBGNN) into the fragment-based many-body expansion (MBE) theory, referred to as FBGNN-MBE, and demonstrated its capacity to reproduce full-dimensional potential energy surfaces (FD-PES) for hierarchic chemical systems with manageable accuracy, complexity, and interpretability. In particular, we divided the entire system into basic building blocks (fragments), evaluated their single-fragment energies using a first-principles QM model and attacked many-fragment interactions using the structure-property relationships trained by FBGNNs. Our development of FBGNN-MBE demonstrated the potential of a new framework integrating deep learning models into fragment-based QM methods, and marked a significant step towards computationally aided design of large functional materials.

cond-mat.mtrl-sci↗

arXiv2024

Gibberish is All You Need for Membership Inference Detection in Contrastive Language-Audio Pretraining

Audio can disclose PII, particularly when combined with related text data. Therefore, it is essential to develop tools to detect privacy leakage in Contrastive Language-Audio Pretraining(CLAP). Existing MIAs need audio as input, risking exposure of voiceprint and requiring costly shadow models. We first propose PRMID, a membership inference detector based probability ranking given by CLAP, which does not require training shadow models but still requires both audio and text of the individual as input. To address these limitations, we then propose USMID, a textual unimodal speaker-level membership inference detector, querying the target model using only text data. We randomly generate textual gibberish that are clearly not in training dataset. Then we extract feature vectors from these texts using the CLAP model and train a set of anomaly detectors on them. During inference, the feature vector of each test text is input into the anomaly detector to determine if the speaker is in the training set (anomalous) or not (normal). If available, USMID can further enhance detection by integrating real audio of the tested speaker. Extensive experiments on various CLAP model architectures and datasets demonstrate that USMID outperforms baseline methods using only text data.

cs.SD↗

arXiv2024

Benchmarking Large Language Models for Image Classification of Marine Mammals

As Artificial Intelligence (AI) has developed rapidly over the past few decades, the new generation of AI, Large Language Models (LLMs) trained on massive datasets, has achieved ground-breaking performance in many applications. Further progress has been made in multimodal LLMs, with many datasets created to evaluate LLMs with vision abilities. However, none of those datasets focuses solely on marine mammals, which are indispensable for ecological equilibrium. In this work, we build a benchmark dataset with 1,423 images of 65 kinds of marine mammals, where each animal is uniquely classified into different levels of class, ranging from species-level to medium-level to group-level. Moreover, we evaluate several approaches for classifying these marine mammals: (1) machine learning (ML) algorithms using embeddings provided by neural networks, (2) influential pre-trained neural networks, (3) zero-shot models: CLIP and LLMs, and (4) a novel LLM-based multi-agent system (MAS). The results demonstrate the strengths of traditional models and LLMs in different aspects, and the MAS can further improve the classification performance. The dataset is available on GitHub: https://github.com/yeyimilk/LLM-Vision-Marine-Animals.git.

cs.CV↗

arXiv2024

The Spectrality of Infinite Convolutions in $\mathbb{R}^d$

In this paper, we study the spectrality of infinite convolutions in $\mathbb{R}^d$, where the spectrality means the corresponding square integrable function space admits a family of exponential functions as an orthonormal basis. Suppose that the infinite convolutions are generated by a sequence of admissible pairs in $\mathbb{R}^d$. We give two sufficient conditions for their spectrality by using the equi-positivity condition and the integral periodic zero set of Fourier transform. By applying these results, we show the spectrality of some specific infinite convolutions in $\mathbb{R}^d$.

math.CA↗

arXiv2024

Spectrality of Infinite Convolutions and Random Convolutions

In this paper, we explore spectral measures whose square integrable spaces admit a family of exponential functions as an orthonormal basis.Our approach involves utilizing the integral periodic zeros set of Fourier transform to characterize spectrality of infinite convolutions generated by a sequence of admissible pairs.Then we delve into the analysis of the integral periodic zeros set. Finally, we show that given finitely many admissible pairs, almost all random convolutions are spectral measures. Moreover, we give a complete characterization of spectrality of random convolutions in some special cases.

math.CA↗

arXiv2024

On the strong separation condition for self-similar iterated function systems with random translations

Given a self-similar iterated function system $Φ=\{ ϕ_i(x)=ρ_i O_i x+t_i \}_{i=1}^m$ acting on $\mathbb{R}^d$, we can generate a parameterised family of iterated function systems by replacing each $t_i$ with a random vector in $\mathbb{R}^d$. In this paper we study whether a Lebesgue typical member of this family will satisfy the strong separation condition. Our main results show that if the similarity dimension of $Φ$ is sufficiently small, then a Lebesgue typical member of this family will satisfy the strong separation condition.

math.DS↗

arXiv2024

AGR: Age Group fairness Reward for Bias Mitigation in LLMs

LLMs can exhibit age biases, resulting in unequal treatment of individuals across age groups. While much research has addressed racial and gender biases, age bias remains little explored. The scarcity of instruction-tuning and preference datasets for age bias hampers its detection and measurement, and existing fine-tuning methods seldom address age-related fairness. In this paper, we construct age bias preference datasets and instruction-tuning datasets for RLHF. We introduce ARG, an age fairness reward to reduce differences in the response quality of LLMs across different age groups. Extensive experiments demonstrate that this reward significantly improves response accuracy and reduces performance disparities across age groups. Our source code and datasets are available at the anonymous \href{https://anonymous.4open.science/r/FairRLHF-D445/readme.md}{link}.

cs.LG↗

arXiv2024

Android Malware Detection Based on RGB Images and Multi-feature Fusion

With the widespread adoption of smartphones, Android malware has become a significant challenge in the field of mobile device security. Current Android malware detection methods often rely on feature engineering to construct dynamic or static features, which are then used for learning. However, static feature-based methods struggle to counter code obfuscation, packing, and signing techniques, while dynamic feature-based methods involve time-consuming feature extraction. Image-based methods for Android malware detection offer better resilience against malware variants and polymorphic malware. This paper proposes an end-to-end Android malware detection technique based on RGB images and multi-feature fusion. The approach involves extracting Dalvik Executable (DEX) files, AndroidManifest.xml files, and API calls from APK files, converting them into grayscale images, and enhancing their texture features using Canny edge detection, histogram equalization, and adaptive thresholding techniques. These grayscale images are then combined into an RGB image containing multi-feature fusion information, which is analyzed using mainstream image classification models for Android malware detection. Extensive experiments demonstrate that the proposed method effectively captures Android malware characteristics, achieving an accuracy of up to 97.25%, outperforming existing detection methods that rely solely on DEX files as classification features. Additionally, ablation experiments confirm the effectiveness of using the three key files for feature representation in the proposed approach.

cs.CR↗

arXiv2024

All Robots in One: A New Standard and Unified Dataset for Versatile, General-Purpose Embodied Agents

Embodied AI is transforming how AI systems interact with the physical world, yet existing datasets are inadequate for developing versatile, general-purpose agents. These limitations include a lack of standardized formats, insufficient data diversity, and inadequate data volume. To address these issues, we introduce ARIO (All Robots In One), a new data standard that enhances existing datasets by offering a unified data format, comprehensive sensory modalities, and a combination of real-world and simulated data. ARIO aims to improve the training of embodied AI agents, increasing their robustness and adaptability across various tasks and environments. Building upon the proposed new standard, we present a large-scale unified ARIO dataset, comprising approximately 3 million episodes collected from 258 series and 321,064 tasks. The ARIO standard and dataset represent a significant step towards bridging the gaps of existing data resources. By providing a cohesive framework for data collection and representation, ARIO paves the way for the development of more powerful and versatile embodied AI agents, capable of navigating and interacting with the physical world in increasingly complex and diverse ways. The project is available on https://imaei.github.io/project_pages/ario/

cs.RO↗

arXiv2024

Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMs

Bias in LLMs can harm user experience and societal outcomes. However, current bias mitigation methods often require intensive human feedback, lack transferability to other topics or yield overconfident and random outputs. We find that involving LLMs in role-playing scenario boosts their ability to recognize and mitigate biases. Based on this, we propose Reinforcement Learning from Multi-role Debates as Feedback (RLDF), a novel approach for bias mitigation replacing human feedback in traditional RLHF. We utilize LLMs in multi-role debates to create a dataset that includes both high-bias and low-bias instances for training the reward model in reinforcement learning. Our approach comprises two modes: (1) self-reflection, where the same LLM participates in multi-role debates, and (2) teacher-student, where a more advanced LLM like GPT-3.5-turbo guides the LLM to perform this task. Experimental results across different LLMs on BBQ and our datasets demonstrate the effectiveness of our approach in bias mitigation. Our source code and datasets are available at \texttt{https://anonymous.4open.science/r/RLDF-E344}.

cs.AI↗

arXiv2024

An Adaptive Differential Privacy Method Based on Federated Learning

Differential privacy is one of the methods to solve the problem of privacy protection in federated learning. Setting the same privacy budget for each round will result in reduced accuracy in training. The existing methods of the adjustment of privacy budget consider fewer influencing factors and tend to ignore the boundaries, resulting in unreasonable privacy budgets. Therefore, we proposed an adaptive differential privacy method based on federated learning. The method sets the adjustment coefficient and scoring function according to accuracy, loss, training rounds, and the number of datasets and clients. And the privacy budget is adjusted based on them. Then the local model update is processed according to the scaling factor and the noise. Fi-nally, the server aggregates the noised local model update and distributes the noised global model. The range of parameters and the privacy of the method are analyzed. Through the experimental evaluation, it can reduce the privacy budget by about 16%, while the accuracy remains roughly the same.

cs.CR↗

arXiv2024

Stability and Dynamics of Atom-Molecule Superfluids Near a Narrow Feshbach Resonance

The recent observations of a stable molecular condensate emerging from a condensate of bosonic atoms and related "super-chemical" dynamics have raised an intriguing set of questions. Here we provide a microscopic understanding of this unexpected stability and dynamics in atom-molecule superfluids; we show one essential element behind these phenomena is an extremely narrow Feshbach resonance in $^{133}$Cs at 19.849G. Comparing theory and experiment we demonstrate how this narrow resonance enables the dynamical creation of a large closed-channel molecular fraction superfluid, appearing in the vicinity of unitarity. Theoretically the observed superchemistry (\textit{i.e.}, Bose enhanced reactions of atoms and molecules), is found to be assisted by the formation of Cooper-like pairs of bosonic atoms that have opposite momenta. Importantly, this narrow resonance opens the possibility to explore the quantum critical point of a molecular Bose superfluid and related phenomena which would not be possible near a more typically broad Feshbach resonance.

cond-mat.quant-gas↗

arXiv2024

When Superconductivity Crosses Over: From BCS to BEC

New developments in superconductivity, particularly through unexpected and often astonishing forms of superconducting materials, continue to excite the community and stimulate theory. It is now becoming clear that there are two distinct platforms for superconductivity: natural and synthetic materials. The study of these artificial materials has greatly expanded in the last decade or so, with the discoveries of new forms of superfluidity in artificial heterostructures and the exploitation of proximitization. Natural superconductors continue to surprise through the Fe-based pnictides and chalcogenides, and nickelates as well as others. It is the goal of this review to present this two-pronged investigation into superconductors, with a focus on those that we have come to understand belong somewhere between the Bardeen-Cooper-Schrieffer (BCS) and Bose-Einstein condensation (BEC) regimes. We characterize in detail the nature of this "crossover" superconductivity, which is to be distinguished from crossover superfluidity in atomic Fermi gases. In the process, we address the multiple ways of promoting a system out of the BCS and into the BCS-BEC crossover regime within the context of concrete experimental realizations. These involve natural materials, such as organic conductors, as well as artificial, mostly two-dimensional materials, such as magic-angle twisted bilayer and trilayer graphene, or gate-controlled devices, as well as one-layer and interfacial superconducting films. This work should be viewed as a celebration of BCS theory by showing that even though this theory was initially implemented with the special case of weak correlations in mind, it can in a very natural way be extended to treat the case of these more exotic strongly correlated superconductors.

cond-mat.supr-con↗

arXiv2024

Fractal Sumset Properties

In this paper we introduce two notions of fractal sumset properties. A compact set $K\subset\mathbb{R}^d$ is said to have the Hausdorff sumset property (HSP) if for any $\ell\in\mathbb{N}_{\ge 2}$ there exist compact sets $K_1, K_2,\ldots, K_\ell$ such that $K_1+K_2+\cdots+K_\ell\subset K$ and $\dim_H K_i=\dim_H K$ for all $1\le i\le \ell$. Analogously, if we replace the Hausdorff dimension by the packing dimension in the definition of HSP, then the compact set $K\subset\mathbb{R}^d$ is said to have the packing sumset property (PSP). We show that the HSP fails for certain homogeneous self-similar sets satisfying the strong separation condition, while the PSP holds for all homogeneous self-similar sets in $\mathbb{R}^d$.

math.CA↗

arXiv2024

Rational Points in Translations of The Cantor Set

Given two coprime integers $p\ge 2$ and $q \ge 3$, let $D_p\subset[0,1)$ consist of all rational numbers which have a finite $p$-ary expansion, and let $$ K(q, \mathcal{A})=\bigg\{ \sum_{i=1}^\infty \frac{d_i}{q^i}: d_i\in \mathcal{A}~ \forall i\in\mathbb{N} \bigg\}, $$ where $\mathcal{A} \subset \{0,1,\ldots, q-1\}$ with cardinality $1<\#\mathcal{A}< q$. In 2021 Schleischitz showed that $\#(D_p\cap K(q,\mathcal{A}))<+\infty$. In this paper we show that for any $r\in\mathbb{Q}$ and for any $α\in\mathbb{R}$, $$ \#\big((r D_p+α)\cap K(q,\mathcal{A})\big)<+\infty. $$

math.NT↗

arXiv2024

On a class of self-similar sets which contain finitely many common points

For $λ\in(0,1/2]$ let $K_λ\subset\mathbb{R}$ be a self-similar set generated by the iterated function system $\{λx, λx+1-λ\}$. Given $x\in(0,1/2)$, let $Λ(x)$ be the set of $λ\in(0,1/2]$ such that $x\in K_λ$. In this paper we show that $Λ(x)$ is a topological Cantor set having zero Lebesgue measure and full Hausdorff dimension. Furthermore, we show that for any $y_1,\ldots, y_p\in(0,1/2)$ there exists a full Hausdorff dimensional set of $λ\in(0,1/2]$ such that $y_1,\ldots, y_p \in K_λ$.

math.DS↗