arXiv subjects

Zihao Fang

Publications and source records attributed to Zihao Fang.

8 recordsLinked to original sources

arXiv2026

Zero-VC: Zero-Lookahead Streaming Voice Conversion via Speaker Anonymization

Streaming zero-shot voice conversion struggles to disentangle timbre from linguistic content without degrading utility or inflating latency. Current methods rely on information bottleneck (IB) or speaker perturbation. While IB filters out timbre, it discards prosody, forcing models to explicitly inject features like fundamental frequency. This often requires buffering future frames, creating algorithmic lookahead latency. On the other hand, existing perturbation methods largely overlook the crucial trade-off between timbre leakage and utility preservation. Recognizing this neglected trade-off, we find that the inherent objective of Speaker Anonymization (SA) aligns well with balancing these factors. Thus, we introduce SA as a novel perturbation mechanism to explicitly mitigate timbre leakage while retaining prosodic utility. Crucially, SA's robust representations significantly alleviate the generator's reliance on future context, enabling our strictly causal, zero-lookahead network. Audio samples are available at https://amphionteam.github.io/Zero-VC-demo/.

cs.SD↗

arXiv2026

WhispEar: A Bi-directional Framework for Scaling Whispered Speech Conversion via Pseudo-Parallel Whisper Generation

Whispered speech lacks vocal fold vibration and fundamental frequency, resulting in degraded acoustic cues and making whisper-to-normal (W2N) conversion challenging, especially with limited parallel data. We propose WhispEar, a bidirectional framework based on unified semantic representations that capture speaking-mode-invariant information shared by whispered and normal speech. The framework contains both W2N and normal-to-whisper (N2W) models. Notably, the N2W model enables zero-shot pseudo-parallel whisper generation from abundant normal speech, allowing scalable data augmentation for W2N training. Increasing generated data consistently improves performance. We also release the largest bilingual (Chinese-English) whispered-normal parallel corpus to date. Experiments demonstrate that WhispEar outperforms strong baselines and benefits significantly from scalable pseudo-parallel data.

cs.SD↗

arXiv2025

Two-stage Bootstrap Percolation

We introduce and study two variants of two-stage growth dynamics in $\mathbb{Z}^2$ with state space $\{0,1,2\}^{\mathbb{Z}^2}$. In each variant, vertices in state $0$ can be changed irreversibly to state $1$, and vertices in state $1$ can be changed permanently to state $2$. In the standard variant, a vertex flips from state $i$ to $i+1$ if it has at least two nearest-neighbors in state $i+1$. In the modified variant, a $0$ changes to a $1$ if it has both a north or south neighbor and an east or west neighbor in state $1$, and a $1$ changes to a $2$ if it has at least two nearest-neighbors in state $2$. We assume that the initial configuration is given by a product measure with small probabilities $p$ and $q$ of $1$s and $2$s. For both variants, as $p$ and $q$ tend to $0$, if $q$ is large compared to $p^{2+o(1)}$, then the final density of $0$s tends to $1$. When $q$ is small compared to $p^{2+o(1)}$, for standard variant the final density of $2$s tends to $1$, while for the modified variant the final density of $1$s tends to $1$. In fact, for the modified variant, the final density of $2$s approaches $0$ regardless of the relative size of $q$ versus $p$. These results remain unchanged if, in either variant, a $1$ changes to a $2$ only if it has both a north or south neighbor and an east or west neighbor in state $2$. An essential feature of these dynamics is that they are not monotone in the initial configuration.

math.PR↗

arXiv2025

SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset

The lack of a publicly-available large-scale and diverse dataset has long been a significant bottleneck for singing voice applications like Singing Voice Synthesis (SVS) and Singing Voice Conversion (SVC). To tackle this problem, we present SingNet, an extensive, diverse, and in-the-wild singing voice dataset. Specifically, we propose a data processing pipeline to extract ready-to-use training data from sample packs and songs on the internet, forming 3000 hours of singing voices in various languages and styles. Furthermore, to facilitate the use and demonstrate the effectiveness of SingNet, we pre-train and open-source various state-of-the-art (SOTA) models on Wav2vec2, BigVGAN, and NSF-HiFiGAN based on our collected singing voice data. We also conduct benchmark experiments on Automatic Lyric Transcription (ALT), Neural Vocoder, and Singing Voice Conversion (SVC). Audio demos are available at: https://singnet-dataset.github.io/.

cs.SD↗

arXiv2025

AnyEnhance: A Unified Generative Model with Prompt-Guidance and Self-Critic for Voice Enhancement

We introduce AnyEnhance, a unified generative model for voice enhancement that processes both speech and singing voices. Based on a masked generative model, AnyEnhance is capable of handling both speech and singing voices, supporting a wide range of enhancement tasks including denoising, dereverberation, declipping, super-resolution, and target speaker extraction, all simultaneously and without fine-tuning. AnyEnhance introduces a prompt-guidance mechanism for in-context learning, which allows the model to natively accept a reference speaker's timbre. In this way, it could boost enhancement performance when a reference audio is available and enable the target speaker extraction task without altering the underlying architecture. Moreover, we also introduce a self-critic mechanism into the generative process for masked generative models, yielding higher-quality outputs through iterative self-assessment and refinement. Extensive experiments on various enhancement tasks demonstrate AnyEnhance outperforms existing methods in terms of both objective metrics and subjective listening tests. Demo audios are publicly available at https://amphionspace.github.io/anyenhance. An open-source implementation is provided at https://github.com/viewfinder-annn/anyenhance-v1-ccf-aatc.

cs.SD↗

arXiv2024

Cutoff for Contingency Table and Torus Random Walks with Low Incremental Correlations

We use the correlation matrix of the generating distribution to determine the mixing time for random walks on the torus $(\mathbb{Z}/q\mathbb{Z})^n$. We present our method in the context of the Diaconis-Gangolli random walk on both the $1 \times n$ and $m \times n$ contingency tables over $\mathbb{Z}/q\mathbb{Z}$. In the $1 \times n$ case, we prove that the random walk exhibits cutoff at time $\dfrac{n q^2 \log(n)}{8 \pi^2}$ when $q \gg n$; in the $m \times n$ case, where $m, n$ are of the same order, we establish cutoff for the random walk at time $\dfrac{mn q^2 \log(mn)}{16 \pi^2}$ when $q \gg n^2$. Our method reveals that a general class of random walks on the torus $(\mathbb{Z}/q\mathbb{Z})^n$ has cutoff. If each coordinate of the lifted random walk onto $\mathbb{Z}^n$ has variance $\sigma^2/n$ in each jump, and the between-coordinate correlations are sufficiently low, then cutoff occurs at time $\dfrac{nq^2 \log(n)}{4\pi^2 \sigma^2}$.

math.PR↗

arXiv2023

Amphion: An Open-Source Audio, Music and Speech Generation Toolkit

Amphion is an open-source toolkit for Audio, Music, and Speech Generation, targeting to ease the way for junior researchers and engineers into these fields. It presents a unified framework that includes diverse generation tasks and models, with the added bonus of being easily extendable for new incorporation. The toolkit is designed with beginner-friendly workflows and pre-trained models, allowing both beginners and seasoned researchers to kick-start their projects with relative ease. The initial release of Amphion v0.1 supports a range of tasks including Text to Speech (TTS), Text to Audio (TTA), and Singing Voice Conversion (SVC), supplemented by essential components like data preprocessing, state-of-the-art vocoders, and evaluation metrics. This paper presents a high-level overview of Amphion. Amphion is open-sourced at https://github.com/open-mmlab/Amphion.

cs.SD↗

arXiv2023

Leveraging Diverse Semantic-based Audio Pretrained Models for Singing Voice Conversion

Singing Voice Conversion (SVC) is a technique that enables any singer to perform any song. To achieve this, it is essential to obtain speaker-agnostic representations from the source audio, which poses a significant challenge. A common solution involves utilizing a semantic-based audio pretrained model as a feature extractor. However, the degree to which the extracted features can meet the SVC requirements remains an open question. This includes their capability to accurately model melody and lyrics, the speaker-independency of their underlying acoustic information, and their robustness for in-the-wild acoustic environments. In this study, we investigate the knowledge within classical semantic-based pretrained models in much detail. We discover that the knowledge of different models is diverse and can be complementary for SVC. Based on the above, we design a Singing Voice Conversion framework based on Diverse Semantic-based Feature Fusion (DSFF-SVC). Experimental results demonstrate that DSFF-SVC can be generalized and improve various existing SVC models, particularly in challenging real-world conversion tasks. Our demo website is available at https://diversesemanticsvc.github.io/.

cs.SD↗