arXiv subjects

Weifeng Zhao

Publications and source records attributed to Weifeng Zhao.

15 recordsLinked to original sources

arXiv2026

HARP: Harmonic-Aware Residual Partitioning for Neural Audio Codecs

Neural audio codecs with residual vector quantization (RVQ) normally treat all frequencies uniformly, so their codebooks become spectrally entangled. Truncating stages then removes an unpredictable mix of frequencies. Parallel band decomposition addresses this by splitting audio into independent bands, but fragments the latent space and loses cross-frequency coherence. We introduce HARP (Harmonic-Aware Residual Partitioning), a training strategy that partitions RVQ stages into frequency-ordered groups where each group refines its target band while the decoder retains access to all lower frequencies. Overtones are reconstructed in the context of their fundamentals, preserving coherence that parallel methods lose. HARP requires no architectural changes; it only modifies the training loss, leaving inference identical to standard RVQ. On speech, music, and general audio, HARP outperforms both standard RVQ and parallel decomposition. MUSHRA listening tests also show perceptual improvements.

cs.SD↗

arXiv2026

Poly-SVC: Polyphony-Aware Singing Voice Conversion with Harmonic Modeling

Singing Voice Conversion (SVC) aims to transform a source singing voice into a target singer while preserving lyrics and melody. Most existing SVC methods depend on F0 extractors to capture the lead melody from clean vocals. However, no existing method can reliably extract clean vocals from accompanied recordings without leaving residual harmonies behind. In this paper, we innovatively propose Poly-SVC, a zero-shot, cross-lingual singing voice conversion system designed to process residual harmonies. Poly-SVC is composed of three key components: a Constant-Q Transform (CQT)-based pitch extractor to preserve both the lead melody and residual harmony, a random sampler to reduce interference information from the CQT and a diffusion decoder based on Conditional Flow Matching (CFM) that fuses pitch, content, and timbre features into natural-sounding polyphonic outputs. Experiments demonstrate that Poly-SVC surpasses the baseline models in naturalness, timbre similarity and harmony reconstruction across both harmony-rich and single-melody recordings.

cs.SD↗

arXiv2026

VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

Human speech conveys expressiveness beyond linguistic content, including personality, mood, or performance elements, such as a comforting tone or humming a song, which we formalize as role-playing and singing. We present VITA-QinYu, the first expressive end-to-end (E2E) spoken language model (SLM) that goes beyond natural conversation to support both role-playing and singing generation. VITA-QinYu adopts a hybrid speech-text paradigm that extends interleaved text-audio modeling with multi-codebook audio tokens, a design enabling richer paralinguistic representation while preserving a clear separation between modalities to avoid interference. We further develop a comprehensive data generation pipeline to synthesize a total of 15.8K hours of natural conversation, role-playing, and singing data for training. VITA-QinYu demonstrates superior expressiveness, outperforming peer SLMs by 7 percentage points on objective role-playing benchmarks, and surpassing peer models by 0.13 points on a 5-point MOS scale for singing. Simultaneously, it achieves state-of-the-art conversational accuracy and fluency, exceeding prior SLMs by 1.38 and 4.98 percentage points on the C3 and URO benchmarks, respectively. We open-source our code and models and provide an easy-to-use demo with full-stack support for streaming and full-duplex interaction.

cs.CL↗

arXiv2026

LaDA-Band: Language Diffusion Models for Vocal-to-Accompaniment Generation

Vocal-to-accompaniment (V2A) generation, which aims to transform a raw vocal recording into a fully arranged accompaniment, inherently requires jointly addressing an accompaniment trilemma: preserving acoustic authenticity, maintaining global coherence with the vocal track, and producing dynamic orchestration across a full song. Existing open-source approaches typically make compromises among these goals. Continuous-latent generation models can capture long musical spans but often struggle to preserve fine-grained acoustic detail. In contrast, discrete autoregressive models retain local fidelity but suffer from unidirectional generation and error accumulation in extended contexts. We present LaDA-Band, an end-to-end framework that introduces Discrete Masked Diffusion to the V2A task. Our approach formulates V2A generation as Discrete Masked Diffusion, i.e., a global, non-autoregressive denoising formulation that combines the representational advantages of discrete audio codec tokens with full-sequence bidirectional context modeling. This design improves long-range structural consistency and temporal synchronization while preserving crisp acoustic details. Built on this formulation, LaDA-Band further introduces a dual-track prefix-conditioning architecture, an auxiliary replaced-token detection objective for weakly anchored accompaniment regions, and a two-stage progressive curriculum to scale Discrete Masked Diffusion to full-song vocal-to-accompaniment generation. Extensive experiments on both academic and real-world benchmarks show that LaDA-Band consistently improves acoustic authenticity, global coherence, and dynamic orchestration over existing baselines, while maintaining strong performance even without auxiliary reference audio. Codes and audio samples are available at https://github.com/Duoluoluos/TME-LaDA-Band .

cs.SD↗

arXiv2026

TQCodec: Towards neural audio codec for high-fidelity music streaming

We propose TQCodec, a neural audio codec designed for high-bitrate, high-fidelity music streaming. Unlike existing neural codecs that primarily target ultra-low bitrates (<= 16kbps), TQCodec operates at 44.1 kHz and supports bitrates from 32 kbps to 128 kbps, aligning with the standard quality of modern music streaming platforms. The model adopts an encoder-decoder architecture based on SEANet for efficient on-device computation and introduces several enhancements: an imbalanced network design for improved quality with low overhead, SimVQ for mid-frequency detail preservation, and a phase-aware waveform loss. Additionally, we introduce a perception-driven band-wise bit allocation strategy to prioritize perceptually critical lower frequencies. Evaluations on diverse music datasets demonstrate that TQCodec achieves superior audio quality at target bitrates, making it well-suited for high-quality audio applications.

cs.SD↗

arXiv2026

Validity of relaxation models arising from numerical schemes for hyperbolic-parabolic systems

This work is concerned with relaxation models arising from numerical schemes for hyperbolic-parabolic systems. Such models are a hyperbolic system with both the hyperbolic part and the stiff source term involving a small positive parameter, and thus are endowed with complicated multiscale properties. Relaxation models are the basis of constructing corresponding numerical schemes and a critical issue is the convergence of their solutions to those of the given target systems, the justification of which is still lacking. In this work, we employ the recently proposed theory for general hyperbolic relaxation systems to validate relaxation models in numerical schemes of hyperbolic-parabolic systems. By verifying the convergence criteria, we demonstrate the convergence, and thereby the approximation validity, of five representative relaxation models, providing a solid basis for the effectiveness of the corresponding numerical schemes. Moreover, we propose a new relaxation model for the general multi-dimensional hyperbolic-parabolic system. With some mild assumptions on the system, we show that the proposed model satisfies the convergence criteria. We remark that the existing relaxation models are constructed only for a special case of hyperbolic-parabolic system, while our new relaxation model is valid for general systems.

math.NA↗

arXiv2025

Deep Dubbing: End-to-End Auto-Audiobook System with Text-to-Timbre and Context-Aware Instruct-TTS

The pipeline for multi-participant audiobook production primarily consists of three stages: script analysis, character voice timbre selection, and speech synthesis. Among these, script analysis can be automated with high accuracy using NLP models, whereas character voice timbre selection still relies on manual effort. Speech synthesis uses either manual dubbing or text-to-speech (TTS). While TTS boosts efficiency, it struggles with emotional expression, intonation control, and contextual scene adaptation. To address these challenges, we propose DeepDubbing, an end-to-end automated system for multi-participant audiobook production. The system comprises two main components: a Text-to-Timbre (TTT) model and a Context-Aware Instruct-TTS (CA-Instruct-TTS) model. The TTT model generates role-specific timbre embeddings conditioned on text descriptions. The CA-Instruct-TTS model synthesizes expressive speech by analyzing contextual dialogue and incorporating fine-grained emotional instructions. This system enables the automated generation of multi-participant audiobooks with both timbre-matched character voices and emotionally expressive narration, offering a novel solution for audiobook production.

eess.AS↗

arXiv2025

CycleFlow: Leveraging Cycle Consistency in Flow Matching for Speaker Style Adaptation

Voice Conversion (VC) aims to convert the style of a source speaker, such as timbre and pitch, to the style of any target speaker while preserving the linguistic content. However, the ground truth of the converted speech does not exist in a non-parallel VC scenario, which induces the train-inference mismatch problem. Moreover, existing methods still have an inaccurate pitch and low speaker adaptation quality, there is a significant disparity in pitch between the source and target speaker style domains. As a result, the models tend to generate speech with hoarseness, posing challenges in achieving high-quality voice conversion. In this study, we propose CycleFlow, a novel VC approach that leverages cycle consistency in conditional flow matching (CFM) for speaker timbre adaptation training on non-parallel data. Furthermore, we design a Dual-CFM based on VoiceCFM and PitchCFM to generate speech and improve speaker pitch adaptation quality. Experiments show that our method can significantly improve speaker similarity, generating natural and higher-quality speech.

cs.SD↗

arXiv2023

VITS-based Singing Voice Conversion System with DSPGAN post-processing for SVCC2023

This paper presents the T02 team's system for the Singing Voice Conversion Challenge 2023 (SVCC2023). Our system entails a VITS-based SVC model, incorporating three modules: a feature extractor, a voice converter, and a post-processor. Specifically, the feature extractor provides F0 contours and extracts speaker-independent linguistic content from the input singing voice by leveraging a HuBERT model. The voice converter is employed to recompose the speaker timbre, F0, and linguistic content to generate the waveform of the target speaker. Besides, to further improve the audio quality, a fine-tuned DSPGAN vocoder is introduced to re-synthesise the waveform. Given the limited target speaker data, we utilize a two-stage training strategy to adapt the base model to the target speaker. During model adaptation, several tricks, such as data augmentation and joint training with auxiliary singer data, are involved. Official challenge results show that our system achieves superior performance, especially in the cross-domain task, ranking 1st and 2nd in naturalness and similarity, respectively. Further ablation justifies the effectiveness of our system design.

cs.SD↗

arXiv2023

MBTFNet: Multi-Band Temporal-Frequency Neural Network For Singing Voice Enhancement

A typical neural speech enhancement (SE) approach mainly handles speech and noise mixtures, which is not optimal for singing voice enhancement scenarios. Music source separation (MSS) models treat vocals and various accompaniment components equally, which may reduce performance compared to the model that only considers vocal enhancement. In this paper, we propose a novel multi-band temporal-frequency neural network (MBTFNet) for singing voice enhancement, which particularly removes background music, noise and even backing vocals from singing recordings. MBTFNet combines inter and intra-band modeling for better processing of full-band signals. Dual-path modeling are introduced to expand the receptive field of the model. We propose an implicit personalized enhancement (IPE) stage based on signal-to-noise ratio (SNR) estimation, which further improves the performance of MBTFNet. Experiments show that our proposed model significantly outperforms several state-of-the-art SE and MSS models.

cs.SD↗

arXiv2022

KaraTuner: Towards end to end natural pitch correction for singing voice in karaoke

An automatic pitch correction system typically includes several stages, such as pitch extraction, deviation estimation, pitch shift processing, and cross-fade smoothing. However, designing these components with strategies often requires domain expertise and they are likely to fail on corner cases. In this paper, we present KaraTuner, an end-to-end neural architecture that predicts pitch curve and resynthesizes the singing voice directly from the tuned pitch and vocal spectrum extracted from the original recordings. Several vital technical points have been introduced in KaraTuner to ensure pitch accuracy, pitch naturalness, timbre consistency, and sound quality. A feed-forward Transformer is employed in the pitch predictor to capture longterm dependencies in the vocal spectrum and musical note. We also develop a pitch-controllable vocoder based on a novel source-filter block and the Fre-GAN architecture. KaraTuner obtains a higher preference than the rule-based pitch correction approach through A/B tests, and perceptual experiments show that the proposed vocoder achieves significant advantages in timbre consistency and sound quality compared with the parametric WORLD vocoder, phase vocoder and CLPC vocoder.

cs.SD↗

arXiv2020

Boundary treatment of high order Runge-Kutta methods for hyperbolic conservation laws

In \cite{ZH2019}, we developed a boundary treatment method for implicit-explicit (IMEX) Runge-Kutta (RK) methods for solving hyperbolic systems with source terms. Since IMEX RK methods include explicit ones as special cases, this boundary treatment method naturally applies to explicit methods as well. In this paper, we examine this boundary treatment method for the case of explicit RK schemes of arbitrary order applied to hyperbolic conservation laws. We show that the method not only preserves the accuracy of explicit RK schemes but also possesses good stability. This compares favourably to the inverse Lax-Wendroff method in \cite{TS2010,TWSN2012} where analysis and numerical experiments have previously verified the presence of order reduction \cite{TS2010,TWSN2012}. In addition, we demonstrate that our method performs well for strong-stability-preserving (SSP) RK schemes involving negative coefficients and downwind spatial discretizations. It is numerically shown that when boundary conditions are present and the proposed boundary treatment is used, that SSP RK schemes with negative coefficients still allow for larger time steps than schemes with all non-negative coefficients. In this regard, our boundary treatment method is an effective supplement to SSP RK schemes with/without negative coefficients for initial-boundary value problems for hyperbolic conservation laws.

math.NA↗

arXiv2019

Boundary treatment of implicit-explicit Runge-Kutta method for hyperbolic systems with source terms

In this paper, we develop a high order finite difference boundary treatment method for the implicit-explicit (IMEX) Runge-Kutta (RK) schemes solving hyperbolic systems with possibly stiff source terms on a Cartesian mesh. The main challenge is how to obtain the solutions at ghost points resulting from the wide stencil of the interior high order scheme. We address this problem by combining the idea of using the RK schemes at the boundary and an inverse Lax-Wendroff procedure. The former preserves the accuracy of the RK schemes and the latter guarantees the stability. Our method is different from the widely used approach for the explicit RK schemes by imposing boundary conditions at intermediate stages, which could not be derived for the IMEX schemes. In addition, the intermediate boundary conditions are only available for explicit RK schemes up to third order while our method applies to arbitrary order IMEX and explicit RK schemes. Moreover, the present boundary treatment method may be adapted to IMEX RK schemes solving many other partial differential equations. For a specific third-order IMEX scheme, we demonstrate the good stability and third-order accuracy of our boundary treatment through both 1D examples and 2D reactive Euler equations.

math.NA↗

arXiv2019

Relaxation-rate formula for the entropic lattice Boltzmann method

An elegant and uniform relaxation-rate formula is presented for the entropic lattice Boltzmann method (ELBM). The formula not only guarantees the discrete time H-theorem at numerical level but also gives full consideration to the consistency with hydrodynamics. With this novel formula, the computational cost of the ELBM is significantly reduced and the method now can be efficiently used for a broad range of hydrodynamics applications including high Renolds number flows. Moreover, we demonstrate that the grid points where flow fields change drastically are effectively marked by the formula.

physics.comp-ph↗

arXiv2017

A family of single-node second-order boundary schemes for the lattice Boltzmann method

In this work, we propose a family of single-node second-order boundary schemes for the lattice Boltzmann method with general collision terms. The construction of the schemes is quite universal and simple, it does not involve concrete lattice Boltzmann models and uses the half-way bounce-back rule as a central step. The constructed schemes are all second-order accurate if so is the bounce-back rule. In addition, the proposed schemes have good stability thanks to convex combinations. The accuracy and stability of several specific schemes are numerically validated for multiple-relaxation-time models in both 2D and 3D.

physics.comp-ph↗