Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.25259

VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

Kaidi Wang·Wenhao Guan·Peijie Chen·Weijie Wu·Xuetao Lin·Lin Li·Qingyang Hong
View original source ↗Read source document ↗Export metadata ↓

Abstract

Automatic video dubbing (AVD) requires temporally aligned speech, transcripts, and video, yet existing corpora vary in audio quality, transcription reliability, and speaking-face visibility. We introduce VoxDub, a bilingual lip-synchronized audio-visual dataset containing 766K clips and approximately 1.58K hours of in-the-wild Chinese and English recordings, covering varied speakers, scenes, speaking styles, and linguistic content. Its curation pipeline combines speech-centered segmentation with dual-ASR transcription checking, speech quality filtering, face visibility checks, and lip-speech synchronization verification. Text-to-speech (TTS) experiments show that training on VoxDub yields lower word error rates and higher speaker similarity than training on the compared audio-visual corpora, LRS3 and CN-CVS. AVD experiments show broadly comparable speech quality and subjective synchronization. These results support the utility of VoxDub for TTS and AVD training. More details are available on the project website (https://voxdub-dataset.github.io/VoxDub/).

Explore related subjects

cs.SDeess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kaidi Wang, Wenhao Guan, Peijie Chen, Weijie Wu, Xuetao Lin, Lin Li, Qingyang Hong. 2026-10-18. VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing. https://arxiv.org/abs/2610.25259

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LARA-Gen: Enabling Continuous Emotion Control for Music Generation Models via Latent Affective Representation Alignment

Recent advances in text-to-music models have enabled coherent music generation from text prompts, yet fine-grained emotional control remains unresolved. We introduce LARA-Gen, a framework for continuous emotion control that aligns the internal hidden states with an external music understanding model through Latent Affective Representation Alignment (LARA), enabling effective training. In addition, we design an emotion control module based on a continuous valence-arousal space, disentangling emotional attributes from textual content and bypassing the bottlenecks of text-based prompting. Furthermore, we establish a benchmark with a curated test set and a robust Emotion Predictor, facilitating objective evaluation of emotional controllability in music generation. Extensive experiments demonstrate that LARA-Gen achieves continuous, fine-grained control of emotion and significantly outperforms baselines in both emotion adherence and music quality. Generated samples are available at https://anonymous2232330.github.io/laragen-web/.

cs.SD↗
arXiv2026

Resonance and Depth: Composing Timbral Identity and Spatial Form in an Artist-in-the-Loop Co-Creative Multi-Agent Percussive Performance

This research-creation artistic report presents Resonance and Depth, a 9-minute immersive electroacoustic/acousmatic performance that treats timbre and space as primary compositional materials. Built from live electronic percussion and a personal corpus recorded, curated, and composed by the author, the work investigates how timbral identity and musical form can emerge through real-time recombination, spatialisation, and performer-system interaction rather than conventional harmonic development. The performance is realised as an artist-in-the-loop co-creative multi-agent system integrating MACAT, MACataRT, a MIDI-based musical agent, a spatialisation layer, and a conductor agent. These components recombine and transform the performer's material through concatenative synthesis, audio mosaicing, symbolic mediation, physically inspired synthesis, and object-based spatialisation. The work approaches spatialisation as an active compositional process in which position, trajectory, depth, diffusion, width, and apparent size participate in musical phrasing and form. Through this practice-based framework, Resonance and Depth examines how performer identity can be retained, transformed, and redistributed across human, algorithmic, timbral, and spatial processes within an immersive co-creative performance ecology.

cs.SD↗
arXiv2026

KunSet: A Corpus and Benchmark for Lyrics Alignment and Rhyme Retrieval in Kunqu Opera

Kunqu is one of the oldest extant forms of Chinese opera, yet its digitization remains scarce and difficult to annotate, with two challenges standing out: first, close analysis of Kunqu performance demands syllable-level lyrics alignment; second, analyzing Kunqu pronunciation requires its phonological attributes (e.g., rhyme) to be linked with each syllable. To address these challenges, we present KunSet, the first Kunqu performance corpus comprising 482,164 fine-grained annotations of syllable-level lyrics and phonological attributes in total media duration of 155.2 h with 106.3 h annotated. To ensure annotation quality, KunSet was preannotated by an iterative ML-assisted pipeline and corrected by over 200 trained annotators via a purpose-built online annotation system under a two-round review process. To assess KunSet as a music information retrieval (MIR) benchmark, we evaluate two tasks: lyrics alignment, by finetuning a forced-alignment model with an acoustic anchor decoder; and same-rhyme retrieval, using frozen Chinese HuBERT features. Models built with KunSet outperform their baselines. We release KunSet and model checkpoints to support MIR research on traditional-music corpora.

cs.SD↗