arXiv subjects

Xuanchen Wang

Publications and source records attributed to Xuanchen Wang.

4 recordsLinked to original sources

arXiv2026

MusicWeaver: Programmable Long-Form Music Generation with Provably Local Editing

Music generation systems produce increasingly realistic audio, yet they expose no interface between a creator's structural intent and the rendered sound. Form can only be steered through prompts, while local revisions require regenerating entire pieces. We recast music creation as program-guided generation, using an explicit, human-interpretable program layer between intent and audio. We present MusicWeaver, which decomposes generation into planning and rendering. The planning stage predicts a structured plan, a multi-level song program encoding musical form, motif recurrence, and bar-level attributes. The rendering stage synthesizes audio conditioned on this plan. We formalize editing as an algebra of typed plan operations, including section replacement, insertion, deletion, motif retagging, and attribute changes, and prove that they preserve plan validity. To realize edits, we propose Projected Diffusion Inpainting, which guarantees that audio outside the edited span is preserved exactly across repeated revisions. For rendering, we design a Global-Local Diffusion Transformer with Motif Memory Retrieval that executes minute-scale plans and produces section returns that are consistent yet varied. Our framework also supports plan induction, recovering an editable program from an existing recording, and a natural-language editor that compiles free-form instructions into validated operation sequences. We introduce plan-faithfulness and edit-fidelity measures and validate them against human judgments. Experiments on generation conditioned on text, video, and their combination show state-of-the-art structural coherence and editability, while edit quality remains stable over sequential revisions where prior methods degrade.

cs.SD↗

arXiv2026

MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding

Music is both an auditory and an embodied phenomenon, closely linked to human motion and naturally expressed through dance. However, most existing audio representations neglect this embodied dimension, limiting their ability to capture rhythmic and structural cues that drive movement. We propose MotionBeat, a framework for motion-aligned music representation learning. MotionBeat is trained with two newly proposed objectives: the Embodied Contrastive Loss (ECL), an enhanced InfoNCE formulation with tempo-aware and beat-jitter negatives to achieve fine-grained rhythmic discrimination, and the Structural Rhythm Alignment Loss (SRAL), which ensures rhythm consistency by aligning music accents with corresponding motion events. Architecturally, MotionBeat introduces bar-equivariant phase rotations to capture cyclic rhythmic patterns and contact-guided attention to emphasize motion events synchronized with musical accents. Experiments show that MotionBeat outperforms state-of-the-art audio encoders in music-to-dance generation and transfers effectively to beat tracking, music tagging, genre and instrument classification, emotion recognition, and audio-visual retrieval. Our project demo page: https://motionbeat2025.github.io/.

cs.SD↗

arXiv2025

ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion

Modern artistic productions increasingly demand automated choreography generation that adapts to diverse musical styles and individual dancer characteristics. Existing approaches often fail to produce high-quality dance videos that harmonize with both musical rhythm and user-defined choreography styles, limiting their applicability in real-world creative contexts. To address this gap, we introduce ChoreoMuse, a diffusion-based framework that uses SMPL format parameters and their variation version as intermediaries between music and video generation, thereby overcoming the usual constraints imposed by video resolution. Critically, ChoreoMuse supports style-controllable, high-fidelity dance video generation across diverse musical genres and individual dancer characteristics, including the flexibility to handle any reference individual at any resolution. Our method employs a novel music encoder MotionTune to capture motion cues from audio, ensuring that the generated choreography closely follows the beat and expressive qualities of the input music. To quantitatively evaluate how well the generated dances match both musical and choreographic styles, we introduce two new metrics that measure alignment with the intended stylistic cues. Extensive experiments confirm that ChoreoMuse achieves state-of-the-art performance across multiple dimensions, including video quality, beat alignment, dance diversity, and style adherence, demonstrating its potential as a robust solution for a wide range of creative applications. Video results can be found on our project page: https://choreomuse.github.io.

cs.GR↗

arXiv2024

Dance Any Beat: Blending Beats with Visuals in Dance Video Generation

Generating dance from music is crucial for advancing automated choreography. Current methods typically produce skeleton keypoint sequences instead of dance videos and lack the capability to make specific individuals dance, which reduces their real-world applicability. These methods also require precise keypoint annotations, complicating data collection and limiting the use of self-collected video datasets. To overcome these challenges, we introduce a novel task: generating dance videos directly from images of individuals guided by music. This task enables the dance generation of specific individuals without requiring keypoint annotations, making it more versatile and applicable to various situations. Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos featuring various dance types and choreographies. The music is analyzed by our specially designed music encoder, which identifies essential features including dance style, movement, and rhythm. DabFusion excels in generating dance videos not only for individuals in the training dataset but also for any previously unseen person. This versatility stems from its approach of generating latent optical flow, which contains all necessary motion information to animate any person in the image. We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment. We propose a 2D Motion-Music Alignment Score (2D-MM Align), which builds on the Beat Alignment Score to more effectively evaluate motion-music alignment for this new task. Experiments show that our DabFusion establishes a solid baseline for this innovative task. Video results can be found on our project page: https://DabFusion.github.io.

cs.CV↗