MusicWeaver: Programmable Long-Form Music Generation with Provably Local Editing

Abstract

Music generation systems produce increasingly realistic audio, yet they expose no interface between a creator's structural intent and the rendered sound. Form can only be steered through prompts, while local revisions require regenerating entire pieces. We recast music creation as program-guided generation, using an explicit, human-interpretable program layer between intent and audio. We present MusicWeaver, which decomposes generation into planning and rendering. The planning stage predicts a structured plan, a multi-level song program encoding musical form, motif recurrence, and bar-level attributes. The rendering stage synthesizes audio conditioned on this plan. We formalize editing as an algebra of typed plan operations, including section replacement, insertion, deletion, motif retagging, and attribute changes, and prove that they preserve plan validity. To realize edits, we propose Projected Diffusion Inpainting, which guarantees that audio outside the edited span is preserved exactly across repeated revisions. For rendering, we design a Global-Local Diffusion Transformer with Motif Memory Retrieval that executes minute-scale plans and produces section returns that are consistent yet varied. Our framework also supports plan induction, recovering an editable program from an existing recording, and a natural-language editor that compiles free-form instructions into validated operation sequences. We introduce plan-faithfulness and edit-fidelity measures and validate them against human judgments. Experiments on generation conditioned on text, video, and their combination show state-of-the-art structural coherence and editability, while edit quality remains stable over sequential revisions where prior methods degrade.