A Unifying Perspective on Audio Generative Modeling: Latent Representations and Modeling Strategies

Abstract

Every audio generative system makes two coupled decisions: what representation to generate, and how to model its distribution. This paper organizes audio generative modeling around this coupling. For representation design, we compare discrete, continuous, and hybrid latents through four objectives: representation burden, distortion, empirical modelability, and streaming compatibility. For distribution modeling, rather than treating a latent's difficulty as an intrinsic scalar, we use two diagnostic dimensions: dependency horizon, how far useful context extends, and conditional ambiguity, how much uncertainty remains after conditioning. These dimensions refine the common semantic-versus-acoustic intuition: variables with a long dependency horizon should receive global modeling capacity. Conditionally ambiguous detail may be delegated to a local or iterative generator. Applied to representative systems, this view shows that RVQ's residual order gives ordered capacity but not ordered semantics, that AudioLM's semantic-versus-acoustic cascade is one explicit placement of this boundary rather than a universal template, and that autoregression, iterative refinement, and hybrid designs differ chiefly in how they trade dependency horizon against critical-path generation cost. The distinction between discrete and continuous latents describes the output interface; dependency horizon, conditional ambiguity, and streaming determine how that interface should be modeled. Rather than cataloguing individual systems, we provide an evaluation and design framework for comparing representation-model pairs.