Unmixing The Crowd: Learning Persistent Speaker Representations from Mixture-Derived Multi-Speaker Embeddings

Abstract

We study whether persistent conversational speaker structure can be extracted directly from local overlapping speech mixtures. We propose a teacher-student framework that learns mixture-derived multi-speaker embeddings using only short overlapping segments and permutation-invariant latent supervision. Despite never being explicitly trained for speaker tracking, diarization, or conversational memory, the learned embedding space supports long-form speaker re-identification when combined with a lightweight online memory mechanism during inference. We additionally observe that the learned representation retains meaningful speaker structure under unseen overlap cardinalities. We further show that embeddings extracted from separation-first pipelines exhibit degraded clustering structure compared to embeddings predicted directly from mixtures. Finally, the learned embeddings remain effective for the downstream target speaker extraction task across multiple architectures. These findings suggest that local mixture-derived representations support persistent conversational speaker re-identification when combined with lightweight inference-time memory consolidation.