Generation-Step-Aware Framework for Cross-Modal Representation and Control in Multilingual Speech-Text Models

Abstract

Multilingual speech-text models rely on cross-modal language alignment to transfer knowledge between speech and text, but it remains unclear whether this reflects shared computation for the same language or modality-specific processing. We introduce a generation-step-aware framework for evaluating cross-modal computation that (i) identifies language-selective neurons for each modality at different decoding steps, (ii) decomposes them into language-representation and language-control roles, and (iii) enables cross-modal comparison via overlap measures and causal intervention. Applying our framework to SeamlessM4T, and additionally evaluating its generality on the decoder-only model Qwen2-Audio, we find that cross-modal language-representation alignment is highly model dependent. SeamlessM4T exhibits pronounced generation-step-dependent specialization, where only 5-7% of language-representation neurons are shared across modalities and overlap shifts from same-language to typologically related languages during autoregressive generation, whereas Qwen2-Audio maintains substantially larger cross-modal sharing (45-47%) together with stable, strongly language-specific alignment across decoding steps. In contrast, language-control neurons identified at later decoding steps exhibit progressively stronger cross-modal transfer from speech to text in SeamlessM4T. Together, these results demonstrate that generation-step-aware analysis can reveal both shared and model-specific patterns of cross-modal computation.