Searcharxiv⌕ Search

subject

eess.AS

eess.AS: explore 28 source-linked works published from 2024 to 2026, with original documents and citations.

This collection is a preview while coverage and quality are evaluated.

Search within this collection

Coverage and selection

Includes records with this source-supplied label or an explicit phrase match in their metadata. Matches indicate a mention, not proof that a paper uses a method or tests a material. Source versions are consolidated by DOI.

Sources: arxiv. Collection updated 2026-09-14. Counts describe this index, not the complete source archives.

arXiv2026

FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation

Scaling Multimodal Large Language Models (MLLMs) to long-form speech is bottlenecked by the explosive growth of input tokens. Existing speech-language models project high-frame-rate acoustic features directly into the LLM input space, making long-context processing computationally prohibitive. Unlike images or videos, speech lacks spatial redundancy, making extreme token compression particularly challenging. To address this limitation, we propose FastSLM, a token-efficient architecture featuring the Hierarchical Temporal Abstractor (HTA), which progressively distills acoustic features across multiple temporal scales. HTA achieves an extreme compression rate of 1.67 tokens per second (97% reduction) while preserving essential linguistic information for downstream speech-language understanding. Experimental results demonstrate that FastSLM achieves competitive performance across diverse speech-language tasks while requiring substantially fewer speech tokens and FLOPs than existing speech-language models. The source code and model checkpoints are available at https://github.com/Lee-junseok1025/FastSLM.

eess.AS↗
arXiv2026

MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models

Safety evaluation of multimodal large language models requires tracking not only whether an attack succeeds, but also how the interaction unfolds across turns and input modalities. We present MUSE (Multimodal Unified Safety Evaluation), an open-source, browser-based, run-centric platform for multimodal safety evaluation. MUSE treats each attack run as the persistent unit of execution, inspection, and analysis, preserving its configuration, multi-turn trajectory, delivered modalities and media, target responses, and safety judgments. A five-level response taxonomy further distinguishes full Compliance from Partial Compliance and refusal behavior, yielding hard ASR, soft ASR, and gray-zone width (GZW). Across 11,700 evaluations on six multimodal LLMs, direct text-only requests yield only 3.1% macro hard ASR and 4.4% soft ASR, while iterative attack procedures are substantially more effective. Attack effectiveness also varies substantially with the attacker backbone. In contrast, Inter-Turn Modality Switching (ITMS), evaluated as a controlled delivery-modality probe, does not consistently increase attack success. These results demonstrate the value of run-centric, fine-grained evaluation for characterizing multimodal safety behavior beyond a single binary success metric.

cs.LG↗
arXiv2026

KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness

Recent advances in large audio language models (LALMs) have enabled multilingual speech understanding. However, benchmarks for evaluating LALMs remain scarce for non-English languages, with Korean being one such underexplored case. In this paper, we introduce KoALa-Bench, a comprehensive benchmark for evaluating Korean speech understanding and speech faithfulness of LALMs. In particular, KoALa-Bench comprises six tasks. Four tasks evaluate fundamental speech understanding capabilities, including automatic speech recognition, speech translation, speech question answering, and speech instruction following, while the remaining two tasks evaluate speech faithfulness, motivated by our observation that several LALMs often fail to fully leverage the speech modality. Furthermore, to reflect Korea-specific knowledge, our benchmark incorporates listening questions from the Korean college scholastic ability test as well as content covering Korean cultural domains. We conduct extensive experiments across six models, including both white-box and black-box ones. Our benchmark, evaluation code, and leaderboard are publicly available at https://ksbench.github.io/Korean-Benchmark/.

cs.CL↗
arXiv2026

Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems

Speech enhancement (SE) is commonly applied as a preprocessing step in spoken AI pipelines under the assumption that better audio quality improves downstream task performance. Whether SE-induced distortions propagate to downstream LLM task performance remains an open question. We introduce Output Divergence Rate (ODR), which measures how often SE changes an LLM's intent classification relative to clean speech, and benchmark five conditions on 2,974 SLURP clips using Whisper large-v3 and wav2vec2-large cascades. Every condition produces ODR significantly above zero ($p < 0.001$, binomial test). MetricGAN{+} more than doubles ODR versus unenhanced noisy speech (0.318 vs. 0.135) despite improving PESQ, and unmitigated echo reaches an ODR of 0.836 through speaker substitution, a failure WER cannot capture. Audio quality metrics range from near-zero to moderate correlation with ODR (SQUIM-MOS $ρ=-0.068$, PESQ $ρ=-0.467$). The MetricGAN{+} and echo results replicate across ASR architectures, indicating that standard audio quality metrics are insufficient for LLM pipeline quality.

cs.SD↗
arXiv2026

Annotation of Soft Onsets in String Ensemble Recordings

Onset detection is the process of identifying the start points of musical note events within an audio recording. While the detection of percussive onsets is often considered a solved problem, soft onsets-as found in string instrument recordings-still pose a significant challenge for state-of-the-art algorithms. The problem is further exacerbated by a paucity of data containing expert annotations and research related to best practices for curating soft onset annotations for string instruments. To this end, we investigate inter-annotator agreement between 24 participants, extend an algorithm for determining the most consistent annotator, and compare the performance of human annotators and state-of-the-art onset detection algorithms. Experimental results reveal a positive trend between musical experience and both inter-annotator agreement and performance in comparison with automated systems. Additionally, onsets produced by changes in fingering as well as those from the cello were found to be particularly challenging for both human annotators and automatic approaches. To promote research in best practices for annotation of soft onsets, we have made all experimental data associated with this study publicly available. In addition, we publish the ARME Virtuoso Strings dataset, consisting of over 144 recordings of professional performances of an excerpt from Haydn's string quartet Op. 74 No. 1 Finale, each with corresponding individual instrumental onset annotations.

eess.AS↗
arXiv2026

Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning

Accent text-to-speech (TTS) aims to synthesize speech with a target accent while preserving speaker identity, but faces two key challenges: disentangling accent from speaker characteristics and effectively conditioning speech generation on the two disentangled factors. In this paper, we propose Joycent, a diffusion-based accent TTS framework that addresses both challenges. Our key idea is to separate accent and speaker information in both representation learning and TTS conditioning. Joycent uses WhisAID, a Whisper-based accent encoder with gradient reversal to learn speaker-disentangled accent representations, and introduces layer-specific conditional layer normalization to inject accent and speaker information at different stages of the text encoder. We evaluate Joycent on the Mandarin Regional Accent Corpus (MRAC) with seen and unseen speakers, including a challenging cross-accent setting where the speaker and accent prompts come from different accents. Experimental results show that Joycent improves accent similarity over existing methods while maintaining strong speaker similarity, with consistent gains under the challenging cross-accent setting. Subjective evaluation further confirms improved naturalness, accent similarity, and speaker preservation. The audio samples are available at https://oshindow.github.io/joycent/.

cs.SD↗
arXiv2026

How Well Do Generative Music Models Follow Emotion Conditioning?

Recent generative music models offer increasingly fine-grained control through text and audio conditioning, yet how faithfully they follow intended emotional cues remains an open question. We address this gap with a unified evaluation pipeline for emotion-following in generated music. Using all 1000 tracks in GTZAN, we extract semantic audio descriptions with DashengLM, an audio captioning model, and estimate source-track valence and arousal with Music2Emotion, a music emotion recognition model. We construct affect-aware text prompts by combining descriptions with top-ranked emotion tags and generate 30-second outputs with three systems, Stable Audio Open, MusicGen, and InspireMusic, evaluating both text- and audio-conditioned generation. To measure emotion-following, we compute valence and arousal on generated audio and compare them with the source tracks using absolute error and Euclidean distance in valence-arousal space. Text-conditioned generation consistently outperforms audio conditioning, with MusicGen (text) and InspireMusic (text) achieving the best performance, while audio-conditioned variants prove less stable. We further find that valence is preserved more reliably than arousal and that emotion-following varies substantially across genres. These findings underscore the importance of evaluating affective controllability directly rather than relying solely on general quality or prompt-relevance metrics.

cs.SD↗
arXiv2026

From Audio Deepfake Detection to AI-Generated Music Detection -- A Pathway and Overview

As Artificial Intelligence (AI) technologies continue to evolve, their use in generating realistic, contextually appropriate content has expanded into various domains. Music, an art form and medium for entertainment deeply rooted in human culture, is seeing an increased involvement of AI into its production. However, the unregulated use of AI music generation (AIGM) tools raises concerns about potential negative impacts on the music industry, copyright, and artistic integrity, underscoring the importance of effective AIGM detection. This paper provides a systematic overview of existing AIGM detection methods. We first establish a four-level detection taxonomy: signal-level, feature-level, watermark, and semantic consistency, organising methods according to the type of trace they exploit. Drawing on the more mature field of audio deepfake detection, we then present a stratified transferability analysis that examines which components may or may not transfer to AIGM detection, and under what conditions. A multi-dimensional classification further organises representative methods along input modality, detection granularity, feature type, model type, detection target, robustness setting, and interpretability. We conclude by discussing implications and proposing directions for future research to address ongoing challenges in the field.

cs.SD↗
arXiv2026

UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models

Spoken dialogue models currently lack the ability for fine-grained speech style control, a critical capability for human-like interaction that is often overlooked in favor of purely functional capabilities like reasoning and question answering. To address this limitation, we introduce UltraVoice, the first large-scale speech dialogue dataset engineered for multiple fine-grained speech style control. Encompassing over 830 hours of speech dialogues, UltraVoice provides instructions across six key speech stylistic dimensions: emotion, speed, volume, accent, language, and composite styles. Fine-tuning leading models such as SLAM-Omni and VocalNet on UltraVoice significantly enhances their fine-grained speech stylistic controllability without degrading core conversational abilities. Specifically, our fine-tuned models achieve improvements of 29.12-42.33% in Mean Opinion Score (MOS) and 14.61-40.09 percentage points in Instruction Following Rate (IFR) on multi-dimensional control tasks designed in the UltraVoice. Moreover, on the URO-Bench benchmark, our fine-tuned models demonstrate substantial gains in core understanding, reasoning, and conversational abilities, with average improvements of +10.84% on the Basic setting and +7.87% on the Pro setting. Furthermore, the dataset's utility extends to training controllable Text-to-Speech (TTS) models, underscoring its high quality and broad applicability for expressive speech synthesis. The complete dataset and model checkpoints are available at: https://github.com/bigai-nlco/UltraVoice.

eess.AS↗
arXiv2026

PairAlign: A Framework for Autoregressive Tokenization via Self-Alignment with Applications to Audio Tokenization

Modern learning systems represent perceptual signals with continuous vectors, but comparison, retrieval, memory, alignment, and reasoning are often symbolic. In language, tokens provide this interface; for speech and audio, it must be learned. Existing audio tokenizers rely on local quantization, clustering, or reconstruction, leaving sequence consistency, compactness, length, termination, and edit geometry only indirectly controlled. We introduce PairAlign, a framework for compact audio tokenization through autoregressive self-alignment. An encoder maps speech to a continuous condition, and an autoregressive decoder emits tokens from BOS to EOS. Given two content-preserving views, PairAlign derives a canonical anchor target and trains both views to predict it, with unrelated in-batch targets as competing sequences. It first learns an autoregressive bridge from VQ targets and then transitions to EMA-teacher self-alignment with grounding and anti-collapse controls. On 3 s speech, PairAlign produces shorter, non-degenerate sequences while retaining ordered paired-view consistency. On the TIMIT retrieval archive, it operates at 8.28 tokens/s, reducing archive token count by 70.65% and the edit-distance work proxy by 91.87% relative to Stage I. Positive-negative probes show strong separation across phone-disjoint, trigram-disjoint, and temporally rearranged negatives, while rate-controlled post-hoc BPE does not recover the same compactness-consistency operating point. These results expose a rate-granularity trade-off: PairAlign does not uniformly outperform denser tokenizers on every local metric, but provides a lower-rate symbolic interface preserving ordered and relational structure. Conceptually, PairAlign follows JEPA-style predictive learning by predicting an abstract target rather than reconstructing the input; here, the target is a learned variable-length symbolic sequence.

cs.LG↗
arXiv2026

FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations

Few-shot prompting provides an effective way to adapt auditory large language models to low-resource tasks such as children's speech recognition. However, most auditory large language models are not explicitly trained to perform inference in this demonstration-conditioned format, limiting the extent to which they can benefit from In-Context Learning (ICL). To address this limitation, we introduce Few-Shot Aware GRPO (FSA-GRPO), an RL-based post-training recipe that uses a specially designed reward to encourage the model to leverage few-shot demonstrations, thereby strengthening its few-shot adaptation ability. Notably, training with only 2k high-resource adult ASR utterances improves the model's general few-shot adaptation ability, yielding gains not only in children's speech recognition (53.9% relative WER reduction without any in-domain training) but also in multilingual ASR (including low-resource languages), speech translation, and audio understanding. We further study data selection and the weight and similarity cutoffs of the auxiliary reward to identify an effective training recipe. Our experiments show that when in-domain data are unavailable or cannot be used for training, FSA-GRPO is more effective than direct tuning on related out-of-domain data.

eess.AS↗
arXiv2026

SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech

Large audio language models (LALMs) are increasingly deployed in real-world applications, yet their safety alignment is still primarily evaluated on monolingual, text-based harmful prompts. This leaves their generalizability under multilingual and spoken settings, particularly code-switched speech, largely underexplored. To address this gap, we introduce SpeechJBB, an audio jailbreak dataset for benchmarking state-of-the-art LALMs across five European languages: English, French, German, Italian, and Spanish, as well as code-switched variants combining pairs of these languages. The extent of safety weaknesses is further probed by introducing an augmented setting where phonologically plausible pseudo-words are inserted around safety-critical terms to simulate localized obfuscation. Across models, code-switched harmful audio yields substantially high jailbreak success rates (JSR), with non-English monolingual and non-English code-switched pairs exhibiting the highest attack success. Pseudo-word insertion monotonically reduces refusal as insertion density increases, even though models rarely attribute harmful meaning to the inserted tokens. Comprehension benchmarks show that these failures are not reducible to multilingual misunderstanding, as several models with strong ASR, spoken language understanding, and spoken reasoning performance are among the most vulnerable.

cs.SD↗
arXiv2026

Disentangling Representation using Attributes-based Gaussian Estimation for Medical Sound Diagnosis

Deep learning has a powerful capability of feature extraction. However, the lack of fairness and interpretability in deep neural networks poses limitations to their adoption in the medical domain. This paper proposes a disentangled representation learning (DisenRL) framework, named the Attributes-based Gaussian Estimation for Disentangled Representation (AGEDR), which incorporates Attribute Mapping Embedding (AME) modules designed to map attributes into vectors and align them with a subset of the latent vectors in a Variational AutoEncoder (VAE). This part of the latent vector will be disentangled from the remaining latent vectors by minimizing mutual information. A classifier is then trained using the mean parameters of the latent vectors from the VAE. Extensive experiments demonstrate that AGEDR outperforms both conventional classification models and existing disentangled representation learning methods. The ablation experiments also indicate the disentangling capability and fairness of AGEDR. The source code is publicly available at https://github.com/ZhaoKe1024/DisentangledRepr.

cs.AI↗
arXiv2026

Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages

Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.

cs.CL↗
arXiv2026

MLAAD: The Multi-Language Audio Anti-Spoofing Dataset

This paper presents the Multi-Language Audio Anti-Spoofing Dataset (MLAAD), version 11: a dataset of synthetic audio to train and evaluate audio deepfake detection models. It features 205 Text-to-Speech (TTS) models, comprising a total of 1153.6 hours of synthetic voice in 54 different languages. To evaluate this dataset, we train three state-of-the-art deepfake detection models with MLAAD and observe that it demonstrates superior performance to comparable datasets like InTheWild and FakeOrReal when used as a training resource. Moreover, compared to the renowned ASVspoof 2019 dataset, MLAAD proves to be a complementary resource. In tests across eight datasets, MLAAD and ASVspoof 2019 alternately outperformed each other, each excelling on four datasets. By publishing the dataset and making a trained model accessible via an interactive webserver, we aim to democratize anti-spoofing technology, making it accessible beyond the realm of specialists, and contributing to global efforts against audio spoofing and deepfakes.

cs.SD↗
arXiv2026

Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity

Modeling speech variation is key to natural, expressive generation. Speaker embeddings are commonly used to condition personalized speech systems, but they are typically trained for speaker recognition, where intra-speaker variability is suppressed and inter-speaker separation is maximized. This objective leads to overly compact representations that may discard variations crucial for generation. We revisit this design choice and propose a sub-center modeling framework for speaker embeddings. Instead of a single prototype per speaker, we learn multiple sub-centers during discriminative training, allowing utterances to align with different prototypes. This strategy preserves structured intra-speaker variability while maintaining discriminability. In zero-shot voice conversion, our method improves intelligibility, increases pitch variability, achieves higher naturalness ratings, and retains strong speaker verification performance.

eess.AS↗
arXiv2026

DiffAnon: Diffusion-based Prosody Control for Voice Anonymization

To preserve or not to preserve prosody is a central question in voice anonymization. Prosody conveys meaning and affect, yet is tightly coupled with speaker identity. Existing methods either discard prosody for privacy or lack a principled mechanism to control the utility-privacy trade-off, operating at fixed design points. We propose DiffAnon, a diffusion-based anonymization method with classifier-free guidance (CFG) that provides explicit, continuous inference-time control over prosody preservation. DiffAnon refines acoustic detail over semantic embeddings of an RVQ codec, enabling smooth interpolation between anonymization strength and prosodic fidelity within a single model. To the best of our knowledge, it is the first voice anonymization framework to provide structured, interpolatable inference-time prosody control. Experiments demonstrate structured trade-off behavior, achieving strong utility while maintaining competitive privacy across controllable operating points.

eess.AS↗
arXiv2026

Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models

Full-duplex spoken dialogue models can listen and speak simultaneously, making them a promising architecture for natural conversation. However, current models are trained solely with supervised learning through token-level likelihood maximization, which does not directly optimize interaction-level behaviors, causing interactivity issues such as excessive silence and ill-timed turn-taking. Recent work has applied reinforcement learning (RL) to improve interactivity, but existing methods address only a limited set of interactive behaviors in their rewards. In this work, we propose a post-training alignment method that comprehensively improves the interactivity of full-duplex spoken dialogue models through RL. We address the four canonical axes of interactivity: pause handling, turn-taking, backchanneling, and user interruption. For each axis, we extract short audio segments from human conversation corpora and optimize the model with axis-specific reward functions. An extra LLM-based reward for response quality prevents semantic degradation. We apply our method to two open-source models, Moshi and PersonaPlex, demonstrating consistent improvements in interactivity on both offline evaluation with pre-recorded audio and real-time multi-turn dialogue evaluation.

cs.CL↗
Compare source metadata on this page
WorkPublishedSource identifierSource
FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation2026-08-312601.06199arxiv
MUSE: A Run-Centric Platform for Multimodal Unified Safety Evaluation of Large Language Models2026-08-312603.02482arxiv
KoALa-Bench: Evaluating Large Audio Language Models on Korean Speech Understanding and Faithfulness2026-08-312604.19782arxiv
Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems2026-08-312608.30348arxiv
Annotation of Soft Onsets in String Ensemble Recordings2026-08-302211.08848arxiv
Joycent: Multi-Accent TTS via Disentangled Accent Modeling and Layer-Specific Conditioning2026-08-302606.16417arxiv
How Well Do Generative Music Models Follow Emotion Conditioning?2026-08-302608.29987arxiv
From Audio Deepfake Detection to AI-Generated Music Detection -- A Pathway and Overview2026-08-292412.00571arxiv
UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models2026-08-292510.22588arxiv
PairAlign: A Framework for Autoregressive Tokenization via Self-Alignment with Applications to Audio Tokenization2026-08-292605.06582arxiv
FSA-GRPO: Teaching Auditory LLMs to Use Few-Shot Demonstrations2026-08-292606.02615arxiv
SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech2026-08-292606.06037arxiv
Disentangling Representation using Attributes-based Gaussian Estimation for Medical Sound Diagnosis2026-08-292608.29026arxiv
Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages2026-08-292608.29239arxiv
MLAAD: The Multi-Language Audio Anti-Spoofing Dataset2026-08-282401.09512arxiv
Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity2026-08-282407.04291arxiv
DiffAnon: Diffusion-based Prosody Control for Voice Anonymization2026-08-282604.26281arxiv
Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models2026-08-282606.11167arxiv

These are bibliographic comparisons, not experimental rankings. Follow the original document for methods and conditions.