subject

cs.SD

cs.SD: explore 212 source-linked works published from 2024 to 2026, with original documents and citations.

This collection is a preview while coverage and quality are evaluated.

Coverage and selection

Includes records with this source-supplied label or an explicit phrase match in their metadata. Matches indicate a mention, not proof that a paper uses a method or tests a material. Source versions are consolidated by DOI.

Sources: arxiv. Collection updated 2026-09-16. Counts describe this index, not the complete source archives.

arXiv2026

VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing

Recent audio-language models are increasingly evaluated on recognizing, describing, and reasoning about audio, but expert-facing applications require a different capability: producing feedback that identifies problems and suggests corrective actions grounded in the input. We introduce VocalCoachBench, a benchmark for evaluating audio-language models on expert vocal coaching feedback for singing. VocalCoachBench contains 515 recordings annotated by 18 professional vocal trainers, yielding 1,056 expert submissions and 12,051 atomic coaching claims. It comprises a same-song subset for controlled comparison and a diverse-song subset for segment-grounded feedback across varied songs and recording conditions. To accommodate the open-ended nature of expert feedback, VocalCoachBench sep- arates deterministic structured targets from claim-based assessment of free-form diagnosis and corrective guidance. Human annotation analysis shows that expert agreement varies strongly with label granularity, motivating hierarchical structured metrics and claim-based evaluation of open-ended feedback. Experiments with 12 recent audio-language models reveal a consistent gap: while models can compare performances and identify broad issue domains in free-form feedback, Top-3 fine-grained issue-label identification remains below label-prior baselines and strict diagnosis alignment stays below 7%. To our knowledge, VocalCoachBench pro- vides the first public testbed for evaluating audio-grounded expert feedback for singing, moving audio-language evaluation beyond description toward analytic feedback.

cs.SD↗

arXiv2026

Integrating Human Linguistic Insights into AI: Theory-Driven Representation for Multilingual Text-to-Speech

This paper explores the integration of human linguistic insights into multilingual text-to-speech (TTS) systems by evaluating the Featurally Underspecified Lexicon (FUL) as a theory-driven input representation. Unlike data-intensive end-to-end models, FUL offers a compact, interpretable feature set grounded in phonological principles, enabling scalable and equitable TTS development for low-resource languages. We provide a mapping from language-specific phones to FUL feature vectors via a SAMPA intermediate and incorporate these features into a modified FastSpeech architecture. Experiments were conducted to evaluate their ability to generate native, non-native, and code-mixed speech in English and Mandarin. We ran an experiment with a small dataset and one with a larger dataset, which showed that TTS with FUL features as input could produce intelligible native speech with as little as 8 hours of training data; with 100 hours of training data, intelligible speech could be generated for a language not present in the training data. The approach further supports code-mixed synthesis while preserving consistent timbre and interpretable phonetic control. These results highlight the potential of theory-driven representations for building efficient, scalable, and linguistically informed TTS systems, demonstrating that phonological features can function as both analytical tools and practical inputs for speech technology.

cs.CL↗

arXiv2026

TurnFSM for Full-Duplex Dialogue System: Internalizing State-Machine Logic for Streaming Semantic Voice Activity Detection and Utterance-Level Rejection

Full-duplex voice assistants must continuously listen while speaking, handling user interruptions under low-latency and resource-constrained streaming conditions. Existing end-to-end full-duplex models can compromise reasoning-related capabilities after speech-domain adaptation, whereas cascaded pipelines introduce extra inference overhead and handcrafted control logic. We propose TurnFSM, an LLM-based state prediction framework that internalizes turn control as explicit finite-state transitions, unifying streaming semantic VAD and utterance-level rejection. TurnFSM decomposes submission and rejection into a serial decision process, reducing multi-task interference while maintaining performance comparable to single-task models. We further introduce a first-order state transition mechanism that enforces the dependency on only the previous state during training, enabling compact inference with the standard causal mask and original LLM positional encoding while avoiding historical state-token accumulation and unnecessary step-by-step state generation. Experimental results show that TurnFSM consistently outperforms the binary-head baseline and remains competitive with task-specific models.

eess.AS↗

arXiv2026

Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework

Compared with half-duplex dialogue systems where the system waits for user turn completion before it responds, natural full-duplex dialogue systems require agents to act proactively in real time, including timely interruptions and backchannels. This creates a key challenge: improving turn timing without sacrificing response quality. To address limitations in realistic proactive turn-taking, we build a generalized style-aware full-duplex framework with three key components. Firstly, we propose LPS-TC, a Lightweight Proactive Speech Turn Controller for plug-and-play integration. It features a fine-grained action space covering both reactive and proactive turn behaviors, enabling half-duplex models with full-duplex capabilities and enhancing existing full-duplex models with superior timing control. Secondly, we construct WildTurn, a large-scale, real-world English dataset containing approximately 2,981 hours of filtered multi-turn stereo conversations from face-to-face and telephone conversations, annotated with five turn-taking and five backchanneling styles. Trained on WildTurn, LPS-TC exhibits rich spoken dynamics that are not captured by existing static full-duplex benchmarks. Thirdly, we introduce a two-tier evaluation scheme that assesses both chunk-level timing precision and turn-level interaction quality under realistic streaming constraints. Our experiments, integrating LPS-TC with half-duplex models like Qwen2.5-Omni and full-duplex models like Freeze-Omni, showcase its superior performance in timing appropriateness and response quality. Our framework also demonstrates fine-grained style controllability and strong generalizability, enabling more natural and human-like spoken interactions.

cs.CL↗

arXiv2026

Robust Speech Emotion Recognition under Tone-Word Conflict: A Benchmark and Framework

Speech emotion recognition (SER) is a crucial component of human-computer interaction, attracting extensive attention from both industry and academia. However, existing SER systems typically assume alignment between vocal tone and lexical semantics, overlooking the real-world scenarios that involve tone-word conflict-where the emotion conveyed by speech contradicts the literal meaning of the words. To bridge this gap, we introduce TWIN-SER (Tone-Word Incongruent SER), a benchmark for systematic evaluation under acoustic-semantic incongruence, and show that state-of-the-art models degrade severely under such incongruence. To address this, we propose DAS (Disentangled Acoustic-Semantic fusion), a framework that mitigates tone-word conflict by explicitly disentangling acoustic and semantic pathways, selecting informative high-energy embeddings, and adaptively fusing them via a lightweight query-based attention mechanism. Specifically, DAS comprises three crucial modules: i) a heterogeneous feature extraction module that separately captures complementary acoustic and semantic representations from raw input; ii) a high-energy embedding selection module that identifies and retains the most discriminative embeddings; and iii) a Q-Former combination module that bridges the two pathways through cross-attention, enabling robust emotion prediction under incongruent conditions. Extensive experiments demonstrate that DAS consistently outperforms existing methods in tone-word conflict scenarios, as well as in standard in-domain and zero-shot settings. Our code and datasets are available at https://github.com/24DavidHuang/FAS

eess.AS↗

arXiv2026

Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling

Recent advances in speech language models leverage discrete speech representations from pretrained codecs to enable scalable training and generation. However, existing codecs are primarily optimized for compression without accounting for the autoregressive nature of language model training, resulting in suboptimal performance when modeling compressed speech tokens. In this work, we revisit speech discretization from a generative modeling perspective and propose a novel framework that explicitly aligns speech tokenization with autoregressive training. Our approach introduces autoregressive-compatible constraints during codec training, encouraging token sequences that exhibit temporal consistency and predictability. In addition, we propose a heterogeneous downsampling strategy for different layers of speech tokens, distinguishing semantic from acoustic layers, to improve the alignment between semantic tokens and corresponding textual content. Extensive experiments across multiple benchmarks demonstrate that our method bridges the gap between speech compression and generative modeling, enabling more effective continued pretraining of existing language models on speech data. The approach consistently improves performance across multiple codecs, validating its generality and applicability to diverse speech modeling scenarios.

eess.AS↗

arXiv2026

Beyond SDR: How Music Source Separation Reshapes Rhythm-Relevant Signal Properties

Music source separation (MSS) is increasingly used not to remix music but to measure it: separated drum stems feed studies of microtiming, dynamics, and groove. The field evaluates separators almost exclusively by signal-to-distortion ratio (SDR), yet microrhythm research shows that a sound's perceived temporal location (its p-centre) is co-determined by its attack and envelope, precisely the properties SDR was not designed to protect. We quantify what four open separators spanning four architecture generations (Spleeter, HT-Demucs, BS-Roformer, SCNetXL) do to rhythm-critical signal properties, using the 50-track MUSDB18-HQ test set, where true stems make every claim falsifiable. Three findings emerge. (1) Onset timing is safe: onset F-measure tracks SI-SDR (Spearman rho = 0.62) and is invariant to input length. (2) Transient and dynamic shape are not: their distortion correlates only weakly with SI-SDR (|rho| <= 0.29), and the model ranking inverts - the SDR leader distorts drum attacks twice as much as its capability-matched CNN counterpart, while the SDR-worst model preserves dynamics better than a mid-pack one. Each model imposes a systematic, model-specific bias on the dynamic profile. (3) Input length reshapes the rendered attack of a fixed passage (marginally more for the transformer; paired p = 0.044) while leaving onset locations untouched. For rhythmic studies, separator choice and input conditions are methodological variables to be reported, and SDR alone cannot stand in for them.

cs.SD↗

arXiv2026

The Trade-off Was in the Labels: Causal Supervision for Turn-Aware Streaming ASR

A voice agent must decide, moment to moment, whether the user has finished; silence rarely settles it: a caller reading a phone number pauses mid-digits, a one-word "Stop!" ends a turn, a long question carries pauses longer than real turn-gaps. A voice-activity detector plus a silence timeout (the deployed default) cannot separate these, because within-turn pauses routinely exceed between-turn gaps; what distinguishes them is whether the words so far form a complete thought: what a recognizer computes to produce a transcript. We present the first open training recipe and benchmark for turn-aware streaming ASR: a small LoRA adapter on Qwen3-ASR-0.6B, trained in hours on one GPU, that transcribes, detects end-of-turn from meaning and silence, handles dictation, and grounds transcription in context. On a deployment-matched benchmark it reaches 0.97 boundary recall at 0.39 s median latency with 0.3 false fires per speech-minute, replicated on a fresh test set; no silence timeout reaches this point. The recipe rests on one principle: every streaming-decision label must be computable from input up to the decision point. Offline corpora violate it, encoding the future; such clairvoyant labels manufactured oscillation and a phantom recall-versus-precision trade-off, exposed when one appended second of silence raised a "broken" model's end-of-turn recall from 0.10 to 1.00. The same leak recurred with context: an always-matching biasing prefix became a copied shortcut (40% intrusion), and counterfactuals disagreeing with the audio cut this to 0.8% while keeping most of a +28.9 pp entity-recall benefit.

eess.AS↗

arXiv2026

Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition

Modern automatic speech recognition (ASR) scenarios require both spoken-form transcripts for faithful transcription and readable written-form transcripts with inverse text normalization (ITN). However, these forms are typically produced by cascaded modules, where a spoken-form ASR output is rewritten by a separate ITN component, making written-form ASR-ITN vulnerable to recognition errors and decoupling normalization from acoustic-contextual modeling, especially for semantically dependent numeric expressions. In this paper, we propose Dual-Form ASR (DF-ASR), a framework that extends spoken-form ASR capability to semantics-aware written-form ITN through paired spoken-form and written-form supervision while retaining prompt-level selection between transcript forms. The dual-form supervision is constructed via a large language model (LLM)-driven generate-and-judge workflow, and training is further enhanced by ITN-MWER, a sequence-level objective that assigns higher cost to errors on normalization-sensitive spans. We also introduce a decision-aware REQUIRE-ITN/\FORBID-ITN protocol to separately measure required normalization and forbidden-span preservation. On manually annotated Chinese subsets from SpeechIO, DF-ASR consistently outperforms open-source ASR-ITN systems, remains competitive with strong closed-source references, and preserves reliable prompt-level control between spoken-form and written-form outputs.

cs.CL↗

arXiv2026

GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue

We present GEPARD (Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue), a streaming text-to-speech model for real-time spoken dialogue. GEPARD generates speech autoregressively with an LLM backbone - text and audio embeddings are trained together in a single decoder-only model - and decodes it to a waveform with an FSQ-based neural codec, streaming audio chunk-by-chunk as text arrives. Our central goal is a TTS architecture served by a standard LLM engine (vLLM) without modifying its compute kernels. This defines the overarching design principle: the backbone is a standard full-attention transformer, while all non-trivial auxiliary mechanisms - zero-shot voice cloning, text augmentation, and classifier-free guidance - are moved out of the autoregressive decode loop into prefill, or distilled directly into the weights. On streaming end-to-end inference, a single stream reaches a Real-Time Factor of about 0.067 (roughly 15x faster than real-time); under 256 concurrent streams the system reaches an aggregate speedup of about 204x on a single server-class GPU. We detail: (1) system-level solutions for vLLM-native serving; (2) the "short register" (1-2 word) failure mode of autoregressive speech decoders, with diagnostic probes and a mitigation; and (3) distillation of two-pass classifier-free guidance over text into single-pass weights via Direct Preference Optimization (DPO).

eess.AS↗

arXiv2026

Auditing Bias and Safety in Voice AI Customer Care

Voice AI systems increasingly mediate customer care interactions where caller presentation cues such as accent, affect, fluency, and urgency are available alongside the service request. Existing fairness and safety evaluations cover speech recognition disparities, spoken dialogue bias, and voice agent capability, but rarely treat customer care voice agents as stateful, multi turn, tool mediated systems where harm can appear as additional burden before any final denial occurs. We formalize a validation gated audit framework for such systems. The framework (i) separates native speech to speech, cascaded ASR to language model to TTS, and hybrid tool mediated architectures; (ii) uses matched service facts across controlled caller presentation conditions; (iii) validates fact invariance, presentation cues, artifacts, and acoustic measurements before inference; and (iv) records both material outcomes and path to service burden. We define the research problem, methodology, seven validation gates, a six family metric set, and claim boundaries for an active industry evaluation program. We illustrate the framework with a fully synthetic worked example of a refund dispute audit instance. Production system results are excluded from this release; public reporting is gated by the validation protocol.

eess.AS↗

arXiv2025

SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysis

Despite growing attention to deepfake speech detection, the aspects of bias and fairness remain underexplored in the speech domain. To address this gap, we introduce the Speaker Characteristics Deepfake (SCDF) dataset: a novel, richly annotated resource enabling systematic evaluation of demographic biases in deepfake speech detection. SCDF contains over 237,000 utterances in a balanced representation of both male and female speakers spanning five languages and a wide age range. We evaluate several state-of-the-art detectors and show that speaker characteristics significantly influence detection performance, revealing disparities across sex, language, age, and synthesizer type. These findings highlight the need for bias-aware development and provide a foundation for building non-discriminatory deepfake detection systems aligned with ethical and regulatory standards.

cs.SD↗

arXiv2024

BUT Systems and Analyses for the ASVspoof 5 Challenge

This paper describes the BUT submitted systems for the ASVspoof 5 challenge, along with analyses. For the conventional deepfake detection task, we use ResNet18 and self-supervised models for the closed and open conditions, respectively. In addition, we analyze and visualize different combinations of speaker information and spoofing information as label schemes for training. For spoofing-robust automatic speaker verification (SASV), we introduce effective priors and propose using logistic regression to jointly train affine transformations of the countermeasure scores and the automatic speaker verification scores in such a way that the SASV LLR is optimized.

cs.SD↗

arXiv2024

Approaching an unknown communication system by latent space exploration and causal inference

This paper proposes a methodology for discovering meaningful properties in data by exploring the latent space of unsupervised deep generative models. We combine manipulation of individual latent variables to extreme values with methods inspired by causal inference into an approach we call causal disentanglement with extreme values (CDEV) and show that this method yields insights for model interpretability. With this, we can test for what properties of unknown data the model encodes as meaningful, using it to glean insight into the communication system of sperm whales (Physeter macrocephalus), one of the most intriguing and understudied animal communication systems. The network architecture used has been shown to learn meaningful representations of speech; here, it is used as a learning mechanism to decipher the properties of another vocal communication system in which case we have no ground truth. The proposed methodology suggests that sperm whales encode information using the number of clicks in a sequence, the regularity of their timing, and audio properties such as the spectral mean and the acoustic regularity of the sequences. Some of these findings are consistent with existing hypotheses, while others are proposed for the first time. We also argue that our models uncover rules that govern the structure of units in the communication system and apply them while generating innovative data not shown during training. This paper suggests that an interpretation of the outputs of deep neural networks with causal inference methodology can be a viable strategy for approaching data about which little is known and presents another case of how deep learning can limit the hypothesis space. Finally, the proposed approach can be extended to other architectures and datasets.

stat.ML↗

Compare source metadata on this page

Work	Published	Source identifier	Source
VocalCoachBench: Benchmarking Audio-Language Models on Expert Feedback for Singing	2026-10-12	2609.04241	arxiv
Integrating Human Linguistic Insights into AI: Theory-Driven Representation for Multilingual Text-to-Speech	2026-08-05	2204.07228	arxiv
TurnFSM for Full-Duplex Dialogue System: Internalizing State-Machine Logic for Streaming Semantic Voice Activity Detection and Utterance-Level Rejection	2026-08-05	2609.04240	arxiv
Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework	2026-08-04	2608.28630	arxiv
Robust Speech Emotion Recognition under Tone-Word Conflict: A Benchmark and Framework	2026-07-27	2609.04236	arxiv
Rethinking Speech Codecs: From Compression to Autoregressive Generative Modeling	2026-07-27	2609.04237	arxiv
Beyond SDR: How Music Source Separation Reshapes Rhythm-Relevant Signal Properties	2026-07-12	2609.04224	arxiv
The Trade-off Was in the Labels: Causal Supervision for Turn-Aware Streaming ASR	2026-07-12	2609.04225	arxiv
Dual-Form ASR: Semantics-Aware Inverse Text Normalization for Chinese Speech Recognition	2026-07-06	2609.02901	arxiv
GEPARD - Generative, Prosody-aware, Autoregressive text-to-speech model for Realtime Dialogue	2026-07-04	2609.04222	arxiv
Auditing Bias and Safety in Voice AI Customer Care	2026-05-18	2609.04206	arxiv
SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysis	2025-08-11	2508.07944	arxiv
BUT Systems and Analyses for the ASVspoof 5 Challenge	2024-08-20	2408.11152	arxiv
Approaching an unknown communication system by latent space exploration and causal inference	2024-02-06	2303.10931	arxiv

These are bibliographic comparisons, not experimental rankings. Follow the original document for methods and conditions.