Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.07107

Neural Representations, Natural Connections: What Transfers From Human Speech Foundation Models to Animal Vocalizations?

Tomás Arias-Vergara·Christopher Hauer·Héloïse Brotier·Elmar Nöth·Andreas Maier·Lee Koren
View original source ↗Read source document ↗Export metadata ↓

Abstract

Speech-pretrained models have shown promise in animal bioacoustics, but the factors governing their cross-species transfer remain poorly understood. We evaluate 15 frozen encoders spanning monolingual and multilingual speech, speaker verification, animal bioacoustics, and general audio on caller identification across four species and call-type classification across three. Differences between the strongest speech- and animal-pretrained representations range from $-0.027$ to $+0.119$ UAR; speech is significantly better in three of seven settings ($p<0.01$) and not significantly different in the remaining four. Controlled comparisons show no systematic advantage from increased language coverage or animal-domain pretraining, while frozen speaker-verification embeddings transfer poorly. The results also show that transfer is strongly layer-dependent: raw-waveform models peak early, whereas patch-spectrogram models peak deeper.

Explore related subjects

cs.SDeess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tomás Arias-Vergara, Christopher Hauer, Héloïse Brotier, Elmar Nöth, Andreas Maier, Lee Koren. 2026-10-05. Neural Representations, Natural Connections: What Transfers From Human Speech Foundation Models to Animal Vocalizations?. https://arxiv.org/abs/2610.07107

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Adaptive Duration Model for Text Speech Alignment

Speech-to-text alignment is a critical component of neural text to speech (TTS) models. Autoregressive TTS models typically use an attention mechanism to learn these alignments on-line, while non-autoregressive end to end TTS models rely on durations extracted from external sources. In this paper, we propose a novel duration prediction framework that can give promising phoneme-level duration distribution with given text. In our experiments, the proposed duration model has more precise prediction and adaptation ability to conditions, compared to previous baseline models. Specifically, it makes a considerable improvement on phoneme-level alignment accuracy and makes the performance of zero-shot TTS models more robust to the mismatch between prompt audio and input audio.

cs.SD↗
arXiv2026

FastTurn: Unifying Acoustic and Streaming Semantic Cues for Low-Latency and Robust Turn Detection

Recent advances in AudioLLMs have enabled spoken dialogue systems to move beyond turn-based interaction toward real-time full-duplex communication, where the agent must decide when to speak, yield, or interrupt while the user is still talking. Existing full-duplex approaches either rely on voice activity cues, which lack semantic understanding, or on ASR-based modules, which introduce latency and degrade under overlapping speech and noise. Moreover, available datasets rarely capture realistic interaction dynamics, limiting evaluation and deployment. To mitigate the problem, we propose \textbf{FastTurn}, a unified framework for low-latency and robust turn detection. To advance latency while maintaining performance, FastTurn combines streaming CTC decoding with acoustic features, enabling early decisions from partial observations while preserving semantic cues. We also release a test set based on real human dialogue, capturing authentic turn transitions, overlapping speech, backchannels, pauses, pitch variation, and environmental noise. Experiments show FastTurn achieves higher decision accuracy with lower interruption latency than representative baselines and remains robust under challenging acoustic conditions, demonstrating its effectiveness for practical full-duplex dialogue systems.

cs.SD↗
arXiv2026

AT-ADD: A Benchmark and Challenge for Robust and All-Type Audio Deepfake Detection

Recent audio generation models can synthesize high-fidelity speech, environmental sound, singing voice, and music, creating new risks for multimedia trust. Existing audio deepfake detection (ADD) benchmarks remain predominantly speech-centric and often under-represent realistic channel variation and heterogeneous audio types. This paper presents all-type ADD (AT-ADD), a large-scale benchmark and challenge designed to evaluate both robust speech deepfake detection and all-type audio deepfake detection. Track 1 evaluates speech deepfake detection under unseen synthesizers, diverse recording conditions, signal perturbations, and replay effects. Track 2 evaluates type-agnostic deepfake detection over speech, sound, singing, and music when the audio type is unknown at test time. We detail the dataset construction, evaluation protocol, and reproducible baselines, and analyze the final systems submitted to the ACM Multimedia 2026 Grand Challenge. The strongest official baseline obtains 76.73% and 79.47% Macro-F1 on the Track 1 and Track 2 evaluation sets, respectively, whereas the winning challenge systems reach 90.71% and 96.10%. Beyond the aggregate rankings, sample-level analysis of the top five submissions reveals concentrated failures on specific synthesis methods and replayed speech. It also quantifies cross-system error complementarity and separates stable leaderboard gaps from statistically ambiguous adjacent ranks. Our results show that large-scale self-supervised representations, condition-aware augmentation, multi-crop inference, and structured fusion or routing are central to generalization, while generator-specific robustness and balanced detection across heterogeneous audio types remain unresolved.

cs.SD↗