arXiv · 2611.18589
OmniTag-LLM: Multilingual Multitask Speech Tagging
Abstract
Multimodal LLMs promise to replace the traditional cascade of specialist speech models with a single system that, from one audio clip, jointly transcribes and labels emotion, gender, language, and named entities. Yet most benchmarks still score these tasks in isolation, and largely in English. We introduce OmniTag-Bench, a five-language benchmark (en/de/es/fr/it) and training recipe for joint understanding: one audio clip in, one JSON object out (transcript, emotion, gender, LID, token-level NER). We evaluate ten open multimodal LLMs and a LoRA fine-tuned Qwen3-Omni on a unified prompt and scorer. Beyond aggregate scores, OmniTag-Bench reveals a systematic failure mode: several strong models often transcribe or translate non-English speech into English, inflating multilingual WER. Fine-tuning improves the base model on NER, emotion, and WER while nearly doubling accuracy on independent human-labeled emotion corpora, suggesting transferable gains beyond pseudo-label imitation. The fine-tuned model also generalizes to unseen but related languages, transferring to held-out Dutch. We release prompts, the scorer, and the labeling pipeline.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Aref Farhadipour, Francesco Verdini, Antonis Asonitis, Pierre-Edouard Honnet, Marzieh Razavi, Vijeta Avijeet, Juan Pablo Zuluaga Gomez. 2026-08-19. OmniTag-LLM: Multilingual Multitask Speech Tagging. https://searcharxiv.com/abs/2611.18589
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.