Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.08966

Concordance-Optimal Continuous Music Emotion Estimation: Variance Decomposition, Dual-Branch Temporal Fusion, and Conformal Prediction Intervals

Mohammad Faraz Akhtar·Aditya Parashar·Ayush Jain·Madhur Gupta·Ritesh Kumar·Soubhik Chakraborty
View original source ↗Read source document ↗Export metadata ↓

Abstract

Continuous music emotion recognition (MER) estimates how Arousal and Valence evolve within a song, yet the field's standard metric, pooled frame-level correlation, is dominated by between-song differences and cannot distinguish a model that tracks emotional dynamics from one that only ranks songs by their average level. We decompose annotation variance into between- and within-song components and show that mean squared error (MSE) targets the former, while windowed Lin's concordance correlation coefficient (CCC = $r \cdot C_b$) targets the latter by penalising the variance compression MSE induces. A single weight $λ$ controls the trade-off: on 1,764 DEAM songs, raising $λ$ from 0 to 1 lowers pooled Arousal $r$ from 0.843 to 0.810 while raising mean within-song $r$ from 0.204 to 0.368. Building on this objective, we present MMAS (Multimodal Music Analysis System), an open-source model that predicts Arousal and Valence every 0.5 s from raw audio. It fuses frozen MERT-v1-95M embeddings with ten symbolic features from automatic transcription through a dual-branch temporal convolutional network with cross-modal attention. Because both branches are audio-derived, one model trains jointly on DEAM and PMEmo without domain conditioning. Under song-grouped 5-fold cross-validation (2,531 songs), pooled $r$ reaches 0.801/0.685 (Arousal/Valence) on DEAM and 0.742/0.746 on PMEmo. Against a ridge baseline on identical features, within-song Arousal $r$ roughly triples (0.344 vs. 0.125; $n = 353$ songs, Wilcoxon $p = 1.7 \times 10^{-21}$, $d = 0.56$). Complete transcription failure costs at most 0.04 pooled $r$. Split-conformal intervals attain 93.7% (Arousal) and 90.0% (Valence) marginal coverage at a 95% target, with the shortfall traced to within-song dependence. MMAS is released as a pip-installable package.

Explore related subjects

cs.SDcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Mohammad Faraz Akhtar, Aditya Parashar, Ayush Jain, Madhur Gupta, Ritesh Kumar, Soubhik Chakraborty. 2026-11-06. Concordance-Optimal Continuous Music Emotion Estimation: Variance Decomposition, Dual-Branch Temporal Fusion, and Conformal Prediction Intervals. https://arxiv.org/abs/2611.08966

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Encoding of musical structures in hidden units of restricted Boltzmann machines

Restricted Boltzmann machines (RBMs) are energy-based models originating from statistical physics. In this study, we investigate how musical structures are represented in the hidden layer of a Bernoulli-Bernoulli RBM trained on binary piano-roll representations of J.~S.~Bach's music. By examining the visible patterns induced by individual hidden units, we find that the hidden layer primarily captures local temporal and pitch-statistical features rather than explicit musical concepts such as melodies, chords, or keys. Free-energy comparisons show that the trained RBM assigns higher probabilities to the examined musical inputs than to digit or noise patterns. Analysis using t-SNE, cosine similarity, and PCA reveals that hidden representations are sensitive to absolute pitch and input overlap, with transposed versions of the same piece not necessarily mapped close together. Generated samples exhibit local pitch organization but limited long-range coherence. These findings illustrate how a simple statistical-physics model captures certain regularities in structured musical data while highlighting its limitations in representing higher-level musical relationships.

cs.SD↗
arXiv2026

Unsupervised Evaluation of Deep Audio Embeddings for Music Structure Analysis

Music Structure Analysis (MSA) aims to uncover the high-level organization of musical pieces. State-of-the-art methods are often based on supervised deep learning, but these methods are bottlenecked by the need for heavily annotated data and inherent structural ambiguities. In this paper, we propose an unsupervised evaluation of nine open-source, generic pre-trained deep audio models, on MSA. For each model, we extract barwise embeddings and segment them using three unsupervised segmentation algorithms (Foote's checkerboard kernels, spectral clustering, and Correlation Block-Matching (CBM)), focusing exclusively on boundary retrieval. Our results demonstrate that modern, generic deep embeddings generally outperform traditional spectrogram-based baselines, but not systematically. Furthermore, our unsupervised boundary estimation methodology generally yields stronger performance than recent linear probing baselines. Among the evaluated techniques, the CBM algorithm consistently emerges as the most effective downstream segmentation method. Finally, we highlight the artificial inflation of standard evaluation metrics and advocate for the systematic adoption of ``trimming'', or even ``double trimming'' annotations to establish more rigorous MSA evaluation standards.

cs.SD↗
arXiv2026

VoxCanvas-41M: A Large-Scale Bilingual Dataset for Instruction-Based Speech Editing and Its Learning Dynamics

Recent advances in speech generation have spurred the development of speech editing techniques. However, achieving high-quality speech editing remains challenging due to two key limitations. First, the major hurdle is the lack of large-scale, high-quality speech editing datasets. Constructing such datasets is costly, while open-source, scalable data pipelines remain limited. Second, the learning dynamics of speech editing models remain poorly understood. In particular, how models acquire editing capabilities and the relationship between generation and editing remain unclear. To address these challenges, we introduce VoxCanvas-41M, a large-scale bilingual dataset comprising approximately 41 million speech editing pairs. VoxCanvas-41M covers two major categories: content editing, including single-span, multi-span, and location-reference editing, and paralinguistic editing, covering the insertion and deletion of 39 classes of nonverbal events such as laughter, sighs, and coughs. Building on VoxCanvas-41M, we train VoxCanvas-Edit, an instruction-based speech editing model, demonstrating the effectiveness of our dataset in improving editing performance. Furthermore, we systematically investigate the learning dynamics of speech editing, examining how models learn different editing operations and how data scale, task interactions, and training strategies affect editing performance. The dataset, pipeline, and demos are publicly available.

cs.SD↗