From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition

Wei Wang; Zhaowu Li; Jianjie Luo; Fu Lee Wang; Lap-Kei Lee; Zhenguo Yang

Home/Archive/arXiv

arXiv · 2610.03016

From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition

Wei Wang·Zhaowu Li·Jianjie Luo·Fu Lee Wang·Lap-Kei Lee·Zhenguo Yang

View original source ↗Read source document ↗Export metadata ↓

Abstract

In this paper, we propose a Role-aware Stimulus-guided (RASG) framework for interlocutor emotion recognition, which predicts listener emotions from listener-only videos and speaker-only audios. RASG consists of Role-aware Visual Transfer (RVT) and Stimulus-guided Boundary Reasoning (SBR) modules, which address supervision mismatch due to the lack of labeled listener data and ambiguity among visually similar listener reactions whose interpretation depends on speaker context, respectively. More specifically, RVT selects speaker samples whose facial expressions support their emotion labels. It then filters listener tracks and uses reliable pseudo-labels to train a listener-centric visual expert. SBR uses a two-class language reasoner only when the visual model is uncertain. It treats speaker audio and text as context rather than direct emotion evidence to distinguish similar listener reactions. Experiments conducted on MER-Cross dataset shows that RASG achieves 76.25\% on MER-Cross and improves the performance of the baseline over 17\%. Our team ranks second in Track 1 (MER-Cross) of the MER Grand Challenge at ACM MM 2026.

Explore related subjects

cs.MM cs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Wei Wang, Zhaowu Li, Jianjie Luo, Fu Lee Wang, Lap-Kei Lee, Zhenguo Yang. 2026-10-02. From Expression to Reaction: Role-aware Visual Transfer and Stimulus-guided Reasoning for Interlocutor Emotion Recognition. https://arxiv.org/abs/2610.03016

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

FATE: Frame-Level Audio-Visual Temporal Embedding

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at \texttt{https://github.com/guankaisi/FATE}.

cs.MM↗

arXiv2026

Emotion Understanding in Streaming Video with Trajectory-Aware Reliability

Video emotion understanding is commonly studied as an offline classification problem, where the complete video segment is available before prediction. Real-time interaction, however, requires emotion decisions from incomplete and evolving evidence. This paper studies streaming video emotion understanding as a reliability-aware decision process over evolving emotion beliefs. In this setting, a single confident prefix prediction can still be unreliable when the underlying belief trajectory is unstable or repeatedly switches across emotion classes. We propose TRACE, a trajectory-aware reliability framework that forms low-latency emotion beliefs from streaming audio prefixes, estimates reliability from confidence, entropy, stability, and class-switching patterns, and selectively invokes contextual belief reinterpretation with visual, textual, and neighboring-utterance evidence. TRACE keeps stable cases in the low-latency online pathway while allocating stronger multimodal reasoning to uncertain cases that remain ambiguous. Experiments on StreamMER, MELD, and MER2024 show that TRACE improves the accuracy-cost trade-off, retaining most full-context gains while reducing unnecessary contextual reasoning.

cs.MM↗

arXiv2026

Supporting Perspective Acquisition and Opinion Formation on Societal Issues Through AI-Generated Japanese Rap Battle Debates

Acquiring diverse perspectives and forming informed opinions on societal issues are essential for critical thinking and informed decision-making. Although observing debates between opposing viewpoints can promote perspective acquisition, conventional debates require substantial time and human resources, limiting their accessibility. This study investigates whether brief AI-generated rap battle-style debates can support perspective acquisition and opinion formation on societal issues. Rap battles are debate-style performances in which opposing viewpoints are expressed through rhymed verses, making them engaging while preserving the structure of argumentative dialogue. To conduct this investigation, we automatically generated rap battle-style debates using GPT-5.4-mini and presented them to participants with synthesized audio through a web-based viewing interface. We evaluated the educational effectiveness of this approach through a user study involving six participants who viewed 40 debates generated from topics on societal issues used in the All Japan Junior and Senior High School Debate Championships. The results showed that watching approximately 1.5 minute rap battle-style debates nearly doubled the number of opinions and discussion points identified by participants. Furthermore, participants reconsidered or changed their stances in 52.5% of the evaluated cases. These findings demonstrate that even brief AI-generated rap battle-style debates can effectively support perspective acquisition and opinion formation on societal issues.

cs.MM↗