arXiv subjects

Yuqing Sun

Publications and source records attributed to Yuqing Sun.

10 recordsLinked to original sources

arXiv2026

Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editing

We present Omni-Video 2, a scalable and computationally efficient model that connects pretrained multimodal large-language models (MLLMs) with video diffusion models for unified video generation and editing. Our key idea is to exploit the understanding and reasoning capabilities of MLLMs to produce explicit target captions to interpret user instructions. In this way, the rich contextual representations from the understanding model are directly used to guide the generative process, thereby improving performance on complex and compositional editing. Moreover, a lightweight adapter is developed to inject multimodal conditional tokens into pretrained text-to-video diffusion models, allowing maximum reuse of their powerful generative priors in a parameter-efficient manner. Benefiting from these designs, we scale up Omni-Video 2 to a 14B video diffusion model on meticulously curated training data with quality, supporting high quality text-to-video generation and various video editing tasks such as object removal, addition, background change, complex motion editing, \emph{etc.} We evaluate the performance of Omni-Video 2 on the FiVE benchmark for fine-grained video editing and the VBench benchmark for text-to-video generation. The results demonstrate its superior ability to follow complex compositional instructions in video editing, while also achieving competitive or superior quality in video generation tasks.

cs.CV↗

arXiv2025

MSF-SER: Enriching Acoustic Modeling with Multi-Granularity Semantics for Speech Emotion Recognition

Continuous dimensional speech emotion recognition captures affective variation along valence, arousal, and dominance, providing finer-grained representations than categorical approaches. Yet most multimodal methods rely solely on global transcripts, leading to two limitations: (1) all words are treated equally, overlooking that emphasis on different parts of a sentence can shift emotional meaning; (2) only surface lexical content is represented, lacking higher-level interpretive cues. To overcome these issues, we propose MSF-SER (Multi-granularity Semantic Fusion for Speech Emotion Recognition), which augments acoustic features with three complementary levels of textual semantics--Local Emphasized Semantics (LES), Global Semantics (GS), and Extended Semantics (ES). These are integrated via an intra-modal gated fusion and a cross-modal FiLM-modulated lightweight Mixture-of-Experts (FM-MOE). Experiments on MSP-Podcast and IEMOCAP show that MSF-SER consistently improves dimensional prediction, demonstrating the effectiveness of enriched semantic fusion for SER.

cs.SD↗

arXiv2025

EMORL-TTS: Reinforcement Learning for Fine-Grained Emotion Control in LLM-based TTS

Recent LLM-based TTS systems achieve strong quality and zero-shot ability, but lack fine-grained emotional control due to their reliance on discrete speech tokens. Existing approaches either limit emotions to categorical labels or cannot generalize to LLM-based architectures. We propose EMORL-TTS (Fine-grained Emotion-controllable TTS with Reinforcement Learning), a framework that unifies global intensity control in the VAD space with local emphasis regulation. Our method combines supervised fine-tuning with reinforcement learning guided by task-specific rewards for emotion category, intensity, and emphasis. Moreover, we further investigate how emphasis placement modulates fine-grained emotion intensity. Experiments show that EMORL-TTS improves emotion accuracy, intensity differentiation, and emphasis clarity, while preserving synthesis quality comparable to strong LLM-based baselines.

cs.SD↗

arXiv2025

Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations

Emotion Recognition in Conversations (ERC) is hard because discriminative evidence is sparse, localized, and often asynchronous across modalities. We center ERC on emotion hotspots and present a unified model that detects per-utterance hotspots in text, audio, and video, fuses them with global features via Hotspot-Gated Fusion, and aligns modalities using a routed Mixture-of-Aligners; a cross-modal graph encodes conversational structure. This design focuses modeling on salient spans, mitigates misalignment, and preserves context. Experiments on standard ERC benchmarks show consistent gains over strong baselines, with ablations confirming the contributions of HGF and MoA. Our results point to a hotspot-centric view that can inform future multimodal learning, offering a new perspective on modality fusion in ERC.

cs.CL↗

arXiv2025

SmartSwap: Swap-Based Memory Optimization for LLM Training under Varying Operator Sequences

The increasing size of large language models (LLMs) has led to a surge in memory requirements during training, often exceeding the capacity of high-bandwidth memory (HBM). Swap-based memory optimization incurs neither accuracy loss nor additional end-to-end overhead when effectively overlapped, thus being an attractive solution. However, existing swap methods assume consistent operator sequences, which is impractical in Eager Mode, where operator sequences can vary during change. We propose Chameleon, which redesigns the end-to-end process of swap-based memory optimization and is the first work to consider varying operator sequences in Eager Mode. Chameleon (i) introduces a lightweight online profiler to enable continuous profiling for monitoring operator sequences, (ii) generates effective swap policies with limited operator information, and (iii) optimizes the policy execution module for accurate policy application and better performance. Experimental results demonstrate that Chameleon reduces profiling overhead by 84.25%, enables training models up to 4x larger than hardware memory while adapting to changes in operator sequences, improves performance by up to 38.94% compared to recomputation or high-degree parallelism.

cs.DC↗

arXiv2025

Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Vision

Chain-of-Thought (CoT) reasoning has been widely adopted to enhance Large Language Models (LLMs) by decomposing complex tasks into simpler, sequential subtasks. However, extending CoT to vision-language reasoning tasks remains challenging, as it often requires interpreting transitions of visual states to support reasoning. Existing methods often struggle with this due to limited capacity of modeling visual state transitions or incoherent visual trajectories caused by fragmented architectures. To overcome these limitations, we propose Uni-CoT, a Unified Chain-of-Thought framework that enables coherent and grounded multimodal reasoning within a single unified model. The key idea is to leverage a model capable of both image understanding and generation to reason over visual content and model evolving visual states. However, empowering a unified model to achieve that is non-trivial, given the high computational cost and the burden of training. To address this, Uni-CoT introduces a novel two-level reasoning paradigm: A Macro-Level CoT for high-level task planning and A Micro-Level CoT for subtask execution. This design significantly reduces the computational overhead. Furthermore, we introduce a structured training paradigm that combines interleaved image-text supervision for macro-level CoT with multi-task objectives for micro-level CoT. Together, these innovations allow Uni-CoT to perform scalable and coherent multi-modal reasoning. Furthermore, thanks to our design, all experiments can be efficiently completed using only 8 A100 GPUs with 80GB VRAM each. Experimental results on reasoning-driven image generation benchmark (WISE) and editing benchmarks (RISE and KRIS) indicates that Uni-CoT demonstrates SOTA performance and strong generalization, establishing Uni-CoT as a promising solution for multi-modal reasoning. Project Page and Code: https://sais-fuxi.github.io/projects/uni-cot/

cs.CV↗

arXiv2025

SRLAgent: Enhancing Self-Regulated Learning Skills through Gamification and LLM Assistance

Self-regulated learning (SRL) is crucial for college students navigating increased academic demands and independence. Insufficient SRL skills can lead to disorganized study habits, low motivation, and poor time management, undermining learners ability to thrive in challenging environments. Through a formative study involving 59 college students, we identified key challenges students face in developing SRL skills, including difficulties with goal-setting, time management, and reflective learning. To address these challenges, we introduce SRLAgent, an LLM-assisted system that fosters SRL skills through gamification and adaptive support from large language models (LLMs). Grounded in Zimmermans three-phase SRL framework, SRLAgent enables students to engage in goal-setting, strategy execution, and self-reflection within an interactive game-based environment. The system offers real-time feedback and scaffolding powered by LLMs to support students independent study efforts. We evaluated SRLAgent using a between-subjects design, comparing it to a baseline system (SRL without Agent features) and a traditional multimedia learning condition. Results showed significant improvements in SRL skills within the SRLAgent group (p < .001, Cohens d = 0.234) and higher engagement compared to the baselines. This work highlights the value of embedding SRL scaffolding and real-time AI support within gamified environments, offering design implications for educational technologies that aim to promote deeper learning and metacognitive skill development.

cs.HC↗

arXiv2006

Deterministic and Unambiguous Dense Coding

Optimal dense coding using a partially-entangled pure state of Schmidt rank $\bar D$ and a noiseless quantum channel of dimension $D$ is studied both in the deterministic case where at most $L_d$ messages can be transmitted with perfect fidelity, and in the unambiguous case where when the protocol succeeds (probability $τ_x$) Bob knows for sure that Alice sent message $x$, and when it fails (probability $1-τ_x$) he knows it has failed. Alice is allowed any single-shot (one use) encoding procedure, and Bob any single-shot measurement. For $\bar D\leq D$ a bound is obtained for $L_d$ in terms of the largest Schmidt coefficient of the entangled state, and is compared with published results by Mozes et al. For $\bar D > D$ it is shown that $L_d$ is strictly less than $D^2$ unless $\bar D$ is an integer multiple of $D$, in which case uniform (maximal) entanglement is not needed to achieve the optimal protocol. The unambiguous case is studied for $\bar D \leq D$, assuming $τ_x>0$ for a set of $\bar D D$ messages, and a bound is obtained for the average $\lgl1/τ\rgl$. A bound on the average $\lglτ\rgl$ requires an additional assumption of encoding by isometries (unitaries when $\bar D=D$) that are orthogonal for different messages. Both bounds are saturated when $τ_x$ is a constant independent of $x$, by a protocol based on one-shot entanglement concentration. For $\bar D > D$ it is shown that (at least) $D^2$ messages can be sent unambiguously. Whether unitary (isometric) encoding suffices for optimal protocols remains a major unanswered question, both for our work and for previous studies of dense coding using partially-entangled states, including noisy (mixed) states.

quant-ph↗

arXiv2002

Optimal unambiguous discrimination between subsets of non-orthogonal quantum states

It is known that unambiguous discrimination among non-orthogonal but linearly independent quantum states is possible with a certain probability of success. Here, we consider a variant of that problem. Instead of discriminating among all of the different states, we shall only discriminate between two subsets of them. In particular, for the case of three non-orthogonal states, we show that the optimal strategy to distinguish between a set containing one of the states from the set containing the other two has a higher success rate than if we wish to discriminate among all three states. Somewhat surprisingly, for unambiguous discrimination the subsets need not be linearly independent. A fully analytical solution is presented, and we also show how to construct generalized interferometers (multiports) that provide an optical implementation of the optimal strategy.

quant-ph↗

arXiv2000

Optimum unambiguous discrimination among linearly-independent non-orthogonal quantum states and its optical realization

The problem of unambiguously distinguishing among nonorthogonal but linearly independent quantum states can be solved by mapping the set of nonorthogonal quantum states onto a set of orthogonal ones, which can then be distinguished without error. Such nonunitary transformations can be performed conditionally on quantum systems; a unitary transformation is carried out on a larger system of which the system of interest is a subsytem, a measurement is performed, and if the proper result is obtained, the desired nonunitary transformation will have been performed on the subsystem. We show how to construct generalized interferometers (multiports), which when combined with measurements on some of the output ports, implement nonunitary transformations of this type. The input states are single-photon states in which the photon is divided among several modes. A number of explicit examples of distinguishing among three nonorthogonal states are discussed, and the networks that optimally distinguish among these states are presented.

quant-ph↗