Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.08996

UniMore: Fine-Grained Supervision for Unified Understanding and Generation

Haoyue Bai·Yinyi Luo·Yicheng He·Qingsong Wen·Sharon Li·Jindong Wang
View original source ↗Read source document ↗Export metadata ↓

Abstract

Unified multimodal models combine image understanding and generation within one system, creating an opportunity to improve both capabilities through paired image--text supervision. We examine the outputs of a pretrained unified model in both directions and observe a loss of specificity: generated images can contain the requested objects with incorrect attributes, while captions identify a scene without expressing its distinguishing details. These failures call for supervision that specifies both the visual content to realize and the descriptive differences to distinguish. We propose UniMore, a post-training method that derives these complementary learning tasks from the fine-grained content of image--text pairs. The generation route predicts grounded crops from their corresponding phrases, matching the scope of the condition and visual target while using the native visual objective. The understanding route combines positive-caption fitting with ranking against captions that omit or alter details, conditioned on the full image. The two routes are optimized independently, and their learned updates are combined into a single unified model without changing its inference procedure. Experiments show gains across multiple understanding and generation benchmarks.

Explore related subjects

cs.MMcs.CV

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Haoyue Bai, Yinyi Luo, Yicheng He, Qingsong Wen, Sharon Li, Jindong Wang. 2026-11-06. UniMore: Fine-Grained Supervision for Unified Understanding and Generation. https://arxiv.org/abs/2611.08996

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation

Multimodal reasoning systems increasingly assume that longer chain-of-thought (CoT) uniformly improves downstream grounding. We show that this assumption fails in referring audio-visual segmentation (Ref-AVS): simple queries can be harmed by excessive reasoning, creating an "overthinking trap" associated with phrase and localization drift, while genuinely ambiguous queries still benefit from multi-step reasoning. To systematically study this, we construct counterfactual reasoning-budget labels under a five-fold out-of-fold evaluation protocol. Our results recast adaptive Ref-AVS reasoning as a pre-decisional representation problem: before the first generated token, the generation-onset state contains linearly decodable information predictive of whether additional reasoning will improve segmentation. A linear probe predicts binary reasoning need with 70.1% accuracy, substantially outperforming text-only features and uncertainty-based proxy signals. As a router, the onset-state controller improves over downstream confidence routing and closely matches Short-then-Decide while using far fewer tokens. Proxy comparisons show that the onset representation contains predictive information not fully recovered by the evaluated hand-crafted features. Reusing this onset state for lightweight, single forward-pass routing reduces autoregressive token consumption by 60% while retaining about 96% of always-long performance. It further provides a cost-quality trade-off on $R^2$-AVSBench without target-label tuning, rather than an accuracy improvement over always-long. Error analysis links harmful long reasoning to phrase and localization drift, illustrating the benefit of bypassing unnecessary reasoning on simple queries. Ultimately, our results support efficient adaptive reasoning in Ref-AVS through pre-decisional internal representations.

cs.MM↗
arXiv2026

When does the head leave? Shared timing and a movement clock in 360-degree video viewing

A $360^\circ$ streaming system only needs high quality where the viewer faces, so knowing when a head will turn away matters as much as where it will go. Across four public head-tracking datasets, two used to fix and validate the method and two held out for testing, three patterns emerge on the held-out sets. First, viewers facing the same region turn away within a second of one another far more often than time-shifted traces allow. That estimate is about five times the pairwise threshold for a video timing one departure in ten, mapping to roughly one and a third to over twice that one-in-ten share. The excess co-departing fraction is 0.0422, with a 90% interval from 0.0279 to 0.0565. Second, time since the last large head movement improves held-out departure prediction beyond video and viewer effects. It adds 0.0793 nats per departure, a predictive dependence rather than proof of an internal timer. Third, on the eye-tracked test set, gaze crosses the region boundary after the head starts moving, lagging head onset by 0.30 s. Because headsets record continuous angles, a departure exists only after a rule marks when a settled hold ends once the head swings past a speed and angle threshold. Before touching the test sets, we picked one primary setting for the headline estimates above and tested 18 settings on synthetic traces with planted answers, requiring at least nine passing settings for robustness. Fifteen settings passed for the clock and all 15 valid settings confirm its gain; 8 passed for synchrony and all 8 valid settings agree, one short of the nine-setting bar; and 6 of 7 dual-validated settings agree on gaze lag. Synchrony validation drops from 17 or 18 passing settings under guessed noise to 8 under measured noise because zero-tolerance rules reject a small baseline offset, consistently positive at one dataset size, as datasets grow.

cs.MM↗
arXiv2026

TV-AudioRemover: Joint Text-Visual Guided Sound Removal with Multi-Task Hard-Mixture Curriculum

Visual object removal can eliminate a target from video frames, yet its acoustic trace persists in the soundtrack, causing obvious audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically driven by text and therefore rely on limited single-modal control, which is less effective than multimodal guidance that provides stronger semantic grounding and temporal synchronization cues. In this paper, we present Text-Visual Guided Sound Removal (TV-AudioRemover), a target sound removal framework that leverages the visually edited video together with a natural-language instruction to suppress the sound associated with the removed visual object from the original audio mixture. To acquire high-quality training data, we devise a pipeline to construct a million-scale dataset of single-object audio-visual aligned samples, from which we synthesize mixture-target pairs customized for model training. To effectively leverage visual context and follow instruction intent, we augment the model architecture with task tokens, generalizable instruction modeling, and modality-specific global guidance. We further adopt multi-task training to strengthen task-role comprehension, and employ a hard-mixture curriculum that leverages semantically similar acoustic mixtures during fine-tuning to enhance fine-grained source discrimination. To support evaluation, we present AV-Remove-Bench, a comprehensive audio-visual object removal benchmark, along with dedicated objective metrics and an MLLM-based evaluation protocol. Experiments demonstrate that our method achieves state-of-the-art performance on both subjective and objective metrics. Project page: https://yjx-research.github.io/TV-AudioRemover/.

cs.MM↗