Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.29587

Guess and Send: World-Model-Guided Semantic Packet Transmission for Robust Video Perception

Yan Gong·Zheng Chu·Wei Koong Chai·Hari Mohan Pandey·Jiankang Zhang·Jingrui Hou·Miao Zhang·Yuanjian Li·Pei Xiao
View original source ↗Read source document ↗Export metadata ↓

Abstract

Reliable video perception over bandwidth-constrained wireless links requires transmitting information that the receiver cannot already recover from its evolving scene representation. This paper introduces WorldGT, a world-model-guided semantic packet transmission framework that prioritizes information according to its contribution to the receiver's perception. A shared feature-space world model predicts the receiver's scene state through affine motion compensation and learned residual displacement, while sender-side verification identifies reusable predictions. An acknowledgment-synchronized receiver twin estimates task-impact scores through two counterfactual evaluations, guiding packet selection and unequal error protection over LDPC-coded wireless channels. A compact 3-bit feature representation increases spatial coverage under stringent bandwidth constraints, while predictive fallback sustains perception during packet outages. Experiments on KITTI and VisDrone under Rayleigh fading and AWGN demonstrate substantial gains in low-SNR conditions. On KITTI at a 4.13% reference bandwidth budget, WorldGT achieves 47.5% J&F at 0 dB Rayleigh SNR, outperforming H.264 by 11.3 percentage points. Controlled ablations show that world-model prediction contributes up to 8.7 points under Rayleigh fading. Object-detection experiments further demonstrate the transferability of receiver-relative predictive transmission across visual perception tasks. These results establish world-model-guided, receiver-state-aware transmission as an effective approach to robust video perception under severe communication constraints.

Explore related subjects

cs.MM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yan Gong, Zheng Chu, Wei Koong Chai, Hari Mohan Pandey, Jiankang Zhang, Jingrui Hou, Miao Zhang, Yuanjian Li, Pei Xiao. 2026-10-21. Guess and Send: World-Model-Guided Semantic Packet Transmission for Robust Video Perception. https://arxiv.org/abs/2610.29587

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A Survey of Generative Categories and Techniques in Multimodal Generative Models

Multimodal generative models (MGMs) have rapidly expanded beyond text generation to encompass images, music, video, human motion, and 3D objects. This survey categorises six primary text-conditioned generative modalities and examines how four techniques, namely self-supervised learning, mixture of experts, reinforcement learning from human feedback, and chain-of-thought prompting, are adapted across them, distinguishing shared mechanisms from experimentally supported benefits. We organise evaluation around faithfulness, compositionality, and robustness, and interpret evidence within each study's protocol. Selected within-study comparisons reported in the reviewed literature support four findings: preference gains need not improve text relevance, and reward optimisation can reduce within-prompt diversity; intermediate plans require tests of their realisation in final outputs; expert granularity changes the mechanism and its storage, computation, and runtime costs; and gains against an optimisation score need confirmation outside that scoring path. We further analyse trustworthiness risks, including bias, privacy leakage, and misuse of generated media, and examine the evidence for mitigation strategies.

cs.MM↗
arXiv2026

MuSP-Bench: Advanced Multimodal Benchmarking of Music Understanding across Score and Performance

Musicians commonly communicate musical intent through scores and performances. To investigate whether models can reason across both modalities, we introduce MuSP-Bench, a human-authored benchmark of 490 questions assessing understanding across Musical Scores and Performances, both individually and in combination. The benchmark spans interpretive and long-horizon reasoning across classical piano and orchestral works, represented through ABC notation, score images, MIDI, and audio. Our evaluation of frontier multimodal large language models reveals substantial difficulties in understanding musical scores and even greater challenges in reasoning about audio. The benchmark is available at https://musp.vaclis.net/.

cs.MM↗
arXiv2026

NESA: Non-stationary Evidence-aware Semantic Alignment for Robust ECG--Text Representation Pretraining

ECG-text pretraining offers transferable electrocardiogram (ECG) representations for zero-shot diagnosis, but its supervision is inherently asymmetric: clinical reports contain sparse diagnostic phrases, while ECG signals provide rich and redundant evidence across leads and time. As a result, cross-modal learning can collapse into coarse prototype matching, relying on a few highly aligned ECG tokens while overlooking complementary evidence. Existing methods enrich text supervision or perturb ECG inputs, yet rarely model how cross-modal correspondence should adapt as the available ECG evidence changes. We propose NESA, a Non-stationary Evidence-aware Semantic Alignment framework that replaces static prototype matching with an evolving alignment trajectory. NESA estimates each ECG token's contribution to the current cross-modal alignment and progressively removes dominant tokens in latent space to expose residual evidence. To preserve semantic correspondence under evidence removal, it constructs adaptive alignment states from text-derived clinical semantics, full-view ECG context, and residual ECG evidence. This trajectory guides the ECG encoder to capture complementary diagnostic cues rather than rely on a small set of tokens. Experiments with MIMIC-IV-ECG pretraining and three downstream benchmarks show improvements in zero-shot diagnosis and label-efficient transfer. Dominant-token deletion analysis further indicates that NESA preserves cross-modal alignment after highly aligned ECG evidence is removed, supporting more distributed ECG-text grounding.

cs.MM↗