arXiv subjects

Xinyi Tu

Publications and source records attributed to Xinyi Tu.

4 recordsLinked to original sources

arXiv2026

Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning

LLMs struggle with Semantic Inertia: the inability to inhibit pre-trained priors (e.g., "Lava is Dangerous") when dynamic, in-context rules contradict them. We probe this phenomenon using Baba Is You, where physical laws are mutable text rules, enabling precise evaluation of models' ability to override learned priors when rules change. We quantatively observe that larger models can exhibit inverse scaling: they perform worse than smaller models when natural language reasoning requires suppressing pre-trained associations (e.g., accepting "Lava is Safe"). Our analysis attributes this to natural language encoding, which entangles descriptive semantics and logical rules, leading to persistent hallucinations of familiar physics despite explicit contradictory rules. Here we show that representing dynamics as executable code, rather than descriptive text, reverses this trend and enables effective prior inhibition. We introduce Code-Grounded Vistas (LCV), which fine-tunes models on counterfactual pairs and identifies states with contradictory rules, thereby forcing attention to logical constraints rather than visual semantics. This training-time approach outperforms expensive inference-time search methods in both efficiency and accuracy. Our results demonstrate that representation fundamentally determines whether scaling improves or impairs contextual reasoning. This challenges the assumption that larger models are universally better, with implications for domains that require dynamic overriding of learned priors.

cs.CL↗

arXiv2024

Towards Diverse and Efficient Audio Captioning via Diffusion Models

We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive diffusion model tailored for diverse and efficient audio captioning. Although existing captioning models relying on language backbones have achieved remarkable success in various captioning tasks, their insufficient performance in terms of generation speed and diversity impede progress in audio understanding and multimedia applications. Our diffusion-based framework offers unique advantages stemming from its inherent stochasticity and holistic context modeling in captioning. Through rigorous evaluation, we demonstrate that DAC not only achieves SOTA performance levels compared to existing benchmarks in the caption quality, but also significantly outperforms them in terms of generation speed and diversity. The success of DAC illustrates that text generation can also be seamlessly integrated with audio and visual generation tasks using a diffusion backbone, paving the way for a unified, audio-related generative model across different modalities.

cs.CL↗

arXiv2024

Video-to-Audio Generation with Hidden Alignment

Generating semantically and temporally aligned audio content in accordance with video input has become a focal point for researchers, particularly following the remarkable breakthrough in text-to-video generation. In this work, we aim to offer insights into the video-to-audio generation paradigm, focusing on three crucial aspects: vision encoders, auxiliary embeddings, and data augmentation techniques. Beginning with a foundational model built on a simple yet surprisingly effective intuition, we explore various vision encoders and auxiliary embeddings through ablation studies. Employing a comprehensive evaluation pipeline that emphasizes generation quality and video-audio synchronization alignment, we demonstrate that our model exhibits state-of-the-art video-to-audio generation capabilities. Furthermore, we provide critical insights into the impact of different data augmentation methods on enhancing the generation framework's overall capacity. We showcase possibilities to advance the challenge of generating synchronized audio from semantic and temporal perspectives. We hope these insights will serve as a stepping stone toward developing more realistic and accurate audio-visual generation models.

cs.SD↗

arXiv2023

Towards enabling reliable immersive teleoperation through Digital Twin: A UAV command and control use case

This paper addresses the challenging problem of enabling reliable immersive teleoperation in scenarios where an Unmanned Aerial Vehicle (UAV) is remotely controlled by an operator via a cellular network. Such scenarios can be quite critical particularly when the UAV lacks advanced equipment (e.g., Lidar-based auto stop) or when the network is subject to some performance constraints (e.g., delay). To tackle these challenges, we propose a novel architecture leveraging Digital Twin (DT) technology to create a virtual representation of the physical environment. This virtual environment accurately mirrors the physical world, accounting for 3D surroundings, weather constraints, and network limitations. To enhance teleoperation, the UAV in the virtual environment is equipped with advanced features that maybe absent in the real UAV. Furthermore, the proposed architecture introduces an intelligent logic that utilizes information from both virtual and physical environments to approve, deny, or correct actions initiated by the UAV operator. This anticipatory approach helps to mitigate potential risks. Through a series of field trials, we demonstrate the effectiveness of the proposed architecture in significantly improving the reliability of UAV teleoperation.

cs.HC↗