arXiv subjects

Xiaolong Wang

Publications and source records attributed to Xiaolong Wang.

At least 109 records · Page 6Linked to original sources

arXiv2025

One-Minute Video Generation with Test-Time Training

Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit

cs.CV↗

arXiv2025

Visual Acoustic Fields

Objects produce different sounds when hit, and humans can intuitively infer how an object might sound based on its appearance and material properties. Inspired by this intuition, we propose Visual Acoustic Fields, a framework that bridges hitting sounds and visual signals within a 3D space using 3D Gaussian Splatting (3DGS). Our approach features two key modules: sound generation and sound localization. The sound generation module leverages a conditional diffusion model, which takes multiscale features rendered from a feature-augmented 3DGS to generate realistic hitting sounds. Meanwhile, the sound localization module enables querying the 3D scene, represented by the feature-augmented 3DGS, to localize hitting positions based on the sound sources. To support this framework, we introduce a novel pipeline for collecting scene-level visual-sound sample pairs, achieving alignment between captured images, impact locations, and corresponding sounds. To the best of our knowledge, this is the first dataset to connect visual and acoustic signals in a 3D context. Extensive experiments on our dataset demonstrate the effectiveness of Visual Acoustic Fields in generating plausible impact sounds and accurately localizing impact sources. Our project page is at https://yuelei0428.github.io/projects/Visual-Acoustic-Fields/.

cs.CV↗

arXiv2025

M3: 3D-Spatial MultiModal Memory

We present 3D Spatial MultiModal Memory (M3), a multimodal memory system designed to retain information about medium-sized static scenes through video sources for visual perception. By integrating 3D Gaussian Splatting techniques with foundation models, M3 builds a multimodal memory capable of rendering feature representations across granularities, encompassing a wide range of knowledge. In our exploration, we identify two key challenges in previous works on feature splatting: (1) computational constraints in storing high-dimensional features for each Gaussian primitive, and (2) misalignment or information loss between distilled features and foundation model features. To address these challenges, we propose M3 with key components of principal scene components and Gaussian memory attention, enabling efficient training and inference. To validate M3, we conduct comprehensive quantitative evaluations of feature similarity and downstream tasks, as well as qualitative visualizations to highlight the pixel trace of Gaussian memory attention. Our approach encompasses a diverse range of foundation models, including vision-language models (VLMs), perception models, and large multimodal and language models (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy M3's feature field in indoor scenes on a quadruped robot. Notably, we claim that M3 is the first work to address the core compression challenges in 3D feature distillation.

cs.CV↗

arXiv2025

Normative implications

We continue to develop a research line initiated in \cite{wollic22}, studying I/O logic from an algebraic approach based on subordination algebras. We introduce the classes of slanted (co-)Heyting algebras as equivalent presentations of distributive lattices with subordination relations. Interpreting subordination relations as the algebraic counterparts of input/output relations on formulas yields (slanted) modal operations with interesting deontic interpretations. We study the theory of slanted and co-slanted Heyting algebras, develop algorithmic correspondence and inverse correspondence, and present some deontically meaningful axiomatic extensions and examples.

math.LO↗

arXiv2025

ExBody2: Advanced Expressive Humanoid Whole-Body Control

This paper tackles the challenge of enabling real-world humanoid robots to perform expressive and dynamic whole-body motions while maintaining overall stability and robustness. We propose Advanced Expressive Whole-Body Control (Exbody2), a method for producing whole-body tracking controllers that are trained on both human motion capture and simulated data and then transferred to the real world. We introduce a technique for decoupling the velocity tracking of the entire body from tracking body landmarks. We use a teacher policy to produce intermediate data that better conforms to the robot's kinematics and to automatically filter away infeasible whole-body motions. This two-step approach enabled us to produce a student policy that can be deployed on the robot that can walk, crouch, and dance. We also provide insight into the trade-off between versatility and the tracking performance on specific motions. We observed significant improvement of tracking performance after fine-tuning on a small amount of data, at the expense of the others.

cs.RO↗

arXiv2025

The pseudo-analytical density solution to parameterized Fokker-Planck equations via deep learning

Efficiently solving the Fokker-Planck equation (FPE) is crucial for understanding the probabilistic evolution of stochastic particles in dynamical systems, however, analytical solutions or density functions are only attainable in specific cases. To speed up the solving process of parameterized FPEs with several system parameters, we introduce a deep learning-based method to obtain the pseudo-analytical density (PAD). Unlike previous numerical methodologies that necessitate solving the FPE separately for each set of system parameters, the PAD simultaneously addresses all the FPEs within a predefined continuous range of system parameters during a single training phase. The approach utilizes a Gaussian mixture distribution (GMD) to represent the stationary probability density, the solution to the FPE. By leveraging a deep residual network, each system parameter configuration is mapped to the parameters of the GMD, ensuring that the weights, means, and variances of the Gaussian components adaptively align with the corresponding true density functions. A grid-free algorithm is further developed to effectively train the residual network, resulting in a feasible PAD obeying necessary normalization and boundary conditions. Extensive numerical studies validate the accuracy and efficiency of our method, promising significant acceleration in the response analysis of multi-parameter, multi-dimensional stochastic nonlinear systems.

physics.comp-ph↗

arXiv2025

Data taking strategy for $ψ(3770)$ and $Υ(4S)$ branching fraction measurements at $e^+e^-$ colliders

The $ψ(3770)$ and $Υ(4S)$ states predominantly decay into open-flavor meson pairs, whereas the decays of $ψ(3770) \to \mbox{\text{non}-}D\bar{D}$ and $Υ(4S) \to \mbox{\text{non}-}B\bar{B}$ are rare but crucial for elucidating the inner structure and decay dynamics of heavy quarkonium states. To achieve precise branching fraction measurements for $ψ(3770) \to \mbox{\text{non}-}D\bar{D}$ and $Υ(4S) \to \mbox{\text{non}-}B\bar{B}$ decays at the high luminosity $e^+e^-$ annihilation experiments, we employed Monte Carlo simulations and Fisher information to evaluate various data taking scenarios, ultimately determining the optimal scheme. The consistent results of both methodologies indicate that the optimal energy points for studies of $ψ(3770) \to \mbox{\text{non}-}D\bar{D}$ decays are $3.769~\gev$ and $3.781~\gev$, whereas those for $Υ(4S) \to \mbox{\text{non}-}B\bar{B}$ decays are $10.574~\gev$ and $10.585~\gev$. In addition, we studied the dependence of the precision in branching fraction measurements on the integrated luminosity, with the branching fractions spanning several orders of magnitude.

hep-ex↗

arXiv2025

Mobile-TeleVision: Predictive Motion Priors for Humanoid Whole-Body Control

Humanoid robots require both robust lower-body locomotion and precise upper-body manipulation. While recent Reinforcement Learning (RL) approaches provide whole-body loco-manipulation policies, they lack precise manipulation with high DoF arms. In this paper, we propose decoupling upper-body control from locomotion, using inverse kinematics (IK) and motion retargeting for precise manipulation, while RL focuses on robust lower-body locomotion. We introduce PMP (Predictive Motion Priors), trained with Conditional Variational Autoencoder (CVAE) to effectively represent upper-body motions. The locomotion policy is trained conditioned on this upper-body motion representation, ensuring that the system remains robust with both manipulation and locomotion. We show that CVAE features are crucial for stability and robustness, and significantly outperforms RL-based whole-body control in precise manipulation. With precise upper-body motion and robust lower-body locomotion control, operators can remotely control the humanoid to walk around and explore different environments, while performing diverse manipulation tasks.

cs.RO↗

arXiv2025

WildLMa: Long Horizon Loco-Manipulation in the Wild

'In-the-wild' mobile manipulation aims to deploy robots in diverse real-world environments, which requires the robot to (1) have skills that generalize across object configurations; (2) be capable of long-horizon task execution in diverse environments; and (3) perform complex manipulation beyond pick-and-place. Quadruped robots with manipulators hold promise for extending the workspace and enabling robust locomotion, but existing results do not investigate such a capability. This paper proposes WildLMa with three components to address these issues: (1) adaptation of learned low-level controller for VR-enabled whole-body teleoperation and traversability; (2) WildLMa-Skill -- a library of generalizable visuomotor skills acquired via imitation learning or heuristics and (3) WildLMa-Planner -- an interface of learned skills that allow LLM planners to coordinate skills for long-horizon tasks. We demonstrate the importance of high-quality training data by achieving higher grasping success rate over existing RL baselines using only tens of demonstrations. WildLMa exploits CLIP for language-conditioned imitation learning that empirically generalizes to objects unseen in training demonstrations. Besides extensive quantitative evaluation, we qualitatively demonstrate practical robot applications, such as cleaning up trash in university hallways or outdoor terrains, operating articulated objects, and rearranging items on a bookshelf.

cs.RO↗

arXiv2025

HOVER: Versatile Neural Whole-Body Controller for Humanoid Robots

Humanoid whole-body control requires adapting to diverse tasks such as navigation, loco-manipulation, and tabletop manipulation, each demanding a different mode of control. For example, navigation relies on root velocity tracking, while tabletop manipulation prioritizes upper-body joint angle tracking. Existing approaches typically train individual policies tailored to a specific command space, limiting their transferability across modes. We present the key insight that full-body kinematic motion imitation can serve as a common abstraction for all these tasks and provide general-purpose motor skills for learning multiple modes of whole-body control. Building on this, we propose HOVER (Humanoid Versatile Controller), a multi-mode policy distillation framework that consolidates diverse control modes into a unified policy. HOVER enables seamless transitions between control modes while preserving the distinct advantages of each, offering a robust and scalable solution for humanoid control across a wide range of modes. By eliminating the need for policy retraining for each control mode, our approach improves efficiency and flexibility for future humanoid applications.

cs.RO↗

arXiv2025

Dynamic Gaussians Mesh: Consistent Mesh Reconstruction from Dynamic Scenes

Modern 3D engines and graphics pipelines require mesh as a memory-efficient representation, which allows efficient rendering, geometry processing, texture editing, and many other downstream operations. However, it is still highly difficult to obtain high-quality mesh in terms of detailed structure and time consistency from dynamic observations. To this end, we introduce Dynamic Gaussians Mesh (DG-Mesh), a framework to reconstruct a high-fidelity and time-consistent mesh from dynamic input. Our work leverages the recent advancement in 3D Gaussian Splatting to construct the mesh sequence with temporal consistency from dynamic observations. Building on top of this representation, DG-Mesh recovers high-quality meshes from the Gaussian points and can track the mesh vertices over time, which enables applications such as texture editing on dynamic objects. We introduce the Gaussian-Mesh Anchoring, which encourages evenly distributed Gaussians, resulting better mesh reconstruction through mesh-guided densification and pruning on the deformed Gaussians. By applying cycle-consistent deformation between the canonical and the deformed space, we can project the anchored Gaussian back to the canonical space and optimize Gaussians across all time frames. During the evaluation on different datasets, DG-Mesh provides significantly better mesh reconstruction and rendering than baselines. Project page: https://www.liuisabella.com/DG-Mesh

cs.CV↗

arXiv2025

WorldModelBench: Judging Video Generation Models As World Models

Video generation models have rapidly progressed, positioning themselves as video world models capable of supporting decision-making applications like robotics and autonomous driving. However, current benchmarks fail to rigorously evaluate these claims, focusing only on general video quality, ignoring important factors to world models such as physics adherence. To bridge this gap, we propose WorldModelBench, a benchmark designed to evaluate the world modeling capabilities of video generation models in application-driven domains. WorldModelBench offers two key advantages: (1) Against to nuanced world modeling violations: By incorporating instruction-following and physics-adherence dimensions, WorldModelBench detects subtle violations, such as irregular changes in object size that breach the mass conservation law - issues overlooked by prior benchmarks. (2) Aligned with large-scale human preferences: We crowd-source 67K human labels to accurately measure 14 frontier models. Using our high-quality human labels, we further fine-tune an accurate judger to automate the evaluation procedure, achieving 8.6% higher average accuracy in predicting world modeling violations than GPT-4o with 2B parameters. In addition, we demonstrate that training to align human annotations by maximizing the rewards from the judger noticeably improve the world modeling capability. The website is available at https://worldmodelbench-team.github.io.

cs.CV↗

arXiv2025

Scintillation and Timing Performance of a 3at% Yttrium-Doped Barium Fluoride Crystal

We report the scintillation and timing performance of a new developed 200 * 20 mm * 20 mm large size barium fluoride crystal doped with 3at% yttrium (BaF2:Y) to enhance the application for high time resolution. This doping effectively suppresses the slow scintillation component while maintaining most of the fast component, as confirmed by X-ray excited luminescence measurements. The BaF2:Y crystal demonstrated a transmittance of near 90% in the visible spectrum and a light response uniformity parameter of delta = (-2.74 +- 1.15)% when coupled with the tail end. The actual yttrium content varied from 2.1at% near the seed end to 3.7at% at the tail end. The assembled large BaF2:Y detector with silicon photomultipliers exhibited a time resolution of (82.2 +- 2.6) ps using constant fraction discrimination method in a cosmic ray test and (140.1 +- 3.8) ps using a low fixed threshold method in a beam test at Shanghai Synchrotron Radiation Facility with an 1.35 GeV electron beam. These results indicate the significant potential of BaF2:Y crystal for various applications, such as detectors for particle physics and nuclear physics.

physics.ins-det↗

arXiv2025

NaVILA: Legged Robot Vision-Language-Action Model for Navigation

This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments. We show more results at https://navila-bot.github.io/

cs.RO↗

arXiv2025

Line shape of the $Jψ\to γη_{c}$ decay

An accurate description of the photon spectrum line shape is essential for extracting resonance parameters of the $η_c$ meson through the radiative transition $J/ψ\to γη_{c}$. However, a persistent challenge remains in the form of a divergent tail at high photon energies, arising from the $E_γ^3$ factor in theoretical calculations. Various damping functions have been proposed to mitigate this effect in practical experiments, but their empirical nature lacks a rigorous theoretical basis. In this study, we introduce two key considerations: incorporating full-order contributions of the Bessel function in the overlap integral of charmonium wave functions and the phase space factor neglected in previous experimental studies. By accounting for these factors, we demonstrate a more rational and effective damping function of the divergent tail associated with the $E_γ^3$ term. We present the implications of these findings on experimental measurements and provide further insights through toy Monte Carlo simulations.

hep-ph↗

arXiv2025

LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Models

The rapid advancements in vision-language models (VLMs), such as CLIP, have intensified the need to address distribution shifts between training and testing datasets. Although prior Test-Time Training (TTT) techniques for VLMs have demonstrated robust performance, they predominantly rely on tuning text prompts, a process that demands substantial computational resources and is heavily dependent on entropy-based loss. In this paper, we propose LoRA-TTT, a novel TTT method that leverages Low-Rank Adaptation (LoRA), applied exclusively to the image encoder of VLMs. By introducing LoRA and updating only its parameters during test time, our method offers a simple yet effective TTT approach, retaining the model's initial generalization capability while achieving substantial performance gains with minimal memory and runtime overhead. Additionally, we introduce a highly efficient reconstruction loss tailored for TTT. Our method can adapt to diverse domains by combining these two losses, without increasing memory consumption or runtime. Extensive experiments on two benchmarks, covering 15 datasets, demonstrate that our method improves the zero-shot top-1 accuracy of CLIP-ViT-B/16 by an average of 5.79% on the OOD benchmark and 1.36% on the fine-grained benchmark, efficiently surpassing test-time prompt tuning, without relying on any external models or cache.

cs.CV↗

arXiv2025

Diffusion Autoencoders are Scalable Image Tokenizers

Tokenizing images into compact visual representations is a key step in learning efficient and high-quality image generative models. We present a simple diffusion tokenizer (DiTo) that learns compact visual representations for image generation models. Our key insight is that a single learning objective, diffusion L2 loss, can be used for training scalable image tokenizers. Since diffusion is already widely used for image generation, our insight greatly simplifies training such tokenizers. In contrast, current state-of-the-art tokenizers rely on an empirically found combination of heuristics and losses, thus requiring a complex training recipe that relies on non-trivially balancing different losses and pretrained supervised models. We show design decisions, along with theoretical grounding, that enable us to scale DiTo for learning competitive image representations. Our results show that DiTo is a simpler, scalable, and self-supervised alternative to the current state-of-the-art image tokenizer which is supervised. DiTo achieves competitive or better quality than state-of-the-art in image reconstruction and downstream image generation tasks.

cs.CV↗

arXiv2025

Integrating LMM Planners and 3D Skill Policies for Generalizable Manipulation

The recent advancements in visual reasoning capabilities of large multimodal models (LMMs) and the semantic enrichment of 3D feature fields have expanded the horizons of robotic capabilities. These developments hold significant potential for bridging the gap between high-level reasoning from LMMs and low-level control policies utilizing 3D feature fields. In this work, we introduce LMM-3DP, a framework that can integrate LMM planners and 3D skill Policies. Our approach consists of three key perspectives: high-level planning, low-level control, and effective integration. For high-level planning, LMM-3DP supports dynamic scene understanding for environment disturbances, a critic agent with self-feedback, history policy memorization, and reattempts after failures. For low-level control, LMM-3DP utilizes a semantic-aware 3D feature field for accurate manipulation. In aligning high-level and low-level control for robot actions, language embeddings representing the high-level policy are jointly attended with the 3D feature field in the 3D transformer for seamless integration. We extensively evaluate our approach across multiple skills and long-horizon tasks in a real-world kitchen environment. Our results show a significant 1.45x success rate increase in low-level control and an approximate 1.5x improvement in high-level planning accuracy compared to LLM-based baselines. Demo videos and an overview of LMM-3DP are available at https://lmm-3dp-release.github.io.

cs.RO↗